Transform Your Data Object-Oriented Style with Formatters

0
30
Transform Your Data Object-Oriented Style with Formatters


I admit it. I’m a design pattern nerd. I’ve been a massive fan of object-oriented design patterns ever since I read Patterns of Enterprise Application Architecture in the mid-2000s. One of the aspects which first drew me to Ruby, via Rails, was the in-depth adoption of so many of these patterns and how easy it is to write elegant Ruby code using OOP (Object-Oriented Programming) design patterns.

Today, we’ll talk about a variant of the Template Method design pattern I’ve used in a number of scenarios. I don’t know of a formal name for this pattern, so I like to call it the Formatter pattern. By executing a sequence of formatters, you achieve a process which I think of as a data pipeline. The data pipeline itself can also be represented as an object.

(At first glance, you might think this pattern is better described as a Chain of Responsibility, but that pattern is better suited to cases where each link in the chain should determine whether it can handle all processing and stop, or pass some of the processing directly on to the next link in the chain.)

Perfect for Refactors

The Formatter pattern is so useful that I recently refactored a huge “spaghetti function” in a TypeScript (yes, that TypeScript) codebase to use this OOP pattern instead. The number of bugs and “gotchas” working with that part of the codebase plummeted to nearly zero—and we were able to write unit tests for each formatter in the pipeline to verify the data transformations therein.

It’s safe to say that you wouldn’t necessarily reach for the Formatter pattern the moment you need to author a sequence of steps to transform some data. Rather, you’d start by writing your code in a simple linear fashion, and then as you slowly begin to realize your process is turning into a big ball of mud, it’s time to refactor. (Remember DOEY!)

NOTE: fans of functional programming, you’ll probably want to leave now. This pattern encompasses everything you don’t like about object-oriented programming: abstract classes, inheritance, encapsulation, mutation, and polymorphism. So don’t at me—you’ve been warned!

The Format of Formatters

Formatters in a particular data pipeline all inherit from a base class. Let’s create a new ExampleFormatters Ruby module to house our Base class and future subclasses:

module ExampleFormatters
  class Base
    attr_reader :data

    def initialize(data)
      @data = data
    end

    def format
      raise "Method should be implemented in subclass"
    end
  end
end

You can name the format method however you wish—perhaps in your case convert would be more appropriate. At any rate, the idea here is you pass a data object of some type into your initializer, and then by calling the format method, a transformation would occur for that data. Let’s implement a subclass to see this in action (and let’s assume the data object is a Hash):

module ExampleFormatters
  class TitleFormatter < Base
    def format
      # ensure the title string is in Title Case
      data[:title] = data[:title].split.map do |str|
        str.capitalize
      end.join(" ")

      self
    end
  end
end

Now let’s call this formatter with some data:

hsh = { title: "hello   world!" }

p ExampleFormatters::TitleFormatter.new(hsh).format.data

# => {:title=>"Hello World!"}

Note that we’re not returning a copy of the data object. We’re mutating the data in-place. This is by design (generally better performance and lower memory usage). So in the above example, both hsh and data’s return value are equivalent after running the formatter. If you need to preserve the original state of the data, you should ensure you make a duplicate object before passing that along to the formatters—perhaps in the data pipeline (described further below).

This is the most basic example of how to write a formatter. However, you might want to add a bit of smarts, such that you can’t run format more than once on the same dataset. We can change our base class then to this:

module ExampleFormatters
  class Base
    attr_reader :data

    def initialize(data)
      self.data = data
    end

    def data=(new_data)
      @already_formatted = false
      @data = new_data
    end

    def format
      if @already_formatted
        raise "Cannot run formatter a second time on the same dataset."
      end
  
      process

      @already_formatted = true

      self
    end

    protected

    def process
      raise "Method should be implemented in subclass"
    end
  end
end

Now refactor your subclasses to override process rather than format:

module ExampleFormatters
  class TitleFormatter < Base
    protected # keep `process` hidden from the public API:

    def process
      # ensure the title string is in Title Case
      data[:title] = data[:title].split.map do |str|
        str.capitalize
      end.join(" ")

      self
    end
  end
end

This will ensure you can only run format once on the same data. In order to run it again, you’ll need to use formatter.data = some_new_data to reset the process.

One reason it’s so nice to have a base class from which to inherit is you can add to the base API and your formatter subclasses will have full access to those methods. For instance, if this “titleize” formatting is something you’ll need to do several times across different formatters, you can abstract the implementation:

module ExampleFormatters
  class Base
    # …
    
    # convert a "lower   case" string to "Title Case"
    def titleize(str)
     str.to_s.split.map do { |str| str.capitalize }.join(" ")
    end  
  end
end
module ExampleFormatters
  class TitleFormatter < Base
    protected # keep `process` hidden from the public API:

    def process
      data[:title] = titleize(data[:title])

      self
    end
  end
end

Not only does this DRY up your code, but it fixes a subtle bug as well. Before, if data[:title] was nil, your code would error and crash. Now, by using a separate method which is written defensively (i.e., don’t naively assume the incoming object will always be a String), the worst you’ll get is a value of "". If a blank string itself will cause a problem down the road, make sure you write that code defensively as well!

Constructing the Data Pipeline

Now that we have some formatters working and transforming our data in bite-sized pieces, we need a way to kick off the whole process. Let’s write our data pipeline object. You’ll notice it contains some similar shapes to the formatter base class.

class ExampleDataPipeline
  attr_reader :data

  def initialize(data)
    self.data = data
  end
  
  # Construct the sequence of formatters for the pipeline.
  # Modify or add to this list to update the pipeline:
  def formatters
    [
      TitleFormatter,
      DatesFormatter,
      LineItemsFormatter,
      # future formatters here
    ]
  end

  def data=(new_data)
    @already_formatted = false
    @data = new_data.dup # make a shallow clone
  end

  def run
    if @already_formatted
      raise "Cannot run pipeline a second time on the same dataset."
    end

    formatters.each do |formatter_class|
      formatter_class.new(data).format
    end

    # further processing here?

    @already_formatted = true

    data
  end
end

For the final step, you can run the pipeline with a single statement!

transformed_data = ExampleDataPipeline.new(input_data).run

Boom.

And thanks to the power of object-oriented programming, you can easily experiment with your pipeline—for example by changing the order of formatters or swapping out one formatter for another.

class FasterDataPipeline < ExampleDataPipeline
  def formatters
    [
      TitleFormatter,
      DatesFormatter,
      FasterLineItemsFormatter
    ]
  end
end

# Try benchmarking these two pipelines…

ExampleDataPipeline.new(input_data).run
# vs.
FasterDataPipeline.new(input_data).run

It goes without saying that unit testing formatters in isolation will ensure your formatters don’t become too dependent on each other (aka it would be bad if a formatter only worked if executed after another particular formatter).

Audit Trails? External APIs? Dynamic Converters?

You can build upon this pattern in increasingly sophisticated ways as your business logic needs grow. One idea might be to log the before/after of each formatter as it runs in the pipeline in order to performing an audit of data integrity after the fact. Another idea might be to encapsulate expensive or complex external API calls within formatters so you can work with your overall pipeline steps at a higher level without messy API calls getting in your way. (And then swapping those formatter(s) out with stubs during unit tests becomes quite straightforward.)

In Bridgetown, the subsystem of template converter plugins (which itself is based on Jekyll’s converter subsystem) is architected in a similar fashion to this Formatter pattern. In the Bridgetown case, the order of converters (aka formatters) is determined by a priority designation. High priority converters run first, then medium priorities, then lower priorities. In Bridgetown, I actually did implement an “audit trail” for the conversions, so if you’re curious how a particular resource changed throughout all the conversions, you can inspect that trail and see the state of the template before/after a particular conversion. The precise nature of the converter pipeline is constructed based on extension matches or other metadata (aka .liquid will be converted differently from .erb). Your data pipeline might similarly need to dynamically construct its list of formatters based on the nature of the incoming data.

In summary, I truly adore this design pattern. Once you know it, you start to see its usefulness across a wide variety of scenarios, codebases, and even programming languages!



Source link

Leave a reply

Please enter your comment!
Please enter your name here