Comparing Ruby Background Processing Libraries: Delayed Job

Once you start working on Rails applications beyond the typical “tutorial style” blog, you will probably need some kind of processing outside of just responding to user requests.

For example, if you need to send emails to some of your users to let them know about their trial plan expiring, you can’t really handle this inside a controller because there isn’t a specific request that a certain user needs to make to set this off. You need this kind of logic to be completely separated from the part of your application that directly responds to HTTP requests (so, for Rails, these are your controllers and the actions within them, and, to some extent, the views).

Another example of where you might need this kind of “thing” (let’s call it background processing or background jobs) would be an operation that takes a significant amount of time (say, as a rule of thumb, anything that takes more than 400ms) and you don’t want to keep your user waiting. Pretend you are running a video site. Once a user uploads a video, your application has to encode it which might take more than ten minutes. Obviously, you can’t just keep the user hanging on the HTTP request (which would likely timeout) while you are encoding the video. Instead, you need to somehow tell Rails, “here, take this code and run in the background while I respond to the user with some HTML”.

Another really common use case is found when your app needs to communicate with other servers. This can take a lot of time and is also a pretty error-prone procedure (e.g. the server at the other end could be down.) In this case, delegating it to the background makes a lot of sense.

Let’s dive in to see how background processing works as well as the alternatives the Ruby community offers to implement it in your application.

Theory

It is very important to understand the basic principles behind background jobs/processing. With that knowledge, you’ll know what you’re working with in case you have to do some debugging.

Modern operating systems allow many processes to run “at once” on a computer. Notice that “at once” was in quotes; usually, all the processes don’t all run at exactly the same moment (simplified explanation: if you have four cores, usually four processes can run at a time). A piece of the operating system called the scheduler gives each process a bit of time to run and then switches to another process, doing this for every process on the system. By doing so, the scheduler provides the illusion of all processes running simultaneously, because it switches between processes so quickly (similar to how an LED flashing very fast seems as though it is on continuously).

Maybe what we can do is create a new process (aside from the ones that respond to HTTP requests) for each job we’d like to process in the background. But, as it turns out, creating processes isn’t exactly a lightweight operation in terms of the computing resources that are used. If we were to go through with this strategy and had 100,000 jobs, we’d need to create 100,000 processes! Instead, background processing can use queues.

A queue is a well-defined data structure that allows us to add stuff to it and take stuff out according to certain rules. It is a FIFO (first-in-first-out) structure, which means that if you put in “a” then you put in “b”, you retrieve “a” before “b” (a bit like adding to the beginning of a list and then reading from the end of it). The point being that we can add “jobs” to queues and then workers (running in their own process) will process them at a later time.

Turns out that managing all of this can be difficult, to say the least. For example, what would the system do if a worker or a job was taking too long and holding up the rest of the queue? How would you make sure your queue was performing well (remember that the queue itself has to be held either in memory or in disk)? Fortunately for us, people in the Ruby community have already figured out a lot of this stuff and rolled them up into several different libraries, such as delayed_job, Resque and Sidekiq which all take different approaches to nearly the same problem.

Delayed::Job

Shopify scratched their own itch with Delayed::Job, which they were nice enough to open source and write up some docs so that the rest of the Ruby community can take advantage of it.

Considering only the stable backends for Delayed::Job (henceforth referred to as DJ), it uses ActiveRecord, Mongoid, etc. as the system on which to base the queue. That means that you don’t need to run anything else (such as Redis) to serve as your queuing system; the ActiveRecord you have in your Rails app will work.

Getting started with DJ is the easiest out of all three, which is why a lot of people prefer it. Add the following to your Gemfile:

gem 'delayed_job_active_record'

Install the gem and set up the necessary database structures by typing this into your shell:

bundle install
rails generate delayed_job:active_record
rake db:migrate

DJ code is really easy to write and understand. All you have to do is add .delay.method(parameters) to any object to call method asynchronously (i.e. in the background; your code will move right along and not wait for the output) with the given parameters.

For example, if you have the following user model:

class User < ActiveRecord::Base
...
  def send_newsletter(email)
    #send the newsletter here, which will take some time and you
    puts 'From send newsletter: ' + email
  end
...
end

You can do the following in a controller:

class IndexController < ApplicationController
  def index
    u = User.new
    #do some processing on the user object
    u.save

    #send the user a newsletter...delayed!
    u.delay.send_newsletter(params[:email])         
  end
end

Or, you could change the model to ensure that send_newsletter is handled asynchronously no matter what:

class User < ActiveRecord::Base
  ...
  handle_asynchronously :send_newsletter
  …
end

Then, you only need this in the controller (i.e. no .delay) in order to call the method asynchronously (i.e. place it in a queue as a job):

u.send_newsletter(params[:email])

To see this running, you can set up the controllers and models as noted and run the Rails server. We need to do a tiny bit of additional setup through our shell to get our queue going:

rails generate delayed_job:active_record
rake db:migrate

Now, visiting “/index/index” will push the job to the queue (since we are using .delay). To process the job, all we have to do is type this into a shell:

rake jobs:work

You should see a “newsletter: done!” in the output. Great!

As you can see, DJ makes it incredibly easy to do some pretty complicated stuff (We’ve created a queue, put a job in it with the email parameter and processed the job!). They also have nice docs as well as a large user community. In particular, DelayedJob makes it really easy to access model parameters since the methods are right in your model. Unfortunately, it does have some downsides as well.

DJ isn’t exactly the fastest option. Most recommend that if you’re doing a ton of jobs in a day, you should probably move to something else. One of the major reasons behind the lack of performance in DJ is the choice of the database to use to support the queue.

DJ basically needs to run on a database to support its queue. This means that you’re actually writing to disk when you really don’t need to be (you could hold the queue entirely in memory for most of the time).

This is where something like Redis comes in – it is a NoSQL datastore which allows one to hold data in memory (a bit like memcached). Redis has been described as a “remote dictionary”, i.e. it is something like a Ruby hash, but on a server. It might seem like Redis is a little pointless if it is only held in memory (i.e. when you restart your server, its all gone), however, Redis can persist to disk as well!

DJ is moving towards supporting Redis. However, the support right now is unstable, so, I wouldn’t use it in production. Resque, on the other hand, is built from the bottom up on Redis.

Example app

I’ve built an example app to show off a simple example of delayedjob in action that you can run locally (it uses the filepicker demo as a starting point). Basically, you upload an attachment and it will asynchronously set its page count. However, if you do a “show” action on one of these uploaded attachments (a resource route), it finds out the page count synchronously. The point is to show how easily you can set up delayedjob code as well as how synchronous and asynchronous code can be mixed using only one method (which, in my opinion, is a downside of delayed_job because it makes it harder to reason about certain methods).

Check it out here

Stay tuned!

Part II of this article will cover Resque and compare it with DelayedJob!