Comparing Background Processing Libraries: Resque

This is part II of the series on comparing Ruby background processing frameworks. You can check out the previous post linked above. Summing up the first post, it covers the principles behind background processing (queues, processes, etc.). Also, the article delves into delayed_job (as well as how to use it to create a background job and process it) which is amazing for getting background processing rolled into your app really quickly. However, it uses ActiveRecord so it is kind of slow and you end up mixing up background and foreground code throughout your app.

Resque hopes to solve both these problems, so, let’s dive in!

Resque

In order to use Resque, you need to have Redis installed. Fortunately, they have great documentation on installation. If you are on Mac OS X (like a lot of us), all you have to do:

brew install redis

On Linux, it is feasible to install redis through your package manager. For example, on Debian/Ubuntu:

sudo apt-get install redis-server

Redis, unfortunately, does not have support on Windows. There was a port submitted, but it was not merged.

Once you have Redis installed and a redis server running, you can add Resque to your Gemfile:

gem 'resque'

And then install the bundle:

bundle install

You need to setup the rake tasks that will allow you to get workers running with your Rails application. To do this, add a file to the “lib/tasks” directory called “resque.rake” (this will define rake tasks related to Resque):

require 'resque/tasks'

If your workers need access to the models (i.e. the background processing that you will be doing will require access to ActiveRecord, etc.), you should add the following to the same file (lib/tasks/resque.rake):

task "resque:setup" => :environment

Finally, you should create a separate directory for your workers since they will be running indpendently of other aspects of your Rails app. I use “app/workers” for this.

Now that we have the initial setup squared away, let’s write our first worker with Resque. Save the following under app/workers/print_worker.rb

class PrintWorker
  @queue = :print

  def self.perform(str)
    puts "From print worker: " + str
  end
end

The code itself is very simple and workers generally follow this sort of pattern. You first define the name of the queue to be an instance variable (you can name these anything you want). Then, all workers have a class method called perform. This is the method that is called to process a job in the queue.

For example, if you are writing a worker that needs to send off some email, this is where the code that sends the email would be written.

In particular, PrintWorker performs a very simple task; it takes a string and then prints it out, with a special “From print worker” note in front of it.

Right now, however, this isn’t really doing any work; the class is simply defined (i.e. there are no jobs in the queue). Let’s fix that.

Here is a simple index controller (obviously, you can take the stuff inside the action and put it in any controller in your application):

class IndexController < ApplicationController
  def index
    Resque.enqueue(PrintWorker, params[:to_print])
  end
end

Another very simple bit of code. All it is doing is getting a user’s request to “index#index” and then adding a job for the print worker. But, the important bit is the params[:to_print], which actually specifies that the string that the PrintWorker will print is determined by the the GET parameter passed to the index action.

If we go to “/index/index?to_print=somestring”, the PrintWorker should print out “somestring”.

However, where on Earth is that going to be printed? We need to set up a worker process! Unlike delayed_job, Resque requires that you set up a worker process which “consumes” the queue of jobs. Let’s do that right now by typing the following in the terminal:

rake resque:work QUEUE=print

We’ve simply used one of the rake tasks provided by resque in order to start a worker process on the queue called “print”. Now, that should give you a process which seems like it isn’t doing anything.

Point your browser to “/index/index?to_print=lol” (with the correct hostname of course) and check out the output of the process and you should see that the print worker printed: “From the print worker: lol”.

Obviously, this seems like a very simple example, and it is. But, it does cover the central idea of resque; workers are defined as classes with a perform method, queues are named, and you can pass variables to the workers.

Okay, now that we know that, let’s take a look at Resque in comparision to delayed_job. Obviously, delayed_job is much easier to get started with. But, in my opinion, delayed_job code tends to get convoluted quite quickly if you’re not careful because you are able to mix asynchronous calls with synchronous calls for the same method. Delayed_job is also likely to be slower since it doesn’t use an in-memory store by default.

However, delayed_job does have some merits. As mentioned, it is extremely easy to get up and running. In addition, its abilities include some things that Resque does not do very well, like prioritizing tasks.

Personally speaking, Resque is what I use on a daily basis. It combines a nice way to structure your code, Redis, and performance all into a great package.

Example App

To order to demonstrate the basic usage of Resque (as well as to test out your Redis instance quickly), I’ve created a small example app that lets you run some simple workers on your machine. It performs exactly the same function as the example app for delayed_job: peforming a page count on a file asynchronously and synchronously. It is a very barebones app, but it serves as a working example for you going forward.

Check it out here

Stay Tuned!

Part III of this article will talk about Sidekiq and also an overall comparision of the three frameworks discussed so far (DelayedJob, Resque, Sidekiq).