IronMQ and Laravel: Delays and Retries

Previously, we saw how to use Iron push queues with Laravel. All we needed to do was set up an Iron account, add a subscriber URL, push a message to queue, and receive the message.

The way Laravel supports Iron push queues out-of-the-box is amazing, but there are always limitations. In this article we focus on those limitations and learn to tackle them.

Scenarios

There are three possible scenarios:

You received the message and you finished the job successfully.
- You need provision to tell Iron that the job was completed successfully and it needs to delete it from the queue.
You received the message and something went wrong. Basically, your job was not successful.
- In this case we should tell Iron that we failed and it should retry or if you couldn’t tell Iron anything, it should be clever enough to retry the job i.e., push the same message again to your subscriber after some time.
You received the message, but your job is a long running process (something which takes more than a minute)
- Iron should wait longer before it resends the message.

These are all practical scenarios and you will face them on a daily basis. And yes, you need to cover these cases in your applications.

It’s possible to use Iron to its full power if we use the Iron MQ PHP Library. During setup we included "iron-io/iron_mq": "1.4.6" in our composer file. The code for the same is hosted at https://github.com/iron-io/iron_mq_php. Let’s take a peek into the documentation and try to understand how to use this library.

We initiate a queue as follows:

$ironmq = new IronMQ(array(
        "token" => 'XXXXXXXXX',
        "project_id" => 'XXXXXXXXX'
    ));

And this is how we post to a queue:

$ironmq->postMessage($queue_name, "Test Message", array(
        'timeout' => 120,
        'delay' => 2,
        'expires_in' => 2*24*3600 # 2 days
    ));

To turn a queue into a push queue (or create one) all we have to do is post to the queue with options like a list of subscribers, push_type etc. Using the library you can do the following:

Note: If a queue doesn’t have any subscribers it is a pull queue.

$params = array(
        "push_type" => "multicast",
        "retries" => 5,
        "subscribers" => array(
            array("url" => "http://your.first.cool.endpoint.com/push"),
            array("url" => "http://your.second.cool.endpoint.com/push")
        )
    );
    
    $ironmq->updateQueue($queue_name, $params);

That’s it!

That’s exactly what Laravel does when you run the artisan queue:subscribe command. To recap, after you do the necessary setup in the queue.php file (filling it with project_id, token and queue_name), you run a command to register your subscribers:

php artisan queue:subscribe queue_name http://foo.com/queue/receive

This command runs the following code:

/vendor/laravel/framework/src/Illuminate/Queue/Console/SubscribeCommand.php

public function fire()
    {
		$iron->getIron()->updateQueue($this->argument('queue'), $this->getQueueOptions());
	}

Some content is excluded from the file for brevity.

This is how Laravel converts our queue into a push queue. This is the same way it’s done when using the Iron MQ library.

Using the Iron MQ PHP Library

Create a new controller IronController in app/controllers

<?php
        
        class IronController extends Controller {
        
            protected $ironmq;

        	public function __construct()
        	{
        		$this->ironmq = new IronMQ(array(
        					    "token" => 'XXXXXXXXXXXXXXXXXXX', //Your token from dashboard
        					    "project_id" => 'XXXXXXXXXXXXXXX' //Your project ID from dashboard
        					));
        	}
        
        }

app/controller/IronController.php

Before we create our methods, let’s create routes to understand what our controller will do:

Route::get('iron/create','IronController@create'); //Create a Queue and add subscribers
        Route::get('iron/push','IronController@push'); //Push a message to Queue
        Route::get('iron/status/{id}','IronController@status'); //Know the status of the message we pushed
        Route::post('iron/receive','IronController@receive'); //Receive the message and process it

app/routes.php

Now, let’s write a method to create a queue with a subscriber

public function create()
    	{
    		$params = array(
    		    "subscribers" => array(
    		        array("url" => url('iron/receive'))
    		    )
    		);
    
    		$this->ironmq->updateQueue('testing', $params);
    
    		return 'Push Queue Created';
    	}

app/controller/IronController.php

You need to visit http://953ffbb.ngrok.com/iron/create i.e., use your ngrok url appended by /iron/create. Note: Your ngrok url will be different.

At this point you can go to Iron DashBoard -> MQ -> Queues -> Find a queue with name ‘testing’ -> Push Queues and make sure your subscriber is listed there similar to http://953ffbb.ngrok.com/iron/receive.

Let’s add methods to push a message onto a queue and check the status.

Method to push a message:

public function push()
        {
        	//Just some data you want to pass
        	$data = array(
        		'first_name' => 'Rajiv', 
        		'last_name'  => 'Seelam'
        	);
        
        	//Convert data into a string so that we can pass it
        	$data = serialize($data);
        
        	//Post the message
        	$job = $this->ironmq->postMessage('testing', $data);
        
        	Log::info(__METHOD__.' - Message pushed with job_id : '.$job->id);
        
            return Redirect::to('iron/status/'.$job->id);
        }

Method to check the status of a message:

public function status($id)
	{
	$status = $this->ironmq->getMessagePushStatuses('testing',$id);
	dd($status);
	}

app/controller/IronController.php

We need a method to receive the message when we push.

public function receive()
        {
        	$req = Request::instance();
    
    		$jobId = $req->header('iron-message-id'); //Get job id 
    
    		$data = unserialize($req->getContent()); //Get content and unserialize it
    
    		Log::info(__METHOD__.' - Job Received From Iron with Job ID : '.$jobId);
    
    		return Response::json(array(),200);
    	}

app/controller/IronController.php

At this point you should understand that this is the method which will receive the message because in the create method we listed url('iron/receive') as the subscriber (check the routes file to confirm this url routes to this method).

Keep your app/storage/logs/laravel.log open to see what is happening.

Now, if you visit http://953ffbb.ngrok.com/iron/push, you should see a message similar to the following in your browser:

array (size=1)
          0 => 
            object(stdClass)[184]
              public 'retries_delay' => int 60
              public 'retries_remaining' => int 3
              public 'retries_total' => int 3
              public 'status_code' => int 200
              public 'status' => string 'deleted' (length=7)
              public 'url' => string 'http://953ffbb.ngrok.com/iron/receive' (length=38)
              public 'id' => string '6019569765635787179' (length=19)

And your logs should be similar to :

[2014-05-31 12:45:04] production.INFO: IronController::push - Message pushed with job_id : 6019569765635787179 [] []
[2014-05-31 12:45:05] production.INFO: IronController::receive - Job Received From Iron with Job ID : 6019569765635787179 [] []

app/storage/logs/laravel.log

These logs confirm our activity. Let’s summarize what we did here :

We pushed a message to a queue called ‘testing’
Then we checked the status of the message on the queue
We received the message and logged something.

Delays and Retries

Now, let’s go a bit deeper. Open your Iron dashboard and go to the queues page (Click on MQ) -> Open testing queue and check ‘Push Queues’, you will see the list of subscribers and under ‘Push Information’ you will see ‘Retries’ and ‘Retries Delay’.

What do these mean?

Retries: The default value is 3, which means Iron will retry the message 3 times.

Retries Delay: The default value is 60, which means Iron will by default push the message again to subscribers after 60 seconds if it thinks the message was not processed successfully.

When does Iron retry a message?:

When it doesn’t get a response from your application:
- Your application failed to give a response because of some error.
- Your application is still processing the job, a long running process.
When it gets an error in response:
- Your app responded with 4xx or 5xx error.
When you send a 202 response:
- You are asking Iron to resend the message after a delay.

If you check the output of status (see above) you will notice the following line:

public 'status' => string 'deleted'

This is how we will get to know the status of the message we pushed. The possible values for status (currently) are:

retrying – The job will be sent to endpoint i.e., subscriber (as many times as ‘Retries’ is set).
deleted – The job is deleted.
reserved – The job will be retried after timeout.
error – There was an error and nothing more will be done.

You have to send a 200 response to delete a message.

One has to remember that Iron will retry a message after timeout if it doesn’t get a response (it will retry after timeout as many times as you specify in ‘retries_total’).

Can we change these parameters? Of course!

While posting a message you can mention timeout (long running jobs may need more than a minute)

$this->ironmq->postMessages('testing', "data", array(
        "timeout" => 300 // Wait for 300 seconds = 5 minutes
    ));

If you want to decrease or increase the number of retries, you have to call the updateQueue method.

$params = array(
	    "retries" => 5
	);

	$this->ironmq->updateQueue('testing', $params);

Can we change the status of a message which is in the queue? Of course! That is how you are supposed to use queues and in fact you are already doing that.

Let’s see how we can alter the status of a message:

retrying – This status is set when you push a message.
deleted – This status is set when the subscriber responds with a 200.
reserved – This status is set when the subscriber responds with a 202.
error – If Iron exhausted number of retries and if it still doesn’t get a 200 response, the status is set to error.

Note: When your application responds with 4xx or 5xx errors (which usually means something went wrong with the server) Iron waits longer than the mentioned timeout.

Conclusion

Push queues aren’t rocket science when you look at them step by step. Try it yourself and tell us what you think in the comments below! I sincerely hope this helped you understand this often intimidating topic. I highly recommend you to read: http://dev.iron.io/mq/reference/push_queues/. Note that you can find the the source code for this article at: https://github.com/rajivseelam/laravel-queues.

Thanks for reading!

Frequently Asked Questions (FAQs) about Laravel Delays and Retries

How can I delay a job in Laravel?

In Laravel, you can delay a job by using the delay method. This method allows you to specify the number of seconds or a DateTime instance representing when the job should be added to the queue. Here’s an example:

$job = (new ProcessPodcast($podcast))
->delay(Carbon::now()->addMinutes(10));

dispatch($job);
In this example, the ProcessPodcast job will be delayed by 10 minutes.

How can I retry a failed job in Laravel?

Laravel provides a command to retry failed jobs. You can use the queue:retry command followed by the ID of the failed job. For example:

php artisan queue:retry 5
This command will retry the failed job with an ID of 5. If you want to retry all failed jobs, you can use the all keyword:

php artisan queue:retry all

How can I specify the number of times a job should be attempted?

You can specify the number of times a job should be attempted before it’s considered as failed by defining a tries property on your job class. For example:

/**
* The maximum number of times the job may be attempted.
*
* @var int
*/
public $tries = 5;
In this example, Laravel will attempt to run the job 5 times before marking it as failed.

How can I handle job failures in Laravel?

Laravel provides a method called failed that you can use to handle job failures. This method is called when a job has exceeded the maximum number of attempts or has thrown an exception. Here’s an example:

/**
* The job failed to process.
*
* @param Exception $exception
* @return void
*/
public function failed(Exception $exception)
{
// Send user notification of failure, etc...
}
In this example, you can add your own code to handle the failure, such as sending a notification to the user.

How can I delay a job in Laravel using IronMQ?

IronMQ is a message queue service that Laravel supports out of the box. To delay a job using IronMQ, you can use the later method. This method accepts a DateTime instance indicating when the job should be added to the queue. Here’s an example:

$job = (new ProcessPodcast($podcast))
->later(Carbon::now()->addMinutes(10));

dispatch($job);
In this example, the ProcessPodcast job will be delayed by 10 minutes using IronMQ.

How can I configure the retry delay in Laravel?

You can configure the retry delay by defining a retryAfter method on your job class. This method should return the number of seconds to wait before retrying the job. Here’s an example:

/**
* Determine the time at which the job should timeout.
*
* @return \DateTime
*/
public function retryAfter()
{
return now()->addSeconds(5);
}
In this example, Laravel will wait 5 seconds before retrying the job.

How can I prioritize jobs in Laravel?

Laravel allows you to prioritize jobs by pushing jobs to different queues. Each queue can have a priority and workers can be instructed to work on the queues in a specific order. Here’s an example:

dispatch((new ProcessPodcast($podcast))->onQueue('high'));
In this example, the ProcessPodcast job is pushed to the ‘high’ queue.

How can I run the Laravel queue worker?

You can run the Laravel queue worker using the queue:work command. This command will start a worker that will process new jobs as they are pushed onto the queue. Here’s an example:

php artisan queue:work
You can also specify the queue or queues the worker should work on:

php artisan queue:work --queue=high,low
In this example, the worker will first process jobs on the ‘high’ queue, then jobs on the ‘low’ queue.

How can I configure the maximum job attempts in Laravel?

You can configure the maximum job attempts in the queue configuration file. In this file, you can set the tries option to the maximum number of times a job should be attempted. Here’s an example:

'connections' => [
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
'tries' => 3,
],
],
In this example, Laravel will attempt to run each job 3 times before marking it as failed.

How can I configure the job timeout in Laravel?

You can configure the job timeout by defining a timeout property on your job class. This property specifies the number of seconds the job can run before it’s terminated. Here’s an example:

/**
* The number of seconds the job can run before timing out.
*
* @var int
*/
public $timeout = 120;
In this example, the job will be terminated if it runs for more than 120 seconds.