If you have an application and need to perform tasks in background then you probably use Resque, Sidekiq or DelayedJob. And if not, then you should really consider using one of these. Each one of them have advantages and fallbacks, Sidekiq uses threads, Resque spawn processes while DelayedJob will store queues in your database.
But for us running applications on Heroku there is always the issue with paying for an extra Dyno only to run a few tasks each day. You may use Hirefire, to scale workers up and down for you, but it will cost you $10, and it can only monitors your application once in a minute. You wouldn’t want that, would you?
Let me present you DynoScaler. With it you can configure your application so that it scales up workers when there are jobs in the queue, and scales down workers when there are no pending or running jobs. It scales up right after enqueueing a job, and scales down after the last job finishes, so that you never have to wait more than you need to.
Let’s get it installed
First you’ll need to add the gem to your Gemfile:
bundle install, and your good to go.
How do we use it?
To use it with Resque, just include
DynoScaler::Workers::Resque in your job class:
MyJob gets queued and there are no workers running DynoScaler will scale up a worker to process the job. It will eventually scale more workers depending on the total number of pending jobs in all queues. This can all be configured among other things.
For example you may tell DynoScaler to always keep at least one worker running, and to scale at most 3 workers. You can also configure how many workers should be spawned given a pending queue size. In a rails application, you could just add this lines to your
To enable DynoScaler in Heroku just configure these two configs:
DynoScaler needs your API key and the name of the application in order to scale workers up and down, and that’s all it needs.
You can get more info by following the README.
Tuning it even further
If you followed the instructions up until now and have give it a try (what? have you not?) with one of your applications in Heroku, you may have notice some slowdown in your actions. What is happening? Well, since we need to talk to Heroku APIs every time we enqueue a job, it will actually perform an HTTP request to Heroku.
This request may take some time, even though the guys at Heroku have done a great job, you can’t predict how much time it will take for this request to return. If we leave it like that it will block your application until it returns, and slowdown things, specially if you enqueue lots of jobs in a single action.
How can we fix this? It's simple, perform the request to scale up workers asynchronously. But how do we do that? We use Girl Friday, which will use Actors, which in turn use Threads, to run tasks. Just add it to your Gemfile:
And configure DynoScaler to be async:
This will spawn a
GirlFriday::WorkQueue with one thread, in your application’s dyno that will handle the requests to the Heroku API to scale workers up when enqueueing jobs. If you want you can pass a block when configuring it so that you can tune it even further (you could keep the queue in Redis, for instance):
This gem was mostly inspired by this post. It may still have some flaws, but we are successfully using them in production for a while in some applications. For now it only scales workers up and down and supports only Resque, but we will probably add support for Sidekiq and DelayedJob.
You’re more than welcome to help, just fork the repository and send us a pull request!