Lately we have been working hard on optimizing the performance and increasing the stability of our platform, Secure Data Kit. One issue that we wanted to handle was the timeouts that would happen on our server. Looking into the issue further, we were able to identify that the timeouts commonly happened when a report was requested. A quick note about some of the reports we run on our platform; they're huge. Generating a report is a resource consuming process that would commonly cause the website's server to exceed the memory quota, resulting in a failed report, and a bummed out user. A report is typically an R script with some complex SQL embedded inside, which produces a PDF for the user. Ultimately, to pull all of this off, reports needed to have it's own process, separate from the main web process. In order to achieve this, we used the Sidekiq gem. Sidekiq allowed us to easily define "worker" processes that would handle these sorts of tasks in the background, leaving the main web process alone. Here is a sample of what the report worker class looks like.
class ReportCreateWorker
include Sidekiq::Worker
def perform report_id, user_id
report = Report.find(report_id)
user = User.find(user_id)
ReportMailer.add_report_email(report, user).deliver_now
end
end
By including the Sidekiq::Worker class, our app knows to run this code in the background, away from the main web process. Freeing up the web process allows the site to stay up for the user, and with the report being run in the background, the user is free to visit other pages within the website. The report will get emailed to them once it is finished.
Something to note about the worker classes would be that the only arguments that can get passed to it are strings. That is why we just pass the id of the objects we need to it. Once inside perform, we can use the string id's to find the user and report we need.
Also, aside from adding Sidekiq to your Gemfile, you will need specify the worker processes in the app's Procfile. Here is what our Procfile looks like.
web: bundle exec puma -t 8:16 -p $PORT -e $RAILS_ENV
worker: bundle exec sidekiq -v -q default -q paperclip
log: tail -f log/development.log
Basically, that tells the the host that the worker processes are managed by Sidekiq.
To conclude, I recommend moving any process that has potential to get intense, like running reports, to a background worker. Ever since we moved the report process to a background worker, we no longer see the timeouts that were once common. Using Sidekiq to do so is a simple way to increase stability in your app.
151 Ted Turner Drive, Atlanta, GA
Let's talk: info@standardco.de
