Skip to content

Latest commit

 

History

History
197 lines (155 loc) · 7.63 KB

production.md

File metadata and controls

197 lines (155 loc) · 7.63 KB
Raw

Make It Production-Ready

Using a Release Build (Manual Setup)

A release build is available on the releases page.

For example, the following commands download and extract the Fireworq binary for Linux AMD64 (x86-64) platform.

$ OS=linux
$ ARCH=amd64
$ curl -L  $(curl -sL  https://api.github.com/repos/fireworq/fireworq/releases/latest | jq -r '.assets[].browser_download_url' | grep "_${OS}_${ARCH}.zip") > fireworq_${OS}_${ARCH}.zip
$ unzip fireworq_${OS}_${ARCH}.zip fireworq

Before running Fireworq, make sure that you have

  1. MySQL running somewhere in your host or network on mysql_host:mysql_port,
  2. some database database on the MySQL server, and,
  3. some DB user user with some password password, who is granted CREATE, INSERT, DELETE, UPDATE and SELECT rights on database.

Then the following commands run Fireworq with the prepared MySQL database.


$ export FIREWORQ_MYSQL_DSN=user:password@tcp(mysql_host:mysql_port)/database
$ export FIREWORQ_QUEUE_DEFAULT=default
$ export FIREWORQ_BIND=0.0.0.0:8080
$ ./fireworq

You can specify different MySQL database names, hosts, users or passwords for FIREWORQ_REPOSITORY_MYSQL_DSN and FIREWORQ_QUEUE_MYSQL_DSN if you prefer.

Preparing a Backup Instance

Fireworq provides a mechanism to run a fail-safe backup instance for redundancy. You have only to run a secondary instance against the same DB configuration to achieve this. You can even run more than two instances. Those instances are typically on a different host from each other.

When multiple instances get ready, whichever instance accepts pushing a job, which will be handled by an active instance for the queue. When the active instance dies, another instance will be active automatically. If the underlying DB server dies, all the instances get inactive until the DB server recovers.

Note that multiple instances theoretically form a cluster; each queue may be handled by a different instance. This situation is unlikely to happen for now because there is no way to deactivate a single queue handling, but there is no guarantee that an active instance handles all the queues and the others are totally inactive.

Graceful Shutdown/Restart

Shutdown

A Fireworq daemon can be terminated gracefully by SIGINT, SIGTERM or SIGHUP. It will wait for accepted API requests to be processed and grabbed jobs to be completed until timeout specified by FIREWORQ_SHUTDOWN_TIMEOUT occurs.

Restart

To restart a Fireworq daemon gracefully, wrap the daemon with a tool like start_server.

$ export ...
$ start_server --port=8080 -- ./fireworq

Sending SIGTERM or SIGHUP to the start_server process will gracefully shutdown or restart the daemon respectively.

Logging

Fireworq has three types of logs: an error log, an access log and a queue log. The error log goes to the standard error in a colorized pretty format and the access log and the queue log goes to the standard output in JSON format by default. You can change the destinations of log outputs to files by specifying FIREWORQ_ERROR_LOG, FIREWORQ_ACCESS_LOG and FIREWORQ_QUEUE_LOG respectively. The error log will also be in JSON format if it is written to a file. You can control what to output to the error log and the queue log by specifying FIREWORQ_ERROR_LOG_LEVEL and FIREWORQ_QUEUE_LOG_LEVEL respectively.

To rotate the file logs, use a tool like logrotate. Be aware that some log lines may be lost by logrotate with copytruncate. Instead, ask Fireworq to reopen the log files by sending USR1 signal after a rotation. It should be something like this:

/var/log/fireworq/*.log {
  rotate 7
  size 10k
  missingok
  notifempty
  sharedscripts
  postrotate
    [ -f /var/run/fireworq.pid ] && kill -USR1 `cat /var/run/fireworq.pid`
  endscript
}

Note that you need FIREWORQ_PID=/var/run/fireworq.pid to get it work.

Monitoring

Fireworq provides some statistics and they can be monitored by your favorite monitoring tool such as Mackerel, Zabbix, Sensu or Munin.

Go Stats

Statistics of Go runtime metrics are provided by /stats. You can easily monitor these metrics by using plugins for your monitoring tool.

Tool Plugin
Mackerel mackerel-plugin-gostats
Zabbix fukata/golang-stats-api-handler-zabbix-userparameter
Sensu sensu-plugins-golang
Munin fukata/golang-stats-api-handler-munin-plugin

Queue Stats

Statistics of job queue metrics are provided by /queues/stats or /queue/{queue_name}/stats. You can easily monitor these metrics by using plugins for your monitoring tool.

Tool Plugin
Mackerel mackerel-plugin-fireworq

Alerts

You can get alerts when a job permanently failed by using plugins for your monitoring tool.

Tool Plugin
Mackerel mackerel-check-fireworq