pmon3 is a process manager for Golang applications. It allows you to keep applications alive forever and to reload them without downtime.
- Introduction
- Installation
- Commands
- Application Configuration
- Event Handling
- Debugging
- Performance
- Problems
Golang currently has no officially supported process management tools. For the deployment of Golang services, some use Linux built-in commands such as nohup [process] &, or the process management tools provided by the operating system such as SystemD. Alternatively, third-party process management tools such as: Python's Supervisor or Node.js PM2 can also be utilized
Each method has certain advantages and disadvantages. We hope to provide a convenient and easy-to-use tool for Golang process deployment. There is no need to install other dependencies besides bash-completion for ease of command line utilization.
Unlike PM2, pmon3 is managed directly by the OS process manager, so even if the pmon3 management tool abnormally terminates, it will not affect the parent pmond process itself. This is currently achieved by separating the pmond daemon process from the pmon3 agent.
By default, if a process abnormally terminates, pmond will try to restart the process. If you don't want a process to restart automatically, then you can provide the --no-autorestart parameter flag.
The systemd installation process entails the following steps:
- create the log, configuration and database directories
- create the log rotation file
- create the bash completion profile (requires the bash-completion package)
- enable and start the
pmondsystem service
#build the project
make build
#install on systemd-based system
make systemd_installwget -O - https://raw.githubusercontent.com/joe-at-startupmedia/pmon3/master/release-installer.bash | bash -s 1.14.7âť—âť— Note âť—âť—
After installing pmon3 for the first time, both installation methods provided above should automatically enable and start the service. if the pmond service does not start automatically, you need to manually start the service.
sudo systemctl start pmond
# Others
sudo /usr/local/pmon3/bin/pmond &Usage:
pmon3 [command]
Available Commands:
completion Generate completion script
del Delete process by id or name
desc Show process extended details
drop Delete all processes
exec Spawn a new process
help Help about any command
init Restart all stopped processes
kill Terminate all processes
log Display process logs by id or name
logf Tail process logs by id or name
ls List all processes
restart Restart a process by id or name
stop Stop a process by id or name
topn Shows processes using the native top
version
Flags:
-h, --help help for pmon3
Use "pmon3 [command] --help" for more information about a command
pmon3 exec [application_binary] [flags]
The starting process accepts several parameters. The parameter descriptions are as follows:
// The process name. It will use the file name of the binary as the default name if not provided
--name
// Where to store logs. It will use the following as the default: /var/log/pmon3/
--log -l
// Only custom log directory, priority is lower than --log
--log_dir -d
// Provide parameters to be passed to the binary, multiple parameters are separated by spaces
--args -a "-arg1=val1 -arg2=val2"
// managing user
--user -u
// Do not restart automatically. It will automatically restart by default.
--no-autorestart -n
pmon3 exec ./bin/gin --args "-prjHome=`pwd`" --user ntt360
âť—âť— Note âť—âť—
Parameter arguments need to use the absolute path.
pmon3 ls
This will output the resource utilization of all processes using the native top command that is pre-installed on most unix-based operating systems. It will only show those processes managed by (and including) the pmond process. The output is updated every few seconds until the process is terminated using Ctrl+C.
pmon3 topn
pmon3 restart [id or name]
pmon3 stop [id or name]
# view logs of the process specified
pmon3 log [id or name]
# view logs of the process specified including those previously rotated/archived
pmon3 log -a [id or name]
# Similar to using `tail -f xxx.log`
pmon3 logf [id or name]
pmon3 del [id or name]
pmon3 show [id or name]
pmon3 kill [--force]
pmon3 init
pmon3 drop [--force]
By default, when pmond is restarted from a previously stopped state, it will load all processes in the database that were previously running, have been marked as stopped as a result of pmond closing and have --no-autorestart set to false (default value).
When an application config is provided pmond will instead refer to the apps array specified based on the following criteria:
- When
pmondis starting from a fresh install - When
pmon3successfully executes a drop command followed by running an init command.
If init is ran while there are still stopped process in the database (resulting from pmond daemon restart or kill, the Application config will NOT be used, and instead the previously-stopped process will be restarted.
apps_config_file: /etc/pmon3/config/apps.config.json{
"apps": [
{
"file": "/usr/local/bin/happac",
"flags": {
"name": "happac1",
"args": "-h startup-patroni-1.node.consul -p 5555 -r 5000",
"user": "vagrant"
}
},
{
"file": "/usr/local/bin/happab",
"flags": {
"name": "happac2",
"log": "happac2",
"args": "-h startup-patroni-1.node.consul -p 5556 -r 5001",
"user": "vagrant"
"no_auto_restart": true
}
},
{
"file": "/usr/local/bin/node",
"flags": {
"name": "metabase-api",
"args": "/var/www/vhosts/metabase-api/index.js",
"user": "dw_user"
}
}
]
}All possible flags values matching those specified in the exec command:
- user
- log
- log_dir
- no_auto_restart
- args
- name
# a script to execute when a process is restarted which accepts the process details json as the first argument
on_process_restart_exec: ""
# a script to execute when a process fails (--no-autorestart) which accepts the process details json as the first argument
on_process_failure_exec: ""1. Specify the executable script to run for the on_process_restart_exec value. pmond will pass a json-escaped string of the process details as the first argument.
on_process_restart_exec: "/etc/pmon3/bin/on_restart.bash"PROCESS_JSON="$1"
PROCESS_ID=$(echo "${PROCESS_JSON}" | jq '.id')
PROCESS_NAME=$(echo "${PROCESS_JSON}" | jq '.name')
echo "process restarted: ${PROCESS_ID} - ${PROCESS_NAME}" >> /var/log/pmond/output.log$ PMON3_DEBUG=true pmond
INFO/vagrant/go_src/pmon3/pmond/observer/observer.go:29 pmon3/pmond/observer.HandleEvent() Received event: &{restarted 0xc0001da630}
WARN/vagrant/go_src/pmon3/pmond/observer/observer.go:47 pmon3/pmond/observer.onRestartEvent() restarting process: happac3 (3)
DEBU/vagrant/go_src/pmon3/pmond/observer/observer.go:70 pmon3/pmond/observer.onEventExec() Attempting event executor(restarted): /etc/pmon3/bin/on_restart.bash "{\"id\":3,\"created_at\":\"2024-05-03T05:44:25.114957302Z\",\"updated_at\":\"2024-05-03T06:09:18.71222185Z\",\"pid\":4952,\"log\":\"/var/log/pmond/acf3f83.log\",\"name\":\"happac3\",\"process_file\":\"/usr/local/bin/happac\",\"args\":\"-h startup-patroni-1.node.consul -p 5557 -r 5002\",\"status\":2,\"auto_restart\":true,\"uid\":1000,\"username\":\"vagrant\",\"gid\":1000}"$ tail /var/log/pmond/output.log
process restarted: 4 - "happac4"You can specify debug verbosity from both the pmon3 client and the pmond daemon process using the PMON3_DEBUG environment variable.
PMON3_DEBUG=true pmond PMON3_DEBUG accepts the following values:
true: sets the debug level to debugdebug: has the same effect as trueinfo: sets the debug level to infowarn: sets the debug level to warnerror: sets the debug level to error
You can also debug the underlying IPC library using QOG_DEBUG=true
QOG_DEBUG=true IPC_DEBUG=true PMON3_DEBUG=true pmon3 lsIf you want more control over the verbosity you can set the loglevel in the yaml configuration file.
# log levels: debug/info/warn/error
log_level: "info"
If you do not specify a value, info will be the default Logrus level.
By default, no underlying libraries require CGO. This allows for portability between machines using different versions of GLIBC and also provides easy installation using the Release Installer . Benchmarking results have confirmed less memory and CPU utilization compared to using the libraries which do require CGO_ENABLED=1 provided below:
The posix_mq build tag can be provided to swap out the underlying golang-ipc library with posix_mq. The posix_mq wrapper does require CGO_ENABLED=1 and is considerably faster but also consumes slightly more CPU and Memory. To enable posix_mq during the build process:
BUILD_FLAGS="-tags posix_mq" make build_cgoBy default, pmon3 utilizes an non-CGO version of sqlite which is unnoticably less performant in most circumstances. To enable the CGO version of sqlite:
BUILD_FLAGS="-tags cgo_sqlite" make build_cgoIt depends on your requirements whether you need one or both. To enable both of these CGO-dependent modules for maximizing overall performance:
BUILD_FLAGS="-tags posix_mq,cgo_sqlite" make build_cgoOr without using the Makefile:
CGO_ENABLED=1 go build -tags "posix_mq,cgo_sqlite" -o bin/pmon3 cmd/pmon3/pmon3.go
CGO_ENABLED=1 go build -tags "posix_mq,cgo_sqlite" -o bin/pmond cmd/pmond/pmond.gopmon3 comes with a logrotate configuration file, which by default utilizes the /var/log/pmond directory. If you require a custom log path, you can customize config.yml and rpm/pmond.logrotate
If there is a path in the parameters you pass, please use the absolute path. The pmon3 startup process will start a new sandbox environment to avoid environmental variable pollution.
pmon3 provides Bash automation. If you find that the command cannot be automatically provided, please install bash-completion and exit the terminal to re-enter:
sudo yum install -y bash-completionautoload -U +X compinit && compinit
autoload -U +X bashcompinit && bashcompinit
sudo sh -c "pmon3 completion zsh > /etc/profile.d/pmon3.sh"
source /etc/profile.d/pmon3.shIf you encounter the error above, make sure the pmond service has started successfully.
sudo systemctl start pmondYou should only use sudo to start the pmond process which requires superuser privileges due to the required process forking commands. However, the pmon3 cli should be used without sudo to ensure that the spawned processes are attached to the correct parent pid. When using sudo, the processes will be attached to ppid 1 and as a result, will become orphaned if the pmond process exits prematurely. Using sudo also prevents non-root users from being able to access the log files. The following Makefile command applies the adequate non-root permissions to the log files.
#This is automatically called by make systemd_install
make systemd_permissionspmon3 exec /usr/local/bin/happac --user root