garm v0.1.4

Welcome to GARM version v0.1.4!

Documentation relevant to this version is available at: https://github.com/cloudbase/garm/tree/v0.1.4. The main branch contains code that is still under development, and the documentation is relevant to that code. When deploying stable versions, please view the docs available under the relevant tag.

Before we dive into the highlights, there are a couple of things we need to mention.

Breaking change warning ⚠️

This version removes the LXD internal provider in favor of two new external providers:

• garm-provider-lxd
• garm-provider-incus

This was done mainly due to the licensing change of LXD. If you're using LXD and are upgrading from previous versions, you must download the LXD external provider and configure it.

Another breaking (sort of) change is in the garm-cli. The --force flag in the runner remove command, is no longer mandatory when removing a runner. Moreover, the --force flag now ignores provider errors. In previous versions, if a provider erred when removing a runner, GARM would keep trying until it succeeded. The problem was that if a provider was misconfigured, we would be stuck with a runner in error state until re manually removed it from the database. The --force flag allows us to remove a runner from GARM even if the provider returns an error.

Database upgrade ⚠️

This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes.

To safely create a backup of you DB, you can use the sqlite3 CLI:

# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.db

Highlights

This release is packed with optimizations, changes and a couple of nice features.

JIT runners

The main highlight of this release is the ability to use just-in-time self-hosted runners. To accommodate this change, GARM now has more robust metadata endpoints that can serve the needed JIT files. This will most likely be expanded in the future to include setup scripts for clouds where we have userdata size constraints.

JIT runners allows us to avoid sending a runner registration token over the wire. Registration tokens can be used to register multiple runners and have a validity of one hour. A bad actor may intercept this token and use it to register their own runners. With JIT runners, only one runner may use the credentials at any given point in time. The credentials are fetched from GARM by the runner when it spins up.

This feature can be disabled in the provider config by specifying:

[[provider]]
disable_jit_config = true

easily access controller info

We now have the ability to gain more insight into how GARM is configured. We've added a new command which can show us some info about the controller:

ubuntu@garm:~$ garm-cli controller-info show
+------------------------+----------------------------------------------------------------------------+
| FIELD                  | VALUE                                                                      |
+------------------------+----------------------------------------------------------------------------+
| Controller ID          | a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3                                       |
| Hostname               | garm                                                                       |
| Metadata URL           | https://garm.example.com/api/v1/metadata                                   |
| Callback URL           | https://garm.example.com/api/v1/callbacks                                  |
| Webhook Base URL       | https://garm.example.com/webhooks                                          |
| Controller Webhook URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
+------------------------+----------------------------------------------------------------------------+

Easy webhook installation

Installing webhooks for entities (repos, orgs, enterprises) can be a chore. In this version, we've added the ability to let GARM install the needed webhook, with the correct settings by running a simple command. To make use of this feature, the PAT you're using must have access to admin:org_hook:

ubuntu@garm:~$ garm-cli org add \
    --credentials org_token \
    --name exampleOrg \
    --install-webhook  \
    --random-webhook-secret
+----------------------+--------------------------------------+
| FIELD                | VALUE                                |
+----------------------+--------------------------------------+
| ID                   | b90911e1-8727-4bb7-a1eb-96855d73a27b |
| Name                 | exampleOrg                           |
| Credentials          | org_token                            |
| Pool manager running | true                                 |
+----------------------+--------------------------------------+

This command will add the org exampleOrg to your GARM controller and install a webhook for it with a random webhook secret. You can also install a webhook for an existing organization or repository:

ubuntu@garm:~$ garm-cli repo webhook install b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD        | VALUE                                                                      |
+--------------+----------------------------------------------------------------------------+
| ID           | 449734752                                                                  |
| URL          | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
| Events       | [workflow_job]                                                             |
| Active       | true                                                                       |
| Insecure SSL | false                                                                      |
+--------------+----------------------------------------------------------------------------+

To view the status of a webhook:

ubuntu@garm:~$ garm-cli repo webhook show b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD        | VALUE                                                                      |
+--------------+----------------------------------------------------------------------------+
| ID           | 449734752                                                                  |
| URL          | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
| Events       | [workflow_job]                                                             |
| Active       | true                                                                       |
| Insecure SSL | false                                                                      |
+--------------+----------------------------------------------------------------------------+

You can also uninstall a webhook:

ubuntu@garm:~$ garm-cli repo webhook uninstall b90911e1-8727-4bb7-a1eb-96855d73a27b

Webhooks installed by the above commands are always namespaced to the controller ID you get when you run garm-cli controller-info show. This way, when we remove a webhook, we don't accidentally remove someone else's hook. You can also manually install we webhook just like before. You don't need to namespace it to the controller ID. GARM will see the webhook and let you know if the hook is already installed.

Removing a repo or org will also clean up the webhook if it was namespaced to our controller. You can opt to keep the hook by passing the --keep-webhook flag.

Webhook installation is not available for enterprises.

Structured logging

GARM has now switched to the slog standard package for structured logging. As part of this change, we now have a dedicated [logging] config section where you can set the log level, log format, enable the log streamer, etc. Check out the sample config for more info.

Other updates

There are a lot of bug fixes and stability updates that should make this version behave better in relation to the GitHub API.

What's Changed

• Update to latest version of garm-provider-common and update docs by @gabriel-samfira in #144
• Add GitHub workflow for integration tests by @mihaelabalutoiu in #145
• Add upload artifacts and log instance details on timeout exceeded by @mihaelabalutoiu in #146
• Add workflow dispatch trigger by @gabriel-samfira in #147
• Set pipefail to the script and log org/repo details on timeout exceeded by @mihaelabalutoiu in #148
• Cleaning up leftover runners for org/repo by @mihaelabalutoiu in #149
• Add controller info by @gabriel-samfira in #150
• Log orgPool/repoPool details on timeout exceeded by @mihaelabalutoiu in https://github...

garm v0.1.4-rc1

Welcome to GARM version v0.1.4-rc1!

This is a pre-release of GARM and it is packed with changes.

Before we dive into the highlights, there are a couple of things we need to mention.

Breaking change warning ⚠️

This version removes the LXD internal provider in favour of two new external providers:

• garm-provider-lxd
• garm-provider-incus

This was done mainly due to the licensing change of LXD. If you're using LXD and are upgrading from previous versions, you must download the LXD external provider and configure it.

Database upgrade ⚠️

This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes.

To safely create a backup of you DB, you can use the sqlite3 CLI:

# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.db

Highlights

This release is packed with optimizations, changes and a couple of nice features.

JIT runners

The main highlight of this release is the ability to use just-in-time self-hosted runners. To accommodate this change, GARM now has more robust metadata endpoints that can serve the needed JIT files. This will most likely be expanded in the future to include setup scripts for clouds where we have userdata size constraints.

JIT runners allows us to avoid sending a runner registration token over the wire. Registration tokens can be used to register multiple runners and have a validity of one hour. A bad actor may intercept this token and use it to register their own runners. With JIT runners, only one runner may use the credentials at any given point in time. The credentials are fetched from GARM by the runner when it spins up.

This feature can be disabled in the provider config by specifying:

[[provider]]
disable_jit_config = true

easily access controller info

We now have the ability to gain more insight into how GARM is configured. We've added a new command which can show us some info about the controller:

ubuntu@garm:~$ garm-cli controller-info show
+------------------------+----------------------------------------------------------------------------+
| FIELD                  | VALUE                                                                      |
+------------------------+----------------------------------------------------------------------------+
| Controller ID          | a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3                                       |
| Hostname               | garm                                                                       |
| Metadata URL           | https://garm.example.com/api/v1/metadata                                   |
| Callback URL           | https://garm.example.com/api/v1/callbacks                                  |
| Webhook Base URL       | https://garm.example.com/webhooks                                          |
| Controller Webhook URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
+------------------------+----------------------------------------------------------------------------+

Easy webhook installation

Installing webhooks for entities (repos, orgs, enterprises) can be a chore. In this version, we've added the ability to let GARM install the needed webhook, with the correct settings by running a simple command. To make use of this feature, the PAT you're using must have access to admin:org_hook:

ubuntu@garm:~$ garm-cli org add \
    --credentials org_token \
    --name exampleOrg \
    --install-webhook  \
    --random-webhook-secret
+----------------------+--------------------------------------+
| FIELD                | VALUE                                |
+----------------------+--------------------------------------+
| ID                   | b90911e1-8727-4bb7-a1eb-96855d73a27b |
| Name                 | exampleOrg                           |
| Credentials          | org_token                            |
| Pool manager running | true                                 |
+----------------------+--------------------------------------+

This command will add the org exampleOrg to your GARM controller and install a webhook for it with a random webhook secret. You can also install a webhook for an existing organization or repository:

ubuntu@garm:~$ garm-cli repo webhook install b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD        | VALUE                                                                      |
+--------------+----------------------------------------------------------------------------+
| ID           | 449734752                                                                  |
| URL          | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
| Events       | [workflow_job]                                                             |
| Active       | true                                                                       |
| Insecure SSL | false                                                                      |
+--------------+----------------------------------------------------------------------------+

To view the status of a webhook:

ubuntu@garm:~$ garm-cli repo webhook show b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD        | VALUE                                                                      |
+--------------+----------------------------------------------------------------------------+
| ID           | 449734752                                                                  |
| URL          | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3     |
| Events       | [workflow_job]                                                             |
| Active       | true                                                                       |
| Insecure SSL | false                                                                      |
+--------------+----------------------------------------------------------------------------+

You can also uninstall a webhook:

ubuntu@garm:~$ garm-cli repo webhook uninstall b90911e1-8727-4bb7-a1eb-96855d73a27b

Webhooks installed by the above commands are always namespaced to the controller ID you get when you run garm-cli controller-info show. This way, when we remove a webhook, we don't accidentally remove someone else's hook. You can also manually install we webhook just like before. You don't need to namespace it to the controller ID. GARM will see the webhook and let you know if the hook is already installed.

Removing a repo or org will also clean up the webhook if it was namespaced to our controller. You can opt to keep the hook by passing the --keep-webhook flag.

Webhook installation is not available for enterprises.

Other updates

There are a lot of bug fixes and stability updates that should make this version behave better in relation to the GitHub API. I'm pretty sure I'm forgetting something, but you can consult the list bellow for a full list of changes.

In any case, take it for a spin! If you run into any issue, feel free to open a new issue.

What's Changed

• Update to latest version of garm-provider-common and update docs by @gabriel-samfira in #144
• Add GitHub workflow for integration tests by @mihaelabalutoiu in #145
• Add upload artifacts and log instance details on timeout exceeded by @mihaelabalutoiu in #146
• Add workflow dispatch trigger by @gabriel-samfira in #147
• Set pipefail to the script and log org/repo details on timeout exceeded by @mihaelabalutoiu in #148
• Cleaning up leftover runners for org/repo by @mihaelabalutoiu in #149
• Add controller info by @gabriel-samfira in #150
• Log orgPool/repoPool details on timeout exceeded by @mihaelabalutoiu in #151
• Fix timeout logic by @mihaelabalutoiu in #152
• Small fixes for the docs by @mihaelabalutoiu in #153
• Enable Windows builds by @gabriel-samfira in #156
• Add webhook management for repositories and organizations by @gabriel-samfira in #154
• Fix e2e secrets generation by @ionutbalutoiu in #162
• Add webhooks integration tests for organization and repository by @mihaelabalutoiu in #161
• Use apg to generate passwords by @gabriel-samfira in #164
• Fix nil pointer dereference by @gabriel-samfira in #165
• Fix TLS client bug by @gabriel-samfira in #166
• Refactor integration E2E tests by @ionutbalutoiu in https://github.com/clo...

garm v0.1.3

Welcome to GARM version v0.1.3!

This release brings a number of really nice improvements.

Highlights

Thanks to @mihaelabalutoiu, we now have proper OpenAPI annotations, as well as a full client generated from them. The GARM CLI has switched to this new client, which is located in the root of the project. If you plan to integrate with GARM in any way, I encourage you to use this client instead of the old, hand-written one we previously had.

Another interesting change in this release is the fact that we split some functionality from GARM itself, into an external package. This new package will hold functionality that is common to GARM itself and external providers.

As part of this migration we also added the ability to:

• Completely override the runner installation script template. This can be done with any provider that leverages the new external package and calls the helper functions that generate the install script. This includes the current LXD, OpenStack and Azure providers.
• Send additional template context (if overriding the install template)
• For Linux only (for now):
  • Add pre-install scripts. These scripts are executed in alphabetical order before the runner install script is run.

This can all be done via extra-specs.

Upgrade notes

This update does not come with database schema changes, however, we do recommend you update any external providers to their latest versions if you update GARM.

What's Changed

• Fix apiserver/routers/routers.go typo by @mihaelabalutoiu in #131
• Add dockerfile and workflow by @gabriel-samfira in #132
• Properly set garm-cli version by @gabriel-samfira in #133
• Implement more features for the swagger client API by @mihaelabalutoiu in #134
• Add more functionality to swagger client library by @mihaelabalutoiu in #136
• Small fixes to the GARM api server by @ionutbalutoiu in #138
• Fix swagger annotations by @mihaelabalutoiu in #139
• Complete the swagger client library with full functionality by @mihaelabalutoiu in #140
• Update some docs by @gabriel-samfira in #129
• Remove unfinished doc by @gabriel-samfira in #141
• Move code to external package by @gabriel-samfira in #143
• Update garm-cli tool to use the new swagger generated client library by @mihaelabalutoiu in #142

Full Changelog: v0.1.2...v0.1.3

garm v0.1.2

Welcome to garm version v0.1.2!

This release brings a a lot of performance improvements and bug fixes.

Highlights

The most notable change in this release is in the way garm reacts to jobs sent by GitHub. Previously, garm reacted to new web hooks, but never recorded them in any way. When a queued hook came in, garm would attempt to create a new runner for it. The outcome of that action would depend on a few factors like if we had room for a new runner (max-runners), or if we had a pool that matched the requested tags.

One other shortcoming of the old approach was the fact that if we had multiple hierarchy levels configured (repo, org, enterprise), they would all receive the web hook with the queued job and would each create one runner for the same job. This had the potential to spin up resources for no reason.

This release introduces a new job tracking feature. From now on, garm will record all jobs that GitHub sends and the entity that received them (repo, org, enterprise). It will then attempt to periodically (roughly every 5 seconds) consume the jobs found in queued state and spin up runners for them. If we have multiple hierarchy levels configured, they will contend to spin up a runner for a job, but in the end only one of them will spin up a runner. This reduces the number of runners that get spun up for the same job. Job tracking also means that if a pool reaches max-runners we no longer lose jobs. We will eventually spin up a runner if the job is still in queued state. Which also means that it should be safe to use pools with min-idle-runners set to zero.

Other highlights

• WAL is now enabled for the DB store. This was an oversight that caused load on large scale deployments.
• Aditional indexes have been created for further speedups
• Pool manager loops have been refactored. Each operation now runs in their own loop.
• If multiple pools match a certain label, garm will now attempt to round-robin runners in matching pools

Upgrade notes

This release brings a number of changes to the database. Please back up your sqlite database before updating. The procedure is simple:

systemctl stop garm
cp /etc/garm/garm.db   /etc/garm/garm.db.backup
# update garm then start
systemctl start garm

What's Changed

• Add more runner/pools.go unit tests by @mihaelabalutoiu in #110
• Fix runner/pools.go typo by @mihaelabalutoiu in #111
• Refactor pool manager loop by @gabriel-samfira in #112
• Use su to install the runner by @gabriel-samfira in #115
• Rotate log file on SIGHUP by @ionutbalutoiu in #116
• Add logging and fix backoff loop by @gabriel-samfira in #118
• Generate client library via swagger by @mihaelabalutoiu in #119
• Add doc about performance considerations by @SystemKeeper in #120
• Replace wait implementation with errgroup by @gabriel-samfira in #121
• Add job tracking by @gabriel-samfira in #86
• Update comment on function by @gabriel-samfira in #122
• Add entity update subcommand by @gabriel-samfira in #125
• Fix entity update by @gabriel-samfira in #124
• Add more swagger client API implementation by @ionutbalutoiu in #126
• Add swagger client auth info by @ionutbalutoiu in #127
• Set on delete for jobs by @gabriel-samfira in #128

New Contributors

• @SystemKeeper made their first contribution in #120

Full Changelog: v0.1.1...v0.1.2

garm v0.1.1

Welcome to garm version v0.1.1!

This release adds some resiliency fixes to garm.

Highlights

• Instances that fail to set up their GitHub agent and transition to failed are now reaped.
• The cloud-init userdata now retries setting up the github agent for up to 5 times before giving up
• You can now specify extra_packages extra specs with the LXD provider, which allows you to install additional packages on runner nodes. (#103)

What's Changed

• Validate provider responses by @gabriel-samfira in #102
• Allow installing additional packages in lxd container by @HippocampusGirl in #103
• Add more users.go unit tests by @mihaelabalutoiu in #105
• Add test cases for the runner/pools.go by @mihaelabalutoiu in #107
• Add more test cases for the /database/sql/users.go by @mihaelabalutoiu in #108
• Reap failed agent by @gabriel-samfira in #106
• Fix runner/pools.go typo by @mihaelabalutoiu in #109

New Contributors

• @HippocampusGirl made their first contribution in #103

Full Changelog: v0.1.0...v0.1.1

garm v0.1.0

This is the first versioned release of garm.

Garm is still under development and may change drastically before it reaches a stable version, but we will always try to keep it as functional and as backwards compatible as possible.

What's Changed

• Fix typo by @ionutbalutoiu in #1
• Add azure external provider by @ionutbalutoiu in #2
• Support building with Podman instead of Docker by @pothos in #4
• Fix NewConfig func & and add more unit tests by @mihaelabalutoiu in #5
• Add unit tests by @mihaelabalutoiu in #6
• Refactor code to allow more unit testing by @mihaelabalutoiu in #7
• Add organizations.go unit tests and PoolManagerController mocks by @mihaelabalutoiu in #8
• Add go tests GitHub workflow by @ionutbalutoiu in #9
• Add repositories.go unit tests by @mihaelabalutoiu in #10
• Add more repositories.go unit testing by @mihaelabalutoiu in #11
• Update organizations.go unit tests by @mihaelabalutoiu in #13
• Add repositories.go unit tests by @mihaelabalutoiu in #12
• Add controller.go unit tests and Fix database.go typo by @mihaelabalutoiu in #14
• Cleanup duplicate code by @mihaelabalutoiu in #15
• Create a testing package for common utilities by @gabriel-samfira in #17
• Add instances.go unit tests by @mihaelabalutoiu in #16
• Cleanup unused struct field and Fix typo by @mihaelabalutoiu in #18
• Attempt to fetch runner name from API by @gabriel-samfira in #19
• Add link to Go Tests README badge by @ionutbalutoiu in #21
• Add more instances.go unit tests by @mihaelabalutoiu in #22
• Add instances.go database unit tests by @mihaelabalutoiu in #23
• Add organizations.go unit tests by @mihaelabalutoiu in #24
• Make owners and repos case insensitive by @gabriel-samfira in #26
• Add more organizations.go unit testing by @mihaelabalutoiu in #27
• feat: webhook does not return error on 'instance not found'. by @maigl in #28
• Fixed error wrap by @mihaelabalutoiu in #31
• Add organizations.go database unit testing by @mihaelabalutoiu in #34
• Add pools.go unit tests by @mihaelabalutoiu in #35
• Add pools.go database unit tests by @mihaelabalutoiu in #36
• Add repositories.go unit testing by @mihaelabalutoiu in #38
• Add more test cases for the database/sql/repositories.go by @mihaelabalutoiu in #39
• Don't remove foreign runners by @gabriel-samfira in #41
• Fixed error wrap by @mihaelabalutoiu in #42
• Add unit tests for the database/sql/repositories.go by @mihaelabalutoiu in #43
• Fix azure and openstack external providers by @ionutbalutoiu in #45
• Add log streamer by @gabriel-samfira in #46
• Add GitHub Enterprise support by @gabriel-samfira in #37
• Add a metadata resource for instances by @gabriel-samfira in #49
• Wait for loop exit and some fixes by @gabriel-samfira in #51
• Slightly better error handling by @gabriel-samfira in #52
• Add sql package enterprises tests by @gabriel-samfira in #53
• Update sample bootstrap params by @gabriel-samfira in #54
• Ensure loop closes properly and provider update by @gabriel-samfira in #55
• feat: scale down idle runners by @maigl in #56
• feat: allow to configure the runner name by @maigl in #57
• fix: shortid generates correctly now by @maigl in #62
• Add a common RunnerPrefix type by @gabriel-samfira in #63
• Move pool type in params by @gabriel-samfira in #64
• Markdown lint by @icadariu in #65
• Fix linting errors and add workflow to check by @gabriel-samfira in #66
• Disable Windows binary build by @gabriel-samfira in #67
• Sanitize log entries by @gabriel-samfira in #68
• Fix webhook signature validation by @matoubidou in #61
• feat: add prometheus metrics & endpoint by @maigl in #60
• fix: don't update empty runner prefix by @maigl in #69
• Move some code around by @gabriel-samfira in #70
• Add extra specs on pools by @gabriel-samfira in #71
• Fix TLS listener by @gabriel-samfira in #72
• fix: skip spawn new runners if enough idle runner available by @maigl in #73
• Fix ListInstances in external provider examples by @gabriel-samfira in #75
• Bail if we fail to cleanup failed instance by @gabriel-samfira in #77
• Add grace period to scale-down by @gabriel-samfira in #78
• Rename module by @gabriel-samfira in #80
• Move some defaults and types from config by @gabriel-samfira in #81
• Add aditional external provider enablement by @gabriel-samfira in #82
• check if runner needs to be installed by @maigl in #84
• Add Userdata options to disable package update during boot / cloud-init by @maigl in #87
• Fix pool ID check when listing instances by @gabriel-samfira in #92
• Let CreateInstance download and cache image by @gabriel-samfira in #94
• Allow disabling updates by @gabriel-samfira in #95
• Fix Makefile typo by @ionutbalutoiu in #99
• Add users.go unit tests by @mihaelabalutoiu in #98
• Parallelization and LXD timeouts by @gabriel-samfira in #97
• Don't merge the second column in the CLI by @gabriel-samfira in #101

New Contributors

• @ionutbalutoiu made their first contribution in #1
• @pothos made their first contribution in #4
• @mihaelabalutoiu made their first contribution in #5
• @maigl made their first contribution in #28
• @icadariu made their first contribution in #65
• @matoubidou made their first contribution in #61

Full Changelog: https://github.com/cloudbase/garm/commits/v0.1.0