Scalable Continuous Integration (CI) Patterns

[djdefi]

Continuous Integration is a real game-changer for development teams. The ability to automatically test and compile code on push drastically shortens the time that it takes to verify that what you're developing is doing what it's supposed to. For a small team building a simple project, CI is easy to set up. However, as you add more developers and projects, and as those projects increase in complexity, the performance impact of connecting your CI system to GitHub Enterprise can grow significantly, affecting everyone on your team.

It doesn't have to be this way. Here, we'll show you one way in which you can lower the performance impact of setting up CI for your projects.

Webhooks vs polling

What is polling? Why do people use it and what are its downsides?

Polling is when a script or application makes a request for a resource at a regular interval. For instance, a build job can query GitHub every minute for the latest commit SHA. Polling can be requesting repository data via Git or as a GitHub REST API request to compare remote information to local data.

An advantage to polling is that it can provide resilience from system or network outages that can result in missed events such as build or deployment jobs. Interval-based polling is often an option with continuous integration build environments where webhooks are not available.

The downside to polling is that in most cases, most of the requests are redundant. For instance, if you have a build job that polls for the latest commit SHA every minute, it's likely that the commit SHA hasn't changed in that time. This means that the build job is "wasting" resources by making a request that will return the same information as the previous request. This raises the baseline consumption of resources on the GitHub Enterprise appliance, which can impact the performance of other users on the system if the polling is excessive.

What are webhooks and how do they work?

A webhook is a way to deliver real-time data to applications. Unlike traditional APIs where you need to poll for data frequently to get quasi real-time information, webhooks are triggers that send data when specific events occur.

You can think about webhooks like push notifications on your mobile phone. Rather than burning up the battery on your phone fetching information (polling) from applications to get updates, push notifications - or webhooks - automatically send data based on event triggers. And, like push notifications, webhooks are less resource-intensive.

For more information about webhooks, check out the GitHub Developer guide on webhooks.

Why use webhooks instead of polling an API?

Webhooks are far more efficient than polling, from a resource and communication standpoint. Webhooks are like push notifications, they send data when something has changed or when a specific event occurs.

Data is always old. The very nature of webhooks and the fact that they are typically event-triggered means they are providing you with near real-time information. Due to this, if you want information as close to real-time as possible, you should elect to use webhooks over polling.

Webhooks are superior to polling in terms of freshness of data, efficiency of communication, and infrastructure costs.

Getting started with webhooks

You can configure webhooks directly in your repository or organization settings:

Once you've configured a hook, the new deliveries section helps you track, troubleshoot, and re-send a webhook event payload:

Other strategies and workarounds

When possible, using webhooks is always the best option for preventing performance problems when implementing CI with GitHub Enterprise. However, sometimes that's not immediately possible if you have a lot of users using polling currently and sometimes switching to using webhooks isn't enough. You can use the following strategies as needed to help mitigate performance issues while you move to a webhooks-based CI infrastructure.

Application optimization

Applications should cache non-critical API resources and use conditional requests. The downside to this is that even though this is available, the entire response has to be computed before we know it hasn't changed. That limits how useful this is for performance improvements.

CI build optimization

Another way to improve the performance of your CI systems is to improve the optimization of the CI systems themselves. You should audit build systems to eliminate redundant build jobs, copy artifacts between build jobs, and spread activity over time to avoid spikes. For more information, see other techniques to speed up builds.

Continue the conversation

What CI patterns have worked for you in the past? What other challenges have you faced in implementing CI at scale? Let us know in the comments below!

Further learning

Interested in learning more and trying some of what you learn in a hands-on way? Check out the GitHub guide on Building a CI Server. In this guide, you'll learn how to build a simple CI server that uses GitHub Webhooks to trigger a Status Check when a commit is pushed to a repository.