Skip to content

gobench-io/gobench

Repository files navigation

A benchmark framework with Golang.

Note: Gobench is under heavy development.

Targets:

  1. Supporting more than HTTP. We are having MQTT, NATs. Websocket, graphQL will be added.
  2. Having complicated benchmarking scenario in Golang. Yo no DSL, and Golang is easy to pickup.
  3. (WIP) To support more than one million connection concurrently.

Build Status codecov

Usage

Install from source

Requirements:

  • golang to compile your scenario
  • gcc to build sqlite3 embedded database

Install the command line tool first

For Go 1.17+

go install github.com/gobench-io/gobench@master

For Go 1.16 and below

go get github.com/gobench-io/gobench

This mechanism will install a build of master at $GOPATH/bin. To test your installation:

$ gobench
{"level":"info","ts":1601432777.7513623,"caller":"master/master.go:71","msg":"new master program","port":8080,"home directory":"/home/nqd/.gobench"}
{"level":"info","ts":1601432777.9341393,"caller":"web/web.go:161","msg":"web server start","port":":8080"}

After that, open http://localhost:8080 to see the dashboard.

Run from docker

docker run -p 8080:8080 nqdinh/gobench:latest

After that, open http://localhost:8080 to see the dashboard.

To setup admin password, add --admin-password to argument. To save the results to the host machine, mount /root/.gobench (the default --dir) to a host path. For example, save the result data to /tmp/abc and set admin password to supertest with the following command:

docker run -p 8080:8080 -v "/tmp/abc:/root/.gobench" nqdinh/gobench:latest --admin-password supertest

Quick start

Start the Gobench server, go to http://localhost:8080 dashboard, create new application.

Input scenario as the following go code. gomod and gosum are used to control specific version of dependency packages. We will leave them empty now.

// Test a server running on a local machine on port 8080.
// Send 10 requests per second for 2 minute from 5 nodes in parallel,
// which totals up to 50 requests per second altogether.

package main

import (
    "context"
    "log"
    "time"

    httpClient "github.com/gobench-io/gobench/clients/http"
    "github.com/gobench-io/gobench/dis"
    "github.com/gobench-io/gobench/executor/scenario"
)

func export() scenario.Vus {
    return scenario.Vus{
        {
            Nu:   5,
            Rate: 1000,
            Fu:   f,
        },
    }
}

func f(ctx context.Context, vui int) {
    client, err := httpClient.NewHttpClient(ctx, "home")
    if err != nil {
        log.Println("create new client fail: " + err.Error())
        return
    }

    url1 := "http://localhost:8080"

    timeout := time.After(2 * time.Minute)

    for {
        select {
        case <-timeout:
            return
        default:
            go client.Get(ctx, url1, nil)
            dis.SleepRatePoisson(10)
        }
    }
}

From the dashboard, you will see the live result:

You also see the status of the host running Gobench: Load average, CPU utilization, RAM usage, network in/out.

How to write scenario

Scenario is a go file that must have a func export() scenario.Vus {...} function. This function return an array of scenario.Vu struct.

Each scenario.Vu defines behavior of a type of virtual user (vu). In previous example, the vu is defines as

{
    Nu:   5,
    Rate: 1000,
    Fu:   f,
}

, on which:

  • Nu defines the number of virtual users for this type of user
  • Rate is the startup rate for all virtual users with Poisson distribution. In this case 1000 virtual users are created in one second.
  • Fu defines the behavior of a virtual user. Fu must be define as func f(ctx context.Context, vui int) {...}.

When your benchmark scenario is more complecated, you can define multiple virtual user types

func export() scenario.Vus {
    return scenario.Vus{
        {
            Nu:   5,
            Rate: 1000,
            Fu:   adminF,
        },
        {
            Nu:   7,
            Rate: 1000,
            Fu:   userF,
        },
    }
}

How to write a new worker

Gobench is supporting 3 clients: HTTP, MQTT, NATs. Creating a new type of worker for Gobench is very simple. The worker has to have the following properties.

Expose the metrics

Exposes to gobench via executor.Setup(groups) calling where groups is []metrics.Group{} structure.

For convenience, one should call the metrics setup at the end of constructor like NewHttpClient on which calling executor.Setup.

Gobench strictly force you to create the metrics hierarchy. Group name (Group.Name) must be unique globally. Also metric title (Metric.Title) must be unique globally.

Gobench is supporting 3 kinds of metric: counter, histogram, and gauge.

Notify the metric

Notify to gobench via executor.Notify(metric name, value).

See clients/http for HTTP worker example.

Benchmark the benchmark

Right, the benchmark tool should show the ability to generate a good amount of traffic given RAM, CPU resources. For HTTP, we compare Gobench with k6 (v0.29.0), hey (v0.1.4), Artillery (v1.6.1), Jmeter (v5.2.1). For MQTT, we compare Gobench with eqmtt_bench.

We benchmark the benchmarks with two c5.4xlarge (16 core CPU, 32 GB RAM) VM. For HTTP case, the server is the high speed Nginx. Each client gets the main Nginx page with vu = 30 during 120 seconds.

HTTP client CPU (%) RAM (MB) RPS
hey 218 150 27220
k6 342 556.3 26314
Jmeter 199 814.6 26259
Gobench 281 114.5 25686
Artillery 107 145.3 1197

For MQTT case, the server is Vernemq. Each client run two scenarios: (1) Just create n clients that to Vernemq with 10 seconds heartbeat, and (2) create n clients, each publishes 1 message per second to topic bench/$i where i is the client id. We choose n = 250, 11250, 25000.

MQTT client CPU (%) RAM (MB)
Gobench 250 conn 12 225
Gobench 11250 conn 31 536.5
Gobench 25000 conn 52 1024
emqtt_bench 250 conn 12 147
emqtt_bench 11250 conn 37 643
emqtt_bench 25000 conn 49 1485
Gobench 250 pub 85 267.3
Gobench 11250 pub 286 741.9
Gobench 25000 pub 669 1450.0
emqtt_bench 250 pub 52 252
emqtt_bench 11250 pub 297 995
emqtt_bench 25000 pub 850 2847

Sponsor