The amount of time and effort a developer dedicates towards writing a function depends on the details they choose to focus on: coding conventions, structure, programming style, etc. Suppose a group of developers is presented a high-level prompt to write the same function: given some input, return some output. For example, given a list of numbers, return a sorted list of numbers. The actual implementation of the function is left entirely to the discretion of the developer. A quick, mathematical way to evaluate each developer's implementation of this function, without any additional code, is by its time complexity. Particularly, knowing each implementation's Big-O complexity tells us how it might perform in the worst case scenario, commonly when the size of the input is very large.

However, time complexity fails to account for the hardware the function is executed upon, and it does not provide any tangible, quantifiable metrics to base decisions on. Metrics such as operation speed and total execution time assign real numerical values to the performance of a function. By adding benchmarks, developers can leverage these metrics to better inform them on how to improve their code. The Go programming language has a benchmarking utility in its built-in, standard library package testing. To benchmark code in Go, define a function with a name prefixed with Benchmark (followed by a capitalized segment of text) and accepts an argument of struct type B, which contains methods and values for determining the number of iterations to run, running multiple benchmarks in parallel, timing execution times, etc.

Example:

Benchmarking Simple Functions

Note: The structure of a benchmark is similar to the structure of a test. Replace Test with Benchmark and t *testing.T with b *testing.B.

This benchmark runs the Sum function for b.N iterations. Here, the benchmark runs for one billion iterations, which allows the benchmark function to reliably time and record each execution. Once the benchmark is completed, the results of this benchmark and the CPU of the machine running this benchmark are outputted to the terminal. On average, each iteration ran 0.6199 ns.

Below, I'm going to show you...

  • How to benchmark a route handler of a Go and chi RESTful API.

  • How to run benchmarks in parallel.

Installation and Setup#

Clone a copy of the Go and chi RESTful API from GitHub to you machine:

This RESTful API specifies five endpoints for performing operations on posts:

  • GET /posts - Retrieve a list of posts.

  • POST /posts - Creates a post.

  • GET /posts/{id} - Retrieve a single post identified by its id.

  • PUT /posts/{id} - Update a single post identified by its id.

  • DELETE /posts/{id} - Delete a single post identified by its id.

If you would like to learn how to build this RESTful API, then please visit this blog post. In the basic-tests branch, a simple unit test is already provided in the routes/posts_test.go file. Because benchmarks must be placed in _tests.go files, let's place the benchmarks for the posts sub-router in the routes/posts_test.go file.

Run the following command to install the project's dependencies:

Note: If you run into installation issues, then verify that the version of Go running on your machine is v1.16.

Run the following command to execute the unit tests within the routes/posts_test.go file:

Writing a Benchmark for a Route Handler#

To get started, open the routes/posts_test.go file. Let's name the benchmark function BenchmarkGetPostsHandler:

  1. Define a sub-test/sub-benchmark with a unique name via b.Run. Using b.Run allows benchmarks to be grouped by the functionality it's testing, and benchmarks can be structured hierarchically with nested benchmarks. Also, in the benchmark's output, Go prints this unique name next to the benchmark's name.go b.Run("Endpoint: GET /posts", func(b *testing.B) { // ... }

  2. To avoid sending network requests during benchmarking, mock out the GetPosts package-scoped variable. go GetPosts = (&JsonPlaceholderMock{}).GetPosts

  3. Create a new GET request to send to /posts via the http package's NewRequest method. This request will be passed to the route handler. Since it is not used in subsequent statements in this benchmark, the request error is set to an underscore.go r, _ := http.NewRequest("GET", "/posts", nil)

  4. Create a response recorder via httptest.NewRecorder to record the mutations of ResponseWriter, which is passed as an argument to the route handler.go w := httptest.NewRecorder()

  5. In the context of this benchmark, the PostsResource{}.List method is an ordinary function. Pass it to the http package's HandlerFunc method to treat it as an HTTP route handler. Since HTTP route handlers have a function signature with the parameters ResponseWriter and Request, and PostsResource{}.List has this same function signature, calling the HTTP route handler will call the PostsResource{}.List method.go handler := http.HandlerFunc(PostsResource{}.List)

  6. Turn on malloc statistics via b.ReportAllocs, which displays an additional two columns in the output of the benchmark. These columns tell how many bytes of memory, on average, were allocated per iteration and how many allocations, on average, were performed per iteration. Having this statement is the same as setting the -benchmem for the go test command.go b.ReportAllocs()

  7. Cut out the expensive setup time from the reported execution time via b.ResetTimer.go b.ResetTimer()

  8. The handler.ServeHTTP method executes the HTTP route handler representation of PostsResource{}.List using the response recorder w and the created request r. Loop over handler.ServeHTTP until the benchmark determines how fast it runs. The value of b.N changes until the benchmark stabilizes and knows how long it must run to properly time the function. go for i := 0; i < b.N; i++ { handler.ServeHTTP(w, r) }

Combine the code snippets together:

(routes/posts_test.go)

Run the benchmark:

  • -bench=. - Tells go test to run benchmarks and tests within the project's _test.go files. . is a regular expression that tells go test to match with everything.

  • ./routes - Tells go test the location of the _test.go files with the benchmarks and tests to run.

  • -run=^$ - Tells go test to run tests with names that satisfy the regular expression ^$. Since none of the tests' names begin with $, go test will not run any tests and will only run benchmarks.

Route Handler Benchmark

Here, benchmarks run sequentially.

The benchmark ran a total of 97220 iterations with each iteration running, on average, 11439 ns. This represents the average time it took for each handler.ServeHTTP function (and by extension, PostsResource{}.List) call to complete. Each iteration involved the allocation of, on average, 33299 bytes of memory. Memory was allocated, on average, 8 times per iteration.

Running Benchmarks in Parallel#

The for loop forces the benchmark to execute the function handler.ServeHTTP sequentially, one after the other. By running the benchmark with b.RunParallel, the total iterations b.N is divided amongst the machine's available threads (distributed amongst multiple goroutines). Having these iterations run concurrently helps to benchmark code that's inherently concurrent, such as sending requests and receiving responses, and deals with mutexes and/or shared resources.

To parallelize benchmarks, wrap the benchmark code in b.RunParallel and replace the for loop with for pb.Next():

(routes/posts_test.go)

Run the benchmark again.

Parallel Benchmark

Notice that the benchmark values are similar to when we ran the benchmark sequentially with a for loop.

To increase the number of cores (and goroutines) to run this benchmark against, add the cpu flag:

Parallel Benchmark - Multiple Cores

Increasing the number of cores increases the number of goroutines running the benchmark iterations, which results in better performance.

Next Steps#

Click here for a final version of the route handler unit test.

Try writing benchmarks for the other route handlers.

Sources#