Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Detecting Performance Regressions

With Gungraun you can define limits for each Callgrind/Cachegrind event kind or DHAT metric over which a performance regression can be assumed. Per default, Gungraun does not perform regression checks, and you have to opt-in with Callgrind::soft_limits, Callgrind::hard_limits, Cachegrind::soft_limits, … at benchmark level in LibraryBenchmarkConfig::tool or BinaryBenchmarkConfig::tool or at a more global level with Command-line arguments or Environment variables, see below.

For a soft limit, a performance regression check consists of an EventKind, CachegrindMetric or DhatMetric and a percentage. If the percentage is negative, then a regression is assumed to be below this limit. Hard limits restrict the EventKind, … by an absolute number.

Perf limits, in contrast, are not defined over an enum event kind but match the names of the reported perf metrics with wildcard patterns, see below.

Note that comparing baselines also detects performance regressions. This can be useful, for example, when setting up Gungraun in the CI to cause a PR to fail when comparing to the main branch.

Regressions are considered errors and will cause the benchmark to fail if they occur, and Gungraun will exit with error code 3.

Defining Limits on the Command-Line

Limits can be defined on the command-line for the following tools with --callgrind-limits (GUNGRAUN_CALLGRIND_LIMITS), --cachegrind-limits (GUNGRAUN_CACHEGRIND_LIMITS), --dhat-limits (GUNGRAUN_DHAT_LIMITS) and --perf-limits (GUNGRAUN_PERF_LIMITS). Command-line limits overwrite the limits specified in the benchmark file (see below).

In order to disambiguate between soft and hard limits, soft limits have to be suffixed with a %. Hard limits are bare numbers. For example to limit the total instructions executed ir (printed as Instructions in the Callgrind terminal output) to 5%:

cargo bench --bench gungraun_benchmark -- --callgrind-limits='ir=5%'

These command-line arguments and environment variables can be used to define soft limits and hard limits in one go with the |-operator (e.g. --callgrind-limits='ir=5%|10000') or multiple limits at once separated by a , (e.g. --callgrind-limits='ir=5%|10000,totalrw=2%').

For a list of all allowed Callgrind metrics (like ir) see the docs of EventKind, for Cachegrind metrics CachegrindMetric and for DHAT metrics DhatMetric. It is sometimes more convenient to define limits for whole groups with the @-operator: --callgrind-limits='@all=5%'. All allowed groups and their members for Callgrind metrics can be found in CallgrindMetrics, for Cachegrind metrics in CachegrindMetrics and DHAT metrics in DhatMetrics.

Multiple specifications of the same EventKind, … overwrite the previous one until the last one wins. This is useful for example to specify a limit for all event kinds and then overwrite the limit for a specific event kind: --callgrind-limits='@all=10%,ir=5%'

The Format, Short Names and Groups in Full Detail

For --callgrind-limits:

arg        ::= pair ("," pair)*
pair       ::= key "=" value ("|" value)*
key        ::= group | event         ; matched case-insensitive
group      ::= "@" ( "default"
                   | "all"
                   | ("cachemisses" | "misses" | "ms")
                   | ("cachemissrates" | "missrates" | "mr")
                   | ("cachehits" | "hits" | "hs")
                   | ("cachehitrates" | "hitrates" | "hr")
                   | ("cachesim" | "cs")
                   | ("cacheuse" | "cu")
                   | ("systemcalls" | "syscalls" | "sc")
                   | ("branchsim" | "bs")
                   | ("writebackbehaviour" | "writeback" | "wb")
                   )
event      ::= EventKind
value      ::= soft_limit | hard_limit
soft_limit ::= (integer | float) "%" ; can be negative
hard_limit ::= (integer | float)     ; float is only allowed for EventKinds which are
                                   ; float like `L1HitRate` but not `L1Hits`

with:

  • Groups with a long name have their allowed abbreviations placed in the same parentheses.
  • EventKind is the exact name of the enum variant (case insensitive)
  • integer is a u64 and float is a f64

For --cachegrind-limits replace the group and event from above with:

group ::= "@" ( "default"
              | "all"
              | ("cachemisses" | "misses" | "ms")
              | ("cachemissrates" | "missrates" | "mr")
              | ("cachehits" | "hits" | "hs")
              | ("cachehitrates" | "hitrates" | "hr")
              | ("cachesim" | "cs")
              | ("branchsim" | "bs")
              )

event ::= CachegrindMetric

For --dhat-limits replace the group and event from above with:

group ::= "@" ( "default" | "all" )
event ::= ( "totalunits" | "tun" )
          | ( "totalevents" | "tev" )
          | ( "totalbytes" | "tb" )
          | ( "totalblocks" | "tbk" )
          | ( "attgmaxbytes" | "gb" )
          | ( "attgmaxblocks" | "gbk" )
          | ( "attendbytes" | "eb" )
          | ( "attendblocks" | "ebk" )
          | ( "readsbytes" | "rb" )
          | ( "writesbytes" | "wb" )
          | ( "totallifetimes" | "tl" )
          | ( "maximumbytes" | "mb" )
          | ( "maximumblocks" | "mbk" )

For --perf-limits:

arg          ::= pair ("," pair)*
pair         ::= glob_pattern "=" value ("|" value)*
glob_pattern ::= glob_pattern ; see the bullet list below for details
value        ::= soft_limit | hard_limit
soft_limit   ::= (integer | float) "%" ; can be negative
hard_limit   ::= (integer | float) [ unit ]
unit         ::= "ms" | "s" | "B" | "Hz" | ... ; this is a non-exhaustive list

with:

  • A glob_pattern is matched against the name of a reported perf metric, where : and / are interchangeable: task-clock/u and task-clock:u address the same metric.
  • In a glob_pattern, * matches any sequence of characters, ? a single character, [...] a character class (negated as [!...], with ranges like [a-zA-Z]) and \ escapes a special character.
  • A soft_limit is a percentage and a hard_limit is an absolute number with an optional unit to which the metric value is normalized; a soft and a hard limit can be combined for one pattern with the |-operator.
  • Examples are *instructions*=5%, task-clock*=40%|2.5ms and task-clock/u=5% (equivalent: task-clock:u=5%).
  • The unit of a metric is shown next to the perf metric in the terminal output.

In contrast to the Valgrind tools, soft limits on perf metrics are only reported if the change between the old and the new value is statistically significant at the alpha level of the perf configuration.

Define a Performance Regression Check in a Benchmark

For example, in a Library Benchmark, define a soft limit of +5% for the Ir event kind for all benchmarks of this file:

extern crate gungraun;
mod my_lib { pub fn bubble_sort(_: Vec<i32>) -> Vec<i32> { vec![] } }
use gungraun::prelude::*;
use gungraun::{Callgrind, EventKind};
use std::hint::black_box;

#[library_benchmark]
#[bench::worst_case(vec![3, 2, 1])]
fn bench_library(data: Vec<i32>) -> Vec<i32> {
    black_box(my_lib::bubble_sort(black_box(data)))
}

library_benchmark_group!(name = my_group, benchmarks = bench_library);

fn main() {
main!(
    config = LibraryBenchmarkConfig::default()
        .tool(Callgrind::default()
            .soft_limits([(EventKind::Ir, 5.0)])
        ),
    library_benchmark_groups = my_group
);
}

Now, if the comparison of the Ir events of the current bench_library benchmark run with the previous run results in an increase of over 5%, the benchmark fails. Running the benchmark from above the first time results in the following output:

lib_bench_regression::my_group::bench_library worst_case:vec! [3, 2, 1]
  Instructions:                         152|N/A                  (*********)
  L1 Hits:                              201|N/A                  (*********)
  LL Hits:                                0|N/A                  (*********)
  RAM Hits:                               5|N/A                  (*********)
  Total read+write:                     206|N/A                  (*********)
  Estimated Cycles:                     376|N/A                  (*********)

Gungraun result: Ok. 1 without regressions; 0 regressed; 0 filtered; 1 benchmarks finished in 0.14477s

Assume there is a change in my_lib::bubble_sort with a negative impact on the performance, then running the benchmark again results in an output something similar to this:

lib_bench_regression::my_group::bench_library worst_case:vec! [3, 2, 1]
  Instructions:                         264|152                  (+73.6842%) [+1.73684x]
  L1 Hits:                              341|201                  (+69.6517%) [+1.69652x]
  LL Hits:                                0|0                    (No change)
  RAM Hits:                               6|5                    (+20.0000%) [+1.20000x]
  Total read+write:                     347|206                  (+68.4466%) [+1.68447x]
  Estimated Cycles:                     551|376                  (+46.5426%) [+1.46543x]
Performance has regressed: Instructions (152 -> 264) regressed by +73.6842% (>+5.00000%)

Regressions:

  lib_bench_regression::my_group::bench_library:
    Instructions (152 -> 264): +73.6842% exceeds limit of +5.00000%

Gungraun result: Regressed. 0 without regressions; 1 regressed; 0 filtered; 1 benchmarks finished in 0.14849s
error: bench failed, to rerun pass `-p gungraun-tests --bench lib_bench_regression`

Caused by:
  process didn't exit successfully: `/home/lenny/workspace/programming/gungraun/target/release/deps/lib_bench_regression-98382b533bca8f56 --bench` (exit status: 3)

Which Event to Choose to Measure Performance Regressions?

For Callgrind/Cachegrind and if in doubt, the answer is Ir (instructions executed). If Ir event counts decrease noticeable the function (binary) runs faster. The inverse statement is also true: If the Ir counts increase noticeable, there’s a slowdown of the function (binary).

These statements are not so easy to transfer to Estimated Cycles, cache metrics and most of the other event counts. But, depending on the scenario and the function (binary) under test, it can be reasonable to define more regression checks.

Who Actually Uses Instructions to Measure Performance?

The ones known to the author of this humble guide are

  • SQLite: They use mainly cpu instructions to measure performance improvements (and regressions).
  • Also in benchmarks of the rustc compiler and compiler-builtins, instruction counts play a great role. But, they also use cache metrics and cycles.
  • SpacetimeDB

If you know of others, please feel free to add them to this list.