Comparing with baselines

Usually, two consecutive benchmark runs let Gungraun compare these two runs. It's sometimes desirable to compare the current benchmark run against a static reference, instead. For example, if you're working longer on the implementation of a feature, you may wish to compare against a baseline from another branch or the commit from which you started off hacking on your new feature to make sure you haven't introduced performance regressions. Gungraun offers such custom baselines. If you are familiar with criterion.rs, the following command line arguments should also be very familiar to you:

  • --save-baseline=NAME (env: GUNGRAUN_SAVE_BASELINE): Compare against the NAME baseline if present and then overwrite it.
  • --baseline=NAME (env: GUNGRAUN_BASELINE): Compare against the NAME baseline without overwriting it
  • --load-baseline=NAME (env: GUNGRAUN_LOAD_BASELINE): Load the NAME baseline as the new data set instead of creating a new one. This option needs also --baseline=NAME to be present.

If NAME is not present, NAME defaults to default.

For example to create a static reference from the main branch and compare it:

git checkout main
cargo bench --bench <benchmark> -- --save-baseline=main
git checkout feature
# ... HACK ... HACK
cargo bench --bench <benchmark> -- --baseline main

Sticking to the above execution sequence,

cargo bench --bench my_benchmark -- --save-baseline=main

prints something like that with an additional line Baselines in the output.

my_benchmark::my_group::bench_library
  Baselines:                   main|main
  Instructions:                 280|N/A             (*********)
  L1 Hits:                      374|N/A             (*********)
  LL Hits:                        1|N/A             (*********)
  RAM Hits:                       6|N/A             (*********)
  Total read+write:             381|N/A             (*********)
  Estimated Cycles:             589|N/A             (*********)

Gungraun result: Ok. 1 without regressions; 0 regressed; 0 filtered; 1 benchmarks finished in 0.49333s

After you've made some changes to your code, running

cargo bench --bench my_benchmark -- --baseline=main`

prints something like the following:

my_benchmark::my_group::bench_library
  Baselines:                       |main
  Instructions:                 214|280             (-23.5714%) [-1.30841x]
  L1 Hits:                      287|374             (-23.2620%) [-1.30314x]
  LL Hits:                        1|1               (No change)
  RAM Hits:                       6|6               (No change)
  Total read+write:             294|381             (-22.8346%) [-1.29592x]
  Estimated Cycles:             502|589             (-14.7708%) [-1.17331x]

Gungraun result: Ok. 1 without regressions; 0 regressed; 0 filtered; 1 benchmarks finished in 0.49333s