Allow choice of reference benchmark

[christianhujer]

When running hyperfine, we get a nice output.

For example, when running hyperfine like this:

hyperfine -L shell sh,ash,ksh,bash,zsh,csh,fish,tcsh,nu,pwsh '{shell} -c "echo \"Hello, world!\""'

The Summary of the output looks like this:

Summary
  'sh -c "echo \"Hello, world!\""' ran
    1.23 ± 0.73 times faster than 'ash -c "echo \"Hello, world!\""'
    1.90 ± 0.45 times faster than 'ksh -c "echo \"Hello, world!\""'
    2.03 ± 1.18 times faster than 'bash -c "echo \"Hello, world!\""'
    2.41 ± 0.55 times faster than 'zsh -c "echo \"Hello, world!\""'
    6.48 ± 1.29 times faster than 'csh -c "echo \"Hello, world!\""'
    7.26 ± 1.62 times faster than 'fish -c "echo \"Hello, world!\""'
    7.93 ± 1.68 times faster than 'tcsh -c "echo \"Hello, world!\""'
   26.55 ± 5.52 times faster than 'nu -c "echo \"Hello, world!\""'
  674.56 ± 131.91 times faster than 'pwsh -c "echo \"Hello, world!\""'

It would be really great if it were possible to pick one of the benchmark runs as "reference". For example, if I'm interested in how nu performs in comparison, I would like it to be on top, and the other benchmarks should be listed as times faster or times slower, depending on their performance.

In this example, when I run it like this:

hyperfine -L shell sh,ash,ksh,bash,zsh,csh,fish,tcsh,nu,pwsh --reference nu '{shell} -c "echo \"Hello, world!\""'

Then the Summary should look like this:
Summary

  'nu -c "echo \"Hello, world!\""' ran
    26.55 ± 5.52 times slower than 'sh -c "echo \"Hello, world!\""'
    21.58 ± 7.56 times slower than 'ash -c "echo \"Hello, world!\""'
    13.97 ± 12.27 times slower than 'ksh -c "echo \"Hello, world!\""'
    13.08 ± 4.68 times slower than 'bash -c "echo \"Hello, world!\""'
    11.02 ± 10.04 times slower than 'zsh -c "echo \"Hello, world!\""'
    4.10 ± 4.27 times slower than 'csh -c "echo \"Hello, world!\""'
    3.66 ± 3.40 times slower than 'fish -c "echo \"Hello, world!\""'
    3.35 ± 3.29 times slower than 'tcsh -c "echo \"Hello, world!\""'
   25.41 ± 23.90 times faster than 'pwsh -c "echo \"Hello, world!\""'

[sharkdp]

That sounds like a very useful feature request, thank you.

How would this work in cases where we do not have a -L option, but rather multiple commands? would the reference be a number? or the full command?
It would be great if we could come up with a proper proposal for the complete command-line interface. And evaluate how this might interfere/combine with other command line options of hyperfine.

[Animeshz]

I believe --reference should always contain a complete command (and separate from normal run parameters, notice no nu in -L shell):

hyperfine --reference 'nu -c "echo \"Hello, world!\"' -L shell sh,ash,ksh,bash,zsh,csh,fish,tcsh,pwsh '{shell} -c "echo \"Hello, world!\"'

This way we can also measure it against not necessarily same programs.

[sharkdp]

@christianhujer what do you think?

[Julian]

I'm finding this issue while searching for a related one -- going to mention that here since it may be relevant to designing --reference, though of course the addition may be deemed out of scope:

But in my case, I want a reference command per parameter I think. Imagine the use case being if I were similarly testing 10 shells running some command, and I want to measure how slowly running the command is relative to a reference point, where the reference point is each shell's own startup time, and therefore basically normalize each shell by how long it took to start up, thereby measuring just the command execution.

(Of course I'd separately want to measure them against each other in absolute value, but sometimes I care just about the runtime once the shell is up. And --setup for what it's worth isn't really appropriate either in my real case, I want to indeed measure the two commands).

But yeah -- is there any thoughts on whether letting --reference itself include the parameter from -L?