Feature request: Remote commands

[Walther]

Use case: sometimes it is nice to run some benchmarks on different hosts, to compare performance across various configurations - amount of resources, choice of OS, and so on.

hyperfine has nice support for comparing things when using --parameter-list and --export-json, with the varying parameters stored into the JSON as parameter fields across the runs.

Currently, it is possible to compare things across hosts by using ssh as part of the command given to hyperfine and using --parameter-list for the hosts. This has a small drawback though: making the remote connection and spawning the remote shell takes time, and can skew the results. (Trivial example: hyperfine "ssh raspberrypi 'hostname'")

Another existing workaround is to first ssh to each host, and run hyperfine there. This has the drawback that the resulting json files will have to be combined manually afterwards for comparison, possibly with extra manual tagging, requiring extra work.

---

One possible improvement for this would be to add ssh remote shell spawning time compensation to hyperfine, similar to the local shell spawning compensation.

Another possible implementation could be to add support for "remote hyperfines", taking inspiration from topgrade. The main hyperfine would run on your local machine, sending commands and parameters to be run with the hyperfine installations on the remote hosts over ssh. This would ensure that hyperfine only measures the time for the actual payload command, and with correct amount of shell spawning correction at the remote host, and so on. This would also let you collect the results into a single combined json locally, with all the varying parameters - including the host - collected for you automatically for easy processing. (This would also allow for an optimization - remote hyperfines could be safely parallelized for simultaneous execution, with the other parameters running serially on each.)

---

Thank you for writing such an incredibly useful tool 🧡

[sharkdp]

Thank you for the feedback!

One possible improvement for this would be to add ssh remote shell spawning time compensation to hyperfine, similar to the local shell spawning compensation.

I wonder if it's possible to use the existing --shell option to run commands remotely? Something like hyperfine --shell "ssh user@host" …. Probably, it doesn't work right out of the box because ssh does not expect a -c option for the command. But using a simple wrapper script (that takes a -c <cmd> option and calls ssh), this might be possible?