Change of daily benchmark machine

[rbergen]

Hi everyone, this is a heads-up that starting tomorrow, I will be using a different machine for the daily benchmark runs, the reports of which are posted to PrimeView.

Up to this point, the benchmarks have been running on 6 core/12 thread i7-8750H system. From tomorrow onwards, this will be a 4 core/8 thread i7-7700HQ. This will come with a rebase of the performance numbers in "my" reports.

The reason is that I have to repurpose the i7-8750H system; the i7-7700HQ system is now the highest-specced system I can make available for this project. If anyone has a more powerful/newer system lying around that they would like to get rid of, do let me know. :)

[mike-barber]

Hi @rbergen and @davepl -- belated Season's greetings and I hope you're both keeping well!

I see you've got the fast Ryzen 5950X hooked up for benchmark runs, which is awesome!
The numbers from that machine look unusually low, however, and I'm not quite sure why.

The reported passes are in the 18,000-19,000 range for the fastest 1-bit, single thread, base solutions (Nim 3, Zig 3, Rust 1). However, on my older Ryzen 3900X, the results look like this:

mike-barber_bit-extreme-hybrid;60322;5.0000042915;1;algorithm=base,faithful=yes,bits=1
GordonBGood_extreme-hybrid;59412;5.0000781;1;algorithm=base,faithful=yes,bits=1
51-ManDeJan&it....;55402;5.00003;1;faithful=yes,algorithm=base,bits=1

Given the performance of the newer 5950X, I'd guesstimate we should be getting more like 70,000. It definitely looks like something weird is going on, but not quite sure what would be causing it.

For reference, my machine is running Windows 10, and I'm running the benchmarks in Docker under WSL2 Ubuntu 20.04.3 (focal). This usually incurs a penalty hit of a couple of percentage points compared to running stuff natively in Linux or Windows, but not really much more than that. I have all the virtualization goodies switched on in the BIOS too. I have the Virtual Machine Platform and Windows Subsystem for Linux features enabled for lightweight WSL2 support, but have not enabled the full Hyper-V stack as it's not required.

The only potential difference is that I'm running Docker directly in Ubuntu instead of by installing Docker Desktop. But I have Docker Desktop at work, and not noticing any real differences there either.

I'm not sure where the issue might be -- it's a really big performance hit, almost as if some hardware virtualization feature is disabled.

I'm more than happy to help trying to reproduce the problem here if you give me some details of the setup. Particularly stuff like

• is it running under Windows or native Linux
• Docker Desktop or alternative
• WSL2 or Hyper-V or VirtualBox
• enabled Windows features

[mike-barber]

Hi @rbergen

I've replicated what looks like the same environment on my machine and this problem is not present. It's about the same speed as my previous WSL2 + Docker runs -- around 60k passes.

Setup is:

• Hyper-V virtual machine on Windows 10
• 8GB minimum RAM assigned, 24 virtual processors assigned, NUMA -> Use Hardware Topology
• Ubuntu 20.04 (installed via the Hyper-V quick start)
• Docker installed as per docker engine instructions -- although just doing sudo apt install docker.io yields the same results.

I'm doing my benchmark runs just to obtain the Rust solution_1 results (a bit quicker than running all the solutions, and I know what values to expect)

npm start --silent -- benchmark -d ../PrimeRust/solution_1 -f json -o benchmark-report.json

I've created a Gist with the results here: https://gist.github.com/mike-barber/15ba7bdbb35999817788b90c19adf4bb

• Dave's results (snipped to only include the Rust results)
• The full Ubuntu VM under Hyper-V as above
• WSL2 + Docker Desktop results

Dave's result:

{
            "implementation": "rust",
            "solution": "1",
            "label": "mike-barber_bit-extreme-hybrid",
            "passes": 20016,
            "duration": 5.0028133392,
            "threads": 1,
            "tags": {
                "algorithm": "base",
                "faithful": "yes",
                "bits": "1"
            }
        },

My result:

{
            "implementation": "rust",
            "solution": "1",
            "label": "mike-barber_bit-extreme-hybrid",
            "passes": 59399,
            "duration": 5.0000753403,
            "threads": 1,
            "tags": {
                "algorithm": "base",
                "faithful": "yes",
                "bits": "1"
            }
        },

I suspect there's a configuration problem of some kind on Dave's machine, but not sure what it could be; it's not something cropping up on my setup. I'm still getting the expected ~60k passes. Compare this with 18-20k on Dave's machine, where we should be expecting well north of 60k on his newer hardware.

It may just be worth double checking the Hyper-V VM settings, and possibly testing with WSL2 + Docker Desktop: if that misbehaves too, it could point to a hardware config problem. If it behaves, it could point to a Hyper-V config or VM problem.

[rbergen]

@mike-barber Thanks for taking the time to look into this. My field of view is limited to what's in the VM, and that does look like a "normal" setup of Ubuntu to me. I think that means that @davepl would need to find the time to look into the setup of the overall machine the VM is running on. I'm in no place to speak on his behalf about if and when he could get around to doing that.

[mike-barber]

Thanks @rbergen! Absolutely agree, and there's no big rush on it. Although there is an upside for @davepl with a 4x faster Linux when it's working :) I'm available to help out where I can.

[rbergen]

@mike-barber Yesterday and today, @davepl and I have done some Geekbench runs on the host machine, within the VM and within Docker. The results are as follows:

• The scores on bare metal are consistent with a machine specced like the host
• The scores in the Hyper-V VM differ less than 4% from the bare metal scores, both single-core and multi-core
• The single-core score within Docker within the VM is approx. 18% lower than directly in the VM. The multi-core score in Docker is approx. 9% lower than in the VM
• The scores within WSL2 differ less than 5% from the bare metal scores, both single-core and multi-core. These scores themselves are irrelevant because we're not running within WSL2, but it does show that the scores are pretty consistent across execution contexts outside of Docker.

This seems to indicate that there is a performance impact from using Docker, but nowhere close to the 70% you feel is missing. I'll look into the 18%, albeit with considerably lower priority, but I'm not sure what else there is left to look at when it comes to the 5950X runner.

Of course, lest we forget that the premise has always been that we're comparing languages, not systems...

[mike-barber]

@rbergen @davepl thanks for looking into it!

I tend to worry about systems reporting results that don't agree with expectations -- unexplainable weirdness makes it difficult to interpret the results. Plus I'm probably a bit OCD about this stuff ;) So thanks for humouring me.

So the great news is that you appear to have fixed the performance problem somehow during the course of GeekBenching stuff -- was probably just a small configuration problem somewhere. 👍

The not so great news is that we may never know what was amiss, unless you can think back over any changes made. Although it's more for curiosity than anything else now :)

The latest run today looks right on the money and roughly where I thought we should land on that hardware (~70k). And it looks like it completed some 35 minutes earlier too (10:45 vs 11:20): faster machine takes much less time to build all those solutions too.

1	Rust	1	mike-barber_bit-extreme-hybrid	69287	5.0000653267	1	13857.218950722956
2	Rust	1	mike-barber_bit-unrolled-hybrid	67888	5.0000257492	1	13577.530077892505
3	Nim	3	GordonBGood_extreme-hybrid	67343	5.000053743	1	13468.455232962084

Compared to yesterday's ~18k:

1	Rust	1	mike-barber_bit-unrolled-hybrid	18239	5.0070137978	1	3642.6901815237493
2	Nim	3	GordonBGood_extreme-hybrid	18062	5.005543116	1	3608.399644439303
3	Rust	1	mike-barber_bit-extreme-hybrid	17519	5.0064377785	1	3499.294463467584

I reckon it's sorted. Thanks again!