%slower reporting seems wrong

[Craig-Macomber]

I wrote a trivial test to make sure the tool was working which tended to be "Infinity" ops per second, which is apparently -Infinift% slower than 2,806,756,516 ops per second. The tool should never report infinite performance. Also, infinite performance should be faster than finite performance.

I then wrote a more real test comparing refs and weakrefs: https://jsperf.app/zinupu

The times were 5,522,394 ops per second and 258,163 ops per second. The ratio here is 21.39.

There are several common ways to report this as a percentage but I do not think it is valid to say B is "95% slower" which is what is reported by the tool.

We can say A is 21.39 times faster than B. We can say A is 2039% faster than B. We can say A is 2139% the speed of B. We can say B is 21.39 times slower than B. Or B takes 2039% longer than A, or that B takes 2139% the time of A. We can also say that A takes 4.7% the time of B.

For a simpler example: if something takes 4 times as long, its a quarter as fast or 4 times as slow. Its not 75% slower.

[rd13]

@Craig-Macomber I think that it is more human readable when comparing multiple tests to the fastest as a %?

I see what you are saying though, the ratio is important, like in your example:

e.g, expressed as a ratio:

A = Fastest (5,522,394 ops/s)
B = 21.39 times slower (258,163 ops/s)
C = 46.12 times slower (119,739 ops/s)

and as a percentage:

A = Fastest
B = 95% slower
C = 98% slower

In the above representations I'd agree that the ratio is useful for also comparing B to C at a glance - B is almost twice as fast as C, whereas it looks very similar when expressed as a percentage compared to the fastest.

[Craig-Macomber]

What I usually actually want is the time per op (usually nanoseconds per op). I know what my CPU can do in a nanosecond for example (~3 cycles), so it lets me get some useful ground truth.

The ratio is great for doing comparisons, but it would be nice to be able to pick which one to compare to (maybe default to the fastest).

Not sure if its useful to you at all, but an example of benchmark results formatting suited to my preferences can be found in https://github.com/microsoft/FluidFramework/blob/main/tools/benchmark/src/Reporter.ts#L165 which I wrote to display what I wanted from Benchmark.js (Which I think you also use). In practice I ended up pretty much only using the Period column.

[rd13]

That is useful information, I could add some custom views for the results, or like a verbose checkbox or something. The current results format was just to replicate classic jsperf.