Add benchmarking CI

[christiangnrd]

Copying over the benchmarks from CUDA.jl.

I'm not sure if I converted them properly. Some function seem to be missing Metal implementations (like reverse).

The final (and biggest) problem is how inconsistent these results have been. Simply rerunning the benchmarks on the same code gives some huge performance differences. There is always at least one benchmark that is >20% slower or faster.

See #418 (comment)

Todo:

• Implement actual CI benchmark solution from GemmKernels.jl
• Ensure Benchmarks are correct

[maleadt]

Do we need another codespeed instance for this? https://speed.juliagpu.org/ is only for CUDA.jl right now. I wonder if there isn't a better solution nowadays; other projects seem to be tackling this slightly differently:

• GemmKernels.jl: store BenchmarkTools results in gh-pages, and generate a comment: https://github.com/JuliaGPU/GemmKernels.jl/blob/master/benchmarks/runbenchmarks.jl
• KernelAbstractions.jl: Github Actions automation, https://github.com/JuliaGPU/KernelAbstractions.jl/blob/main/.github/workflows/benchmark_pr.yml
• LuxDL: uses github-actions-benchmark, perf: setting up continuous benchmarking LuxDL/LuxLib.jl#128

[christiangnrd]

I think the GemmKernels.jl solution would be the best fit since these cannot be run with Github Actions. I'll finish this up when I get more time.

[maleadt]

I think the GemmKernels.jl solution would be the best fit since these cannot be run with Github Actions.

The LuxDL.jl approach also runs on Buildkite: JuliaGPU/KernelAbstractions.jl#510 (comment)

[christiangnrd]

Moved over to #420 with a JuliaGPU branch in an attempt to figure out why it stopped working. Will continue development on that branch. Sorry for the noise.