Help us run Plexe on MLE Bench Lite

[vaibs-d]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
We're looking for any engineers who would like to help us run Plexe on MLE Bench lite: https://github.com/openai/mle-bench

Describe the solution you'd like
A clear and concise description of what you want to happen.
Run with 10 iterations on Plexe and record results to be shared.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[rayaanoidPrime]

Hey, I found this issue interesting and want to give it a shot. My initial plan is basically to -

• Whip up the standard mle-bench agent files for Plexe (Dockerfile, config.yaml, start.sh).
• The start.sh will translate mle-bench's competition inputs (like problem descriptions & data files) into something Plexe understands.
• Make sure Plexe’s outputs (submission files, its MLflow logs, etc.) land in the right spot for mle-bench.
• Write a straightforward guide for your repo explaining how users can set this up with their mle-bench clone.

Does this sound good ?
Also, if you were looking a particular approach for this issue, do lmk!

[vaibs-d]

Hey, I found this issue interesting and want to give it a shot. My initial plan is basically to -

• Whip up the standard mle-bench agent files for Plexe (Dockerfile, config.yaml, start.sh).
• The start.sh will translate mle-bench's competition inputs (like problem descriptions & data files) into something Plexe understands.
• Make sure Plexe’s outputs (submission files, its MLflow logs, etc.) land in the right spot for mle-bench.
• Write a straightforward guide for your repo explaining how users can set this up with their mle-bench clone.

Does this sound good ? Also, if you were looking a particular approach for this issue, do lmk!

Hey @rayaanoidPrime! That sounds like a great plan! Excited to see how you go about it :)
Feel free to ping us on Discord and ask any questions if you need any help!

[marcellodebernardi]

Hey @rayaanoidPrime, thanks a lot for wanting to take a stab at this! In case it helps, we had already started working on this, you can find the existing implementation in tests/benchmark/mlebench.

We had some remaining issues with the runner script that we never got a chance to come back to, but hopefully the existing code can be a helpful starting point!

[samarthify]

Hi I am interested in taking up this issue. Going through the existing implementation it seems you've already got plexe running on ML bench but need someone to record the results with 10 iterations and share the results. Is that correct? @marcellodebernardi @vaibs-d