Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
168 Open 1,717 Closed
168 Open 1,717 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Remove newline that broke unimodal task list table
#3489 opened Jan 9, 2026 by ntenenz Loading…
1
feat: support local directory as dataset_path
#3485 opened Jan 5, 2026 by fanjingxiang Loading…
1
Implement new translation tasks for google WMT24++ datasets
#3480 opened Dec 25, 2025 by grzegorz-aniol Loading…
6
Fix utils.py for MATH500 evaluation
#3478 opened Dec 24, 2025 by sheriyuo Loading…
1
Add Windows ML Backend Support for LM Evaluation Harness
#3470 opened Dec 16, 2025 by chapsiru • Draft
2
nit: double logging
#3468 opened Dec 16, 2025 by baberabb Loading…
Fix MGSM stop criteria in Iberian languages to exclude line breaks
#3465 opened Dec 16, 2025 by juliafalcao Loading…
Update command for listing tasks in README
#3464 opened Dec 15, 2025 by DavidUlloa6310 Loading…
1
Rename every bigbench .yaml to be identified using bigbench as task
#3459 opened Dec 10, 2025 by MigueXl Loading…
2
Multiple Bangla Benchmark datasets added
#3454 opened Dec 9, 2025 by Ismail-Hossain-1 Loading…
5
Add Uncheatable Eval
#3442 opened Dec 2, 2025 by ziqing-huang Loading…
5
Refactor TaskManager
#3432 opened Nov 26, 2025 by baberabb Loading…
Fix gsm8k_platinum description
#3411 opened Nov 17, 2025 by fxmarty-amd Loading…
1
2
Fix wrong gpqa_diamond_generative_n_shot answer template
#3407 opened Nov 15, 2025 by fxmarty-amd Loading…
1
Fix MBPP task configuration and code extraction logic
#3387 opened Nov 6, 2025 by tanios13 Loading…
4
feat: refine Chain-of-Thought removal logic
#3386 opened Nov 6, 2025 by Co-Cl2 Loading…
4
[feat] Add Countdown Task
#3384 opened Nov 4, 2025 by StephenXie Loading…
7
Math 500
#3381 opened Nov 1, 2025 by seldereyy Loading…
4
Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py
#3372 opened Oct 28, 2025 by vnayakde Loading…
2
Add support for configurable chrF metric parameters in task YAML, fix…
#3363 opened Oct 23, 2025 by augustlakia Loading…
4
Add gsm_symbolic and gsm_symbolic_cot tasks
#3354 opened Oct 19, 2025 by MengAiDev Loading…
1
[AIME24 | AIME25] Enable Multiple Generation Repeats with Pass@k and Majority@k Metrics
#3351 opened Oct 17, 2025 by ihebchaa Loading…
11
Added ULQA benchmark
#3340 opened Oct 13, 2025 by keramjan Loading…
1
feat: Add support for accelerate-wrapped models in simple_evaluate()
#3313 opened Sep 26, 2025 by DhruvaKashyap Loading…
5
Support torchrun vllm DP
#3304 opened Sep 19, 2025 by luccafong Loading…
1
ProTip! Find all pull requests that aren't related to any open issues with -linked:issue.