This release continues our steady stream of community contributions with a batch of new benchmarks, expanded model support, and important fixes. A notable change: Python 3.10 is now the minimum required version. New Benchmarks & Tasks A big wave of new evaluation tasks this release: AIME and MATH500 math reasoning benchmarks by @jannalulu in #3248, #3311 BabiLong and Longbench v2 for long-context evaluation by @jannalulu in #3287, #3338 GraphWalks by @jannalulu in #3377 ZhoBLiMP, BLiMP-NL, TurBLiMP, LM-SynEval, and BHS linguistic benchmarks by @jmichaelov in #3218, #3221, #3219, #3184, #3265 Icelandic WinoGrande by @jmichaelov in #3277 CLIcK Korean benchmark by @shing100 in #3173 MMLU-Redux (generative) and Spanish translation by @luiscosio in #2705 EsBBQ and CaBBQ bias benchmarks by @valleruizf in #3167 EQBench in Spanish and Catalan by @priverabsc in #3168 Anthropic discrim-eval by @Helw150 in #3091 XNLI-VA by @FranValero97 in #3194 Bangla MMLU (Titulm) by @Ismail-Hossain-1 in #3317 HumanEval infilling by @its-alpesh in #3299 CNN-DailyMail 3.0.0 by @preordinary in #3426 Global PIQA and new acc_norm_bytes metric by @baberabb in #3368 Fixes & Improvements Core Changes: Python 3.10 minimum by @jannalulu in #3337 Unpinned datasets library by @baberabb in #3316 BOS token handling: Delegate to tokenizer; add_bos_token now defaults to None by @baberabb in #3347 Renamed LOGLEVEL env var to LMEVAL_LOG_LEVEL to avoid conflicts by @fxmarty-amd in #3418 Resolve duplicate task names with safeguards by @giuliolovisotto in #3394 Task Fixes: Fixed MMLU-Redux to exclude samples without error_type="ok" and display summary table by @fxmarty-amd in #3410, #3406 Fixed AIME answer extraction by @jannalulu in #3353 Fixed LongBench evaluation and group handling by @TimurAysin, @jannalulu in #3273, #3359, #3361 Fixed crows_pairs dataset by @jannalulu in #3378 Fixed Gemma tokenizer add_bos_token not updating by @DarkLight1337 in #3206 Fixed lambada_multilingual_stablelm by @jmichaelov, @HallerPatrick in #3294, #3222 Fixed CodeXGLUE by @gsaltintas in #3238 Pinned correct MMLUSR version by @christinaexyou in #3350 Updated minerva_math by @baberabb in #3259 Backend Fixes: Fixed vLLM import errors when not installed by @fxmarty-amd in #3292 Fixed vLLM data_parallel_size>1 issue by @Dornavineeth in #3303 Resolved deprecated vllm.utils.get_open_port by @DarkLight1337 in #3398 Fixed GPT series model bugs by @zinccat in #3348 Fixed PIL image hashing to use actual bytes by @tboerstad in #3331 Fixed additional_config parsing by @brian-dellabetta in #3393 Fixed batch chunking seed handling with groupby by @slimfrkha in #3047 Fixed no-output error handling by @Oseltamivir in #3395 Replaced deprecated torch_dtype with dtype by @AbdulmalikDS in #3415 Fixed custom task config reading by @SkyR0ver in #3425 Model & Backend Support OpenAI GPT-5 support by @babyplutokurt in #3247 Azure OpenAI support by @zinccat in #3349 Fine-tuned Gemma3 evaluation support by @LearnerSXH in #3234 OpenVINO text2text models by @nikita-savelyevv in #3101 Intel XPU support for HFLM by @kaixuanliu in #3211 Attention head steering support by @luciaquirke in #3279 Leverage vLLM's tokenizer_info endpoint to avoid manual duplication by @m-misiura in #3185 What's Changed Remove trust_remote_code: True from updated datasets by @Avelina9X in https://github.com/EleutherAI/lm-evaluation-harness/pull/3213 Add support for evaluating with fine-tuned Gemma3 by @LearnerSXH in https://github.com/EleutherAI/lm-evaluation-harness/pull/3234 Fix add_bos_token not updated for Gemma tokenizer by @DarkLight1337 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3206 remove incomplete compilation instructions, solves #3233 by @ceferisbarov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3242 Update utils.py by @Anri-Lombard in https://github.com/EleutherAI/lm-evaluation-harness/pull/3246 Adding support for OpenAI GPT-5 model by @babyplutokurt in https://github.com/EleutherAI/lm-evaluation-harness/pull/3247 Add xnli_va dataset by @FranValero97 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3194 Add ZhoBLiMP benchmark by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3218 Add BLiMP-NL by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3221 Add TurBLiMP by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3219 Add LM-SynEval Benchmark by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3184 Fix unknown group key to tag in yaml config for lambada_multilingual_stablelm by @HallerPatrick in https://github.com/EleutherAI/lm-evaluation-harness/pull/3222 update minerva_math by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3259 feat: Add CLIcK task by @shing100 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3173 Adds Anthropic/discrim-eval to lm-evaluation-harness by @Helw150 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3091 Add support for OpenVINO text2text generation models by @nikita-savelyevv in https://github.com/EleutherAI/lm-evaluation-harness/pull/3101 Update MMLU-ProX task by @weihao1115 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3174 Support for AIME dataset by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3248 feat(scrolls): delete chat_template from kwargs by @slimfrkha in https://github.com/EleutherAI/lm-evaluation-harness/pull/3267 pacify pre-commit by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3268 Fix codexglue by @gsaltintas in https://github.com/EleutherAI/lm-evaluation-harness/pull/3238 Add BHS benchmark by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3265 Add acc_norm metric to BLiMP-NL by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3272 Add acc_norm metric to ZhoBLiMP by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3271 Add EsBBQ and CaBBQ tasks by @valleruizf in https://github.com/EleutherAI/lm-evaluation-harness/pull/3167 Add support for steering individual attention heads by @luciaquirke in https://github.com/EleutherAI/lm-evaluation-harness/pull/3279 Add the Icelandic WinoGrande benchmark by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3277 Ignore seed when splitting batch in chunks with groupby by @slimfrkha in https://github.com/EleutherAI/lm-evaluation-harness/pull/3047 [fix][vllm] Avoid import errors in case vllm is not installed by @fxmarty-amd in https://github.com/EleutherAI/lm-evaluation-harness/pull/3292 Fix LongBench Evaluation by @TimurAysin in https://github.com/EleutherAI/lm-evaluation-harness/pull/3273 add intel xpu support for HFLM by @kaixuanliu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3211 feat: Add mmlu-redux and it's spanish transaltion as generative task definitions by @luiscosio in https://github.com/EleutherAI/lm-evaluation-harness/pull/2705 Add BabiLong by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3287 Add AIME to task description by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3296 Add humaneval_infilling task by @its-alpesh in https://github.com/EleutherAI/lm-evaluation-harness/pull/3299 Add eqbench tasks in Spanish and Catalan by @priverabsc in https://github.com/EleutherAI/lm-evaluation-harness/pull/3168 [fix] add math and longbench to test dependencies by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3321 Fix: VLLM model when data_parallel_size>1 by @Dornavineeth in https://github.com/EleutherAI/lm-evaluation-harness/pull/3303 unpin datasets; update pre-commit by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3316 bump to python 3.10 by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3337 Longbench v2 by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3338 Leverage vllm's tokenizer_info endpoint to avoid manual duplication by @m-misiura in https://github.com/EleutherAI/lm-evaluation-harness/pull/3185 Add support for Titulm Bangla MMLU dataset by @Ismail-Hossain-1 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3317 remove duplicate tags/groups by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3343 Align humaneval_64_instruct task label in README to name in yaml file by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3344 Fixes bugs when using gpt series model by @zinccat in https://github.com/EleutherAI/lm-evaluation-harness/pull/3348 [fix] aime doesn't extract answers by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3353 add global_piqa; add acc_norm_bytes metric by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3368 [fix] crows_pairs dataset by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3378 Fix issue 3355 assertion error by @marksverdhei in https://github.com/EleutherAI/lm-evaluation-harness/pull/3356 fix(gsm8k): align README to yaml file by @neoheartbeats in https://github.com/EleutherAI/lm-evaluation-harness/pull/3388 added azure openai support by @zinccat in https://github.com/EleutherAI/lm-evaluation-harness/pull/3349 Delegate BOS to the tokenizer; add_bos_token defaults to None by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/pull/3347 fix trust_remote_code=True for longbench by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3361 [feat] add graphwalks by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3377 Longbench group fix by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3359 Resolve deprecation of vllm.utils.get_open_port by @DarkLight1337 in https://github.com/EleutherAI/lm-evaluation-harness/pull/3398 Trim whitespace in remove_whitespace filter by @ziqing-huang in https://github.com/EleutherAI/lm-evaluation-harness/pull/3408 Fixes #3391 avoid error on no-output by @Oseltamivir in https://github.com/EleutherAI/lm-evaluation-harness/pull/3395 Fix PIL image hashing to use actual bytes instead of object repr by @tboerstad in https://github.com/EleutherAI/lm-evaluation-harness/pull/3331 [MMLU redux] Do not use samples which do not have error_type="ok" by @fxmarty-amd in https://github.com/EleutherAI/lm-evaluation-harness/pull/3410 fix: resolve duplicate task names and add safeguards. by @giuliolovisotto in https://github.com/EleutherAI/lm-evaluation-harness/pull/3394 Add MATH500 by @jannalulu in https://github.com/EleutherAI/lm-evaluation-harness/pull/3311 [bugfix] additional_config parsing by @brian-dellabetta in https://github.com/EleutherAI/lm-evaluation-harness/pull/3393 fix(tasks):pin correct MMLUSR version by @christinaexyou in https://github.com/EleutherAI/lm-evaluation-harness/pull/3350 Fix lambada_multilingual_stablelm by @jmichaelov in https://github.com/EleutherAI/lm-evaluation-harness/pull/3294 Fix descriptions in the Moral Stories and Histoires Morales tasks. by @upunaprosk in https://github.com/EleutherAI/lm-evaluation-harness/pull/3374 Replace deprecated torch_dtype parameter with dtype by @AbdulmalikDS in https://github.com/EleutherAI/lm-evaluation-harness/pull/3415 [fix] Fix mmlu_redux not displaying summary table + display to the user the tasks / yaml that are actually pulled by @fxmarty-amd in https://github.com/EleutherAI/lm-evaluation-harness/pull/3406 Rename the conflicting environment variable LOGLEVEL to LMEVAL_LOG_LEVEL by @fxmarty-amd in https://github.com/EleutherAI/lm-evaluation-harness/pull/3418 Update SGLang installation and documentation links by @Bobchenyx in https://github.com/EleutherAI/lm-evaluation-harness/pull/3422 Fix reading custom task configs by @SkyR0ver in https://github.com/EleutherAI/lm-evaluation-harness/pull/3425 New Task: Add CNN-DailyMail (3.0.0) by @preordinary in https://github.com/EleutherAI/lm-evaluation-harness/pull/3426 New Contributors @LearnerSXH made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3234 @ceferisbarov made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3242 @Anri-Lombard made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3246 @babyplutokurt made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3247 @FranValero97 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3194 @HallerPatrick made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3222 @Helw150 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3091 @nikita-savelyevv made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3101 @weihao1115 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3174 @jannalulu made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3248 @slimfrkha made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3267 @gsaltintas made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3238 @valleruizf made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3167 @TimurAysin made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3273 @kaixuanliu made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3211 @its-alpesh made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3299 @priverabsc made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3168 @Dornavineeth made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3303 @m-misiura made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3185 @Ismail-Hossain-1 made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3317 @zinccat made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3348 @marksverdhei made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3356 @neoheartbeats made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3388 @ziqing-huang made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3408 @Oseltamivir made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3395 @tboerstad made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3331 @brian-dellabetta made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3393 @christinaexyou made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3350 @AbdulmalikDS made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3415 @Bobchenyx made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3422 @SkyR0ver made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3425 @preordinary made their first contribution in https://github.com/EleutherAI/lm-evaluation-harness/pull/3426 Full Changelog: https://github.com/EleutherAI/lm-evaluation-harness/compare/v0.4.9.1...v0.4.9.2