<script type="text/javascript">
<!--
document.write('<div id="oa_widget"></div>');
document.write('<script type="text/javascript" src="https://www.openaire.eu/index.php?option=com_openaire&view=widget&format=raw&projectId=undefined&type=result"></script>');
-->
</script>

COPY SCRIPT

For further information contact us at helpdesk@openaire.eu

IgakuQA119: LLM Evaluation on the 119th Japanese Medical Licensing Examination

Name: IgakuQA119: LLM Evaluation on the 119th Japanese Medical Licensing Examination
Creator: Iwase, Naoto
Keywords: LLM, Benchmark, Japanese Medical Licensing Examination, Question Answering

Research datakeyboard_double_arrow_right Dataset 26 Jun 2025Publisher:Zenodo

Authors: Iwase, Naoto;

doi: 10.5281/zenodo.15743221 , 10.5281/zenodo.15743222

IgakuQA119: LLM Evaluation on the 119th Japanese Medical Licensing Examination

- Summary
- Subjects
- Metrics

Abstract

v1.0.0: Initial Public Release This is the first public release of IgakuQA119, a comprehensive framework for evaluating Large Language Models (LLMs) on the 119th Japanese Medical Licensing Examination (JMLE). This project, inspired by the nmle-rta repository, provides a complete and reproducible workflow for assessing the capabilities of modern LLMs in a specialized, high-stakes domain. Key Features in this Release: Comprehensive Evaluation Dataset: Includes the full question set from the 119th JMLE. Flexible LLM Support: Natively supports major cloud APIs (OpenAI, Anthropic, Gemini, OpenRouter) and local LLMs via Ollama. Streamlined YAML-based Workflow: A new, unified run_exp.sh script, controlled by a central experiments.yaml file, manages the entire lifecycle of an evaluation: setup, execution, re-running skipped questions, and grading. Automated Leaderboard: The grade task automatically calculates scores and updates the leaderboard in the README.md. Transparent Data Provenance: Includes details and scripts related to dataset acquisition and preprocessing, ensuring transparency and reproducibility. Dataset Source: The question data (text, choices, images) was processed from official JMLE PDFs via OCR by the author of the nmle-rta project, with permission obtained for its use. The grading logic is based on official information from Japan's Ministry of Health, Labour and Welfare (MHLW). For detailed setup and usage instructions, please refer to the main README.md file. This project is licensed under the Apache License 2.0.

Keywords

LLM, Benchmark, Japanese Medical Licensing Examination, Question Answering

Impact byBIP!

	citations This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).	0
	popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.	Average
	influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).	Average
	impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.	Average

Found an issue? Give us feedback

Average