Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Dataset . 2025
License: CC BY
Data sources: ZENODO
ZENODO
Dataset . 2025
License: CC BY
Data sources: Datacite
ZENODO
Dataset . 2025
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

Reference Set and MLLM Visual Information Extraction Prototype

Authors: Dilán-Pantojas, Israel; Duong, Phu T.; Boyce, Richard;

Reference Set and MLLM Visual Information Extraction Prototype

Abstract

Manually extracted data from figures and tables reported in pharmacology studies previously collected as part of an internal reference set. The dataset contains images from the visual elements and their corresponding values from 45 published pharmacology studies with distinct PubMed IDs. Multiple images could be sampled from each of these papers, and multiple values were often sampled from each image. Therefore, the reference set contains multiple rows with values from images of tables or from figures corresponding to graphs, plots, or charts. Dataset contains annotated information from 43 images of figueres and 40 images of tables. The visual elements contain data from any of eight different types of experiments, namely in vitro enzyme inhibition, induction, & kinetics, in vitro transporter inhibition, induction, and kinetics, as well as in vivo enzyme kinetics and in vivo interaction studies. The selected sample represents a wide range of styles, layouts, and structures for both figures and tables. We also provide code from our MLLLMs Visual Information Extraction prototype using the Pydantic AI v1.25 Python module to connect with multiple models to perform VIE and produce a structured JSON output. Our pilot VIE system was used to process images from the reference set along with the rest of the annotated information to generate prompts. We have evaluated the following models. Inference Provider Model Company Model Name Context Window Number of Parameters AWS Bedrock Anthropic Claude Sonnet 3.7 128K * Claude Sonnet 4.0 1M * AWS Nova Pro 300K * Nova Premier 1M * Meta Llama 3.2 128K 90B Llama 4 Scout 10M 109B Llama 4 Maverick 1M 400B Open AI API Open AI GPT-4o 128K * GPT-5 400K * Google Vertex Google Gemini 2.5 Pro 1M * *The actual number of parameters for this model has not been made publicly available. Error corrections: Within the "Manuscript Results folder" > "Tolerance Based ACC.ods" the calculation of Tolerance based accuracy for cells F12-J21 was incorrectly calculated by dividing the corresponding cell F1-J10 over 172 instead of 162. For example the correction for the value of cell F12 is to change it's content from "=ROUND(F1/172,3)*100" to "=ROUND(F1/162,3)*100".

Keywords

Visual Information Extraction, Artificial Intelligence, Information Extraction

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average