About OntoLAMA is a set of language model (LM) probing datasets for ontology subsumption inference. The work follows the "LMs-as-KBs" literature but focuses on conceptualised knowledge extracted from formalised KBs such as the OWL ontologies. Specifically, the subsumption inference (SI) task is introduced and formulated in the Natural Language Inference (NLI) style, where the sub-concept and the super-concept involved in a subsumption axiom are verbalised and fitted into a template to form the premise and hypothesis, respectively. The sampled axioms are verified through ontology reasoning. The SI task is further divided into Atomic SI and Complex SI where the former involves only atomic named concepts and the latter involves both atomic and complex concepts. Real-world ontologies of different scales and domains are used for constructing OntoLAMA and in total there are four Atomic SI datasets and two Complex SI datasets. Dataset Source #Concepts #EquivAxioms #Datasets(Train/Dev/Test) Schema.org 894 N/A Atomic SI: 808/404/2, 830 DOID 11,157 N/A Atomic SI: 90,500/11,312/11,314 FoodOn 30,995 2,383 Atomic SI: 768,486/96,060/96,062 Complex SI: 3,754/1,850/13,080 GO 43,303 11,456 Atomic SI: 772,870/96,608/96,610 Complex SI: 72,318/9,040/9,040 MNLI N/A N/A biMNLI: 235,622/26,180/12,906 Citation The relevant paper has been accepted at Findings of ACL 2023: https://aclanthology.org/2023.findings-acl.213/. ```@inproceedings{he2023language, title={Language Model Analysis for Ontology Subsumption Inference}, author={He, Yuan and Chen, Jiaoyan and Jimenez-Ruiz, Ernesto and Dong, Hang and Horrocks, Ian}, booktitle={Findings of the Association for Computational Linguistics: ACL 2023}, pages={3439--3453}, year={2023} }``` Links See instructions at: https://krr-oxford.github.io/DeepOnto/ontolama/ We have made available a convenient access of these datasets through Huggingface: https://huggingface.co/datasets/krr-oxford/OntoLAMA The arxiv version is available at: https://arxiv.org/abs/2302.06761 Contact Yuan He (yuan.he(at)cs.ox.ac.uk)