Benchmark Overview Supported Tasks We construct a benchmark dataset based on student-built ontologies to enable the experimental investigation of four knowledge engineering sub-tasks: Detection of modeling issues Classification of modeling issues Explanation of modeling issues Generation of alternative modeling solutions as possible corrections Contents The benchmark contains axioms that include the following OWL property restrictions: Existential restrictions (owl:someValuesFrom) Universal restrictions (owl:allValuesFrom) Cardinality restrictions owl:minQualifiedCardinality owl:maxQualifiedCardinality owl:qualifiedCardinality Creation and Further Details The complete methodology for creating the benchmark is described in the following publication: Tsaneva, S., Herwanto, G. B., Llugiqi, M., and Sabou, M. Knowledge Engineering with Large Language Models: A Capability Assessment in Evaluating Ontology Property Restrictions. Submitted to Semantic Web Journal (Under Review) View Publication Sample Test Dataset Sample.csv To facilitate the piloting of the expert annotation of the benchmark, 14 axioms (15%) were annotated by two experts each in an initial evaluation round. These 14 axioms were excluded from the final benchmark and can be used to pilot experimental investigations. Benchmark Dataset Benchmark-NoEval.csv The complete set of axioms is provided without the expert annotation to prevent potential data leaks into the training data of large language models. The complete dataset including expert annotations can be provided upon request. LLM Dataset Additional datasets produced during the experimental investigations are also available. These include outputs of 4 LLMs: GPT 4o (gpt-4o-2024-08-06) Claude Sonnet 3.7 (claude-3-7-sonnet-20250219) Llama 3.3 (Llama-3.3-70B-Instruct-Turbo) DeepSeek V3 (version from 2024/12/26) on the 4 knowledge engineering subtasks (detection, classification, explanation and generation). The complete additional datasets including expert annotations can be provided upon request.