Powered by OpenAIRE graph
Found an issue? Give us feedback
ZENODOarrow_drop_down
ZENODO
Project deliverable . 2025
License: CC BY
Data sources: Datacite
ZENODO
Project deliverable . 2025
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

Evaluating Graph-RAG Systems for Historical Transport Networks: A Berlin Case Study

Authors: Kim-Baumann, Noah Jefferson;

Evaluating Graph-RAG Systems for Historical Transport Networks: A Berlin Case Study

Abstract

This research project develops and evaluates specialised Retrieval-Augmented Generation (RAG) approaches for historical knowledge graphs, using Berlin's public transportation system during the Cold War era (1945-1989) as a case study While traditional RAG systems excel at processing unstructured text, they often struggle with the complex, highly structured temporal data found in historical databases. This study addresses that gap by establishing a comprehensive evaluation framework for "Graph-RAG" systems in the digital humanities. Key Findings The research demonstrates that no single pipeline excels at all historical query types. While NL-to-Cypher approaches dominated factual retrieval, they struggled with interpretive synthesis. The study found that a multi-pipeline architecture, which intelligently routes user queries to the most appropriate retrieval strategy, achieved a marked improvement in answer quality compared to single-pipeline approaches. Additionally, the findings highlight significant limitations in applying standard community detection algorithms (like Leiden) to sparse, linear infrastructure networks. Files in this Record: Abschlussbericht.pdf: The complete final research report (13 pages) detailing methodology, pipeline architecture, evaluation framework, and findings. question_design_methodology.md: A comprehensive guide to the user-centered evaluation framework, including theoretical grounding in digital humanities tool adoption. rubric.md: The detailed scoring criteria (0-3 scale) used for evaluating historical accuracy, context retention, and explanatory capability across 33 questions. berlin_transport_questions.csv: The complete taxonomy of 33 evaluation questions spanning 6 user personas, 5 difficulty levels, and 5 dimensional categories. user_personas.md: Six detailed user archetypes (from neighborhood historians to data journalists) representing the target audience for the public-facing system. Related Resources Live Demo: https://berlin-transport-history.de/chat Code Repository: https://scm.cms.hu-berlin.de/baumanoa/graph-rag Funding Acknowledgement Dieses Projekt wurde durch das BMFTR-Datenkompetenzzentrum HERMES gefördert.

Keywords

Public transport, Digital humanities

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Related to Research communities
Upload OA version
Are you the author of this publication? Upload your Open Access version to Zenodo!
It’s fast and easy, just two clicks!