Retrieval-augmented generation systems promise grounded answers from large language models (LLM), yet performance depends critically on how source documents are segmented before indexing. This study investigates how pre-index chunking strategies affect both retrieval accuracy and answer quality in domain-specific scenarios. We curated a corpus of software-as-a-service editorial content and constructed a high-quality evaluation dataset containing 2,419 question-answer pairs generated through automated prompting and quality control. We compared four chunking approaches including fixed-size, structure-aware recursive, semantic, and LLM-based methods. Our evaluation protocol assessed retrieval through document localization, semantic similarity, and context relevance, while generation quality was evaluated using chain-of-thought criteria driven by LLM judges. Results demonstrate that recursive chunking consistently outperforms other approaches across all metrics. Smaller chunks improve document localization, while moderately larger chunks enhance semantic alignment and generation scores. LLM-based chunking variants show competitive performance but do not exceed top recursive configurations on the dataset. These findings indicate that preserving document structure through recursive chunking is beneficial for practical RAG implementations, providing actionable guidance for chunk size selection while highlighting token budget constraints in current long-context models.

This public repository contains the question-answering (QA) dataset we created for the research work that we submit for review at International Journal of Artificial Intelligence (IJ-AI). The research work is titled "From fixed-size to structure-aware: evaluating document chunking approaches for retrieval-augmented generation in SaaS editorial content", by LAVAREC and DU. The dataset contains 2,419 QA pairs in SaaS editorial content, suitable for evaluating large language models (LLMs).