PROSTO24: The Evolution of Enterprise RAG: From Basic Search to Hybrid Architectures
Developing effective information retrieval systems for vast repositories of enterprise documents presents a significant challenge. PROSTO24 reports on a practical case study detailing the creation and evolution of a Retrieval-Augmented Generation (RAG) service designed to navigate internal regulatory and normative documents. This project illustrates a journey from a foundational solution to a sophisticated hybrid architecture capable of processing complex data.
From Standard Pipelines to Hybrid Solutions
Initially, the project utilized a standard approach, incorporating tools such as n8n and Qdrant for indexing and search. However, simple text chunking proved ineffective for normative documents, necessitating the development of a more advanced solution. The adoption of a hybrid architecture became a pivotal step, integrating various methodologies:
- BM25 and Embeddings: Employing the BM25 algorithm alongside vector representations (embeddings) to enhance search relevance.
- RRF (Reciprocal Rank Fusion): A method for combining ranking results from multiple sources to improve output quality.
- Reranking: An additional stage of reordering retrieved documents to refine their relevance to the query.
- Structural Chunking: Specialized document segmentation that considers document structure, crucial for regulatory texts.
- Complex Document Processing: Development of techniques to handle tables and other intricate elements within documents.
To further improve information extraction quality, libraries like Natasha and specialized corporate terminology dictionaries were utilized. Multimodal models were also integrated to expand the system’s capabilities.
Measuring Quality and Practical Outcomes
A critical aspect of the project involved establishing a robust quality measurement system. This leveraged tools such as Arize Phoenix and an LLM-as-a-Judge approach, enabling the evaluation of system responses using large language models. Real-world A/B tests with a reranker, evaluated across nine metrics, demonstrated that in production RAG systems, the quality of source document preparation, the efficiency of retrieval mechanisms, and the ability to reproducibly assess changes are more crucial than merely having the “smartest LLM.” This experience underscores that the development of RAG systems is an iterative process demanding deep domain understanding and continuous adaptation.
The transition from basic text chunking to structural chunking and the integration of BM25 with embeddings, fused via RRF, reflects a sophisticated understanding of information retrieval challenges in enterprise RAG. The emphasis on source document preparation and reproducible quality assessment, using tools like Arize Phoenix and LLM-as-a-Judge, highlights a mature development pipeline. This approach directly addresses the limitations of purely semantic search for normative documents, underscoring that retrieval efficiency and data integrity are paramount, often more so than just the LLM’s raw intelligence.