01 · RAG Engineering / Comparative Study
A comparative engineering exercise examining what generative-AI-assisted development contributes to a functional RAG product, and what additional discipline is required to evaluate and improve reliability.
The question
How far can AI-assisted development take a functional RAG prototype, and what engineering work is required to evaluate whether that RAG is actually reliable?
Baseline · ~10 h
Built in approximately 10 hours, INFOPAY-AI is the rapid AI-assisted baseline. Claude Code helped produce a functional product for importing varied payslip PDFs, extracting structured salary and contribution data, visualizing multiple months and answering natural-language questions.
Its hybrid architecture routes authoritative numerical operations through Pandas. Document-oriented questions use RAG backed by ChromaDB, while LangChain and LangGraph support orchestration. This preserves natural-language interaction without asking the language model to perform authoritative calculations.
The prototype can appear satisfactory in ordinary use, but plausible answers and apparent functionality alone do not establish retrieval reliability. INFOPAY evaluation metrics are still pending.
View live project: INFOPAY-AIFor four fully off-topic general-knowledge questions, INFOPAY is explicitly instructed to answer from general model knowledge rather than search payslips. A 0% abstention rate for that category is intentional behavior and is not classified as a retrieval failure.
Engineered · ~20 h
Built in approximately 20 hours, AEROSPEC-AI works with technical documentation and focuses on measurable retrieval quality, reproducibility, source traceability, citations, abstention and error analysis.
Its explicit boundaries separate React/Vite, FastAPI, Chroma retrieval, OpenAI generation, citation validation and the JSON interface. /search exposes retrieval without generation; /ask adds generation and citations; deterministic /convert operations do not require an LLM.
Pydantic validates contracts, request IDs and structured logs support diagnostics, and sensitive question, prompt, chunk and response content is excluded from logs. Citation identifiers are validated before provenance is returned.
View live project: AEROSPEC-AIReference architecture
Retrieval, generation, validation, API and interface boundaries remain explicit.
/search retrieval inspection
/ask retrieval + generation + citations
/convert deterministic operations
What changed
Evaluation methodology
INFOPAY-AI was evaluated on September 24, 2026 using 36 questions: 24 answerable, 12 unanswerable and 7 fictitious payslips. AEROSPEC-AI was evaluated on September 22, 2026 using 72 questions: 48 answerable, 24 unanswerable and 5 technical PDF documents.
Source Hit asks whether the expected document was retrieved. Evidence Hit asks the stricter question: whether the retrieved chunk contains the complete evidence required to answer. Abstention measures when the system declines to answer, and citation checks validate traceable provenance.
The two prototypes were evaluated on separate benchmarks. The differences below describe their respective results and illustrate the evolution of the engineering approach; they are not an A/B experiment on an identical corpus. The benchmarks differ in size, domain and source documents, and neither measures production-wide performance.
INFOPAY records 33.3% Source Hit@3 and 20.8% Evidence Hit@3; AEROSPEC records 89.6% and 54.2% on its own benchmark. A system may identify the correct source without retrieving the evidence needed to support the answer. AEROSPEC improves both measured rates on its benchmark, but its remaining gap still identifies retrieval precision as an area for improvement.
The system correctly abstains on many unanswerable cases, but false abstentions remain significant. This is an area for investigation, not a solved behavior.
| Metric | INFOPAY-AI | AEROSPEC-AI | Evolution |
|---|---|---|---|
| Source Hit@1 | 25.0% | 87.5% | +62.5 pts (Improved) |
| Source Hit@3 | 33.3% | 89.6% | +56.3 pts (Improved) |
| Evidence Hit@1 | 12.5% | 35.4% | +22.9 pts (Improved) |
| Evidence Hit@3 | 20.8% | 54.2% | +33.4 pts (Improved) |
| Correct abstentions | 66.7% | 83.3% | +16.6 pts (Improved) |
| False abstentions | 54.2% | 39.6% | -14.6 pts (Improved) |
Traceability
Not directly comparable
Sources are mechanically reconstructed from LangGraph ToolMessages as month/excerpt metadata. Five of 24 measured answerable cases produced at least one source, and 9/9 extracted records had non-empty fields. This does not verify semantic support for a generated claim.
AEROSPEC emitted 37/37 valid citation IDs, resolved all 37 to source provenance and emitted no unknown IDs. This validates citation structure and provenance, not semantic claim support.
Human evaluation
Answer correctness, grounding, coverage and support are not applicable for either current benchmark because no case-by-case human review was performed. They are not recorded as zero.
Error analysis
Controlled iteration
The benchmark stays fixed and one variable changes at a time. These are planned experiments; no improvement is claimed yet.
top_k: 3 → 5If needed, chunk_size: 1000 → 600Only then, modify the generation promptTechnical stack
Conclusion
A RAG system that works in a demo is not necessarily a reliable RAG system.
The current baseline shows why retrieval evaluation, evidence-level measurement, abstention analysis, traceability and reproducible experimentation matter. AEROSPEC-AI is not presented as fully reliable; its benchmark makes remaining weaknesses visible.