The problem
Retrieval-augmented generation lives or dies on retrieval: if the right passage isn't found, nothing downstream can fix it. And an assistant that never says "I don't know" can't be trusted.
The goal was a grounded question-answering system that runs on a CPU in one serverless function, with no external API, and whose accuracy is measured exactly rather than eyeballed.
Architecture
How data moves through the system, left to right.
Key decisions
Every SQuAD question has a known source passage, so Recall@k is exact. 10,570 dev questions are never used for training or tuning.
Naive reciprocal-rank fusion dropped Recall@1 from 71.5% to 58.7%. A small listwise re-ranker over both scores turned the combination into a +5.3-point gain over BM25.
Answers are verbatim sentences from the cited passage (tested by construction), so they can't hallucinate facts.
A logistic model on the top result's evidence decides whether to answer. Its threshold is set on a calibration split and reported on a separate test split.
Static WordLlama embeddings ship inside the pip package, and BM25 is implemented over flat numpy arrays, so the whole pipeline runs in one serverless function at 13 ms p95.
Results
| Measure | Result |
|---|---|
| Recall@1 (hybrid + re-ranker) | 76.8% |
| Recall@10 | 94.0% |
| vs. BM25 / naive hybrid | +5.3 pts / +18.1 pts Recall@1 |
| Out-of-scope questions declined | 83.3% of 3,660 |
| Quoted answers verbatim | 100% by construction |
| Latency | 6.9 ms p50 · 13.2 ms p95 on 2 vCPUs |
Evaluated on the 10,570 SQuAD v1.1 dev questions. Splits are by article, so nothing leaks between training and evaluation. CI rebuilds the index, retrains every model and fails if the numbers drift.
What I learned & what's next
- Keyword search beat embeddings here, because SQuAD questions reuse the passage's wording. The small embedding model is weak alone but useful as a re-ranking signal.
- Of the wrong answers, 43% cite the right passage but quote the wrong sentence; 57% cite the wrong passage. A cross-encoder re-ranker and a neural reader would target those in turn.
- About 18% of answerable questions are declined. That threshold is a product decision, and it's one number in a JSON file.