The problem
OCR on crumpled, low-resolution receipts is noisy, and a wrong total silently written to a database is worse than no total. The system needs calibrated confidence, not just a best guess.
It also has to run as a real cloud service: encrypted storage, least-privilege access, idempotent processing and alarms.
Architecture
How data moves through the system, left to right.
Key decisions
Four Tesseract passes (two models × two page-segmentation modes) vote on each field: +7 points on totals over a single pass.
Totals are checked against cash − change and subtotal + tax + rounding: 79.5% → 85.5% exact.
Known vendor names and branch addresses from the training labels let logo-only receipts be recognised: 92.9% exact company names for known vendors.
Thresholds chosen by 5-fold cross-validation for 95% precision; everything below goes to review.
SAM template with KMS, blocked public access, least-privilege IAM, DLQ, reserved concurrency and CloudWatch alarms; tested end to end against mocked AWS in CI.
Results
| Measure | Result |
|---|---|
| Totals exactly right (200 unseen receipts) | 85.5% |
| Dates exactly right | 87.0% |
| Fields auto-accepted | 55.2% at 95.0% precision |
| Wrong values stored | 215 → 22 (−89.8%) |
| Extraction + validation time | 34 ms p50 per receipt |
| OCR time (4 passes) | 4.1 s p50 on one core |
SROIE 2019: 626 real scanned receipts with corrected labels, split with a fixed seed into 426 training and 200 test receipts. The test receipts are used only by the evaluation script.
What I learned & what's next
- Upscaling images made Tesseract worse, and a popular OCR model dropped the spaces between words, so measuring each change mattered.
- Address exact-match is hard (80% near-match, 47.5% exact), so only 15% of addresses clear the auto-accept bar. Honest routing beats optimistic accuracy.
- Vercel can't run the Tesseract binary, so the demo runs Tesseract.js in the browser and sends only the words to the extractor.