The problem
A card network sees a constant stream of purchases. Three patterns are cheap to detect and expensive to miss: a burst of rapid purchases, an amount far above what the card normally spends, and two purchases too far apart to travel between in the time available.
The detector has to keep up with the stream, keep each card's events in order, survive bad messages, and give the same answer if the stream is replayed.
Architecture
How data moves through the system, left to right.
Key decisions
Every message is keyed by card_id, so a card's events stay on one partition and in order. That makes in-memory per-card state safe without cross-replica coordination.
Offsets are committed only after a batch's alerts are flushed; producers use idempotence and acks=all. A crash can re-process a batch but never silently drop one.
Every message is schema-validated. Invalid ones go to transactions.dlq with the error in a header, so one bad record can't block a partition.
Rules use the transaction's own timestamp, so replaying a topic reproduces exactly the same alerts. CI checks that the Kafka path matches the offline engine transaction for transaction.
Non-root user, read-only root filesystem, no privilege escalation; Kubernetes HPA scales detectors from 2 to 6 replicas on CPU.
Results
| Measure | Result |
|---|---|
| Fraud incidents caught | 96.9% (velocity 100%, travel 100%, amount 90.6%) |
| Alert precision | 92.6% (0.13% false-positive rate) |
| Throughput through Kafka | ~49,000 msgs/s (3 replicas, 6 partitions) |
| Latency at 5,000 msgs/s | p50 46 ms · p95 61 ms · p99 65 ms |
| Rule evaluation cost | ~2 µs per transaction |
| Malformed messages | 3/3 routed to the dead-letter topic |
Synthetic data: 20,000 cards, ~206K transactions over one simulated week, 1,500 injected fraud incidents, with legitimate international trips as hard negatives. Benchmarked on a single 2-vCPU VM.
What I learned & what's next
- Most missed amount spikes are cold starts: the card had fewer than 4 prior purchases, so there was no baseline yet.
- Per-card state lives in memory. A partition that moves during a rebalance loses its history; production would back it with a compacted changelog topic (the Kafka Streams / Flink approach).
- The thresholds are hand-tuned. The next step is to feed alert outcomes back as labels and train a model.