Answer quality, not just retrieval
Retrieval answers score 0.9593 on groundedness and hallucinate on two thirds of the questions the corpus cannot answer. Both numbers are correct, which is why groundedness alone certifies nothing.
What the measurements say
The findings worth reading first.
An agent, and the keyword router that beats it
Local models drive all five services as tools. Neither beats a keyword router on task success (0.8000 against 0.4250 at 0.5B and 0.6250 at 1.5B). At 0.5B the agent refused 0 of 7 unanswerable questions and fabricated on 2 of 5 tasks whose service was killed mid-run; at 1.5B both failures are gone.
Load, and a dependency killed mid-run
With sales killed while ops serves traffic, p95 goes from 4185 ms with the breaker closed to 74.6 ms once it opens — below the healthy figure, because a skipped call beats a successful one. Every request returned 200 throughout.
A ceiling found, fixed, and re-measured
Retrieval appeared to peak at concurrency 4 and fall. That was reported as a GIL limit and it was wrong — the event store capped every endpoint. The obvious repair (WAL alone) measured worse than nothing; connection reuse plus WAL was ~5x, and throughput more than doubled. Both readings are kept.
What it would cost to run
Retrieval compute is 9 cents per million requests; the cheapest LLM answer on top is $1,100, so generation is 12,629x everything underneath it. Doubling throughput moves the bill 0.004%; changing model tier moves it 400%.
One request across five services
Request IDs propagate across process boundaries and are stored beside each event, so one decision reconstructs into a single ordered timeline — 18 hops across 5 services. A service that cannot be reached is named in the output rather than omitted.
Project matrix
What each system measures.
Bring the whole stack up with docker compose up from this
repository; each service is then on the port shown. To run one alone,
clone its repository and follow its DEMO.md.
| Project | Problem | Headline result | Real-data track | Tests | Compose port |
|---|---|---|---|---|---|
| Enterprise RAG | Ranking retrieval methods against each other on benchmarks with human relevance judgments. | Dense nDCG@10 0.3727 against BM25's 0.2831 on BEIR/NFCorpus, with a per-query-type breakdown showing where each method wins and loses. | BEIR/NFCorpus — real human relevance judgments. | 93 | :8001 |
| Customer Ops | Deciding what to do about an inbound customer message, auditably. | Intent macro-F1 0.9164 across 77 real customer intents on BANKING77; 0.6476 on held-out synthetic phrasings the model has never seen. | BANKING77 — 13,083 queries real customers sent an online bank. | 92 | :8004 |
| Incident Detection | Finding anomalous service behaviour in telemetry without drowning the on-call in alerts. | Precision 0.7895 with all 17 held-out incidents caught and 32% fewer alerts than the z-score baseline. | Five weeks of real server telemetry, anomalies labelled by operators from actual incident reports. | 77 | :8003 |
| Sales Intelligence | Scoring account renewal propensity with an explanation that reconstructs the score. | Held-out ROC-AUC 0.8614 against a measured Bayes ceiling of 0.8898 — the model recovers most of the signal the generating process actually contains. | 41,188 actual campaign outcomes from the UCI Bank Marketing dataset. | 50 | :8002 |
| Meeting Intelligence | Pulling decisions and action items with owners out of a meeting transcript. | Macro-F1 0.5894 against 0.3235 for the keyword extractor it replaced, on the same held-out data. | AMI Meeting Corpus — 137 recorded meetings with human-annotated decisions and action items. | 67 | :8005 |
Deep dives
One section per service, linked to the runnable repository.
Enterprise RAG Knowledge System
Repository- Problem
- Ranking retrieval methods against each other on benchmarks with human relevance judgments.
- How it works
- Four retrievers behind one
Retrieverinterface — BM25, LSA, a sentence-transformers dense bi-encoder, and reciprocal-rank fusion — with one evaluation harness, a fitted and a pretrained reranker, extractive answer generation with citation spans, and fitted abstention thresholds. - Headline result
- Dense nDCG@10 0.3727 against BM25's 0.2831 on BEIR/NFCorpus, with a per-query-type breakdown showing where each method wins and loses.
- Validated on real data
- BEIR/NFCorpus — real human relevance judgments.
- What came out worse than hoped
- A cross-encoder reranker worth +0.0011 nDCG@10 for 176x the retrieval compute, and weighted fusion losing to dense alone.
- API surface
GET /health,GET /metrics,GET /v1/query,POST /v1/query,POST /v1/documents,GET /v1/drift— see DEMO.md.- Tests
- 93, run by CI on every push, which also regenerates the datasets and retrains to prove the committed metrics.
- Known limits
- Answer quality is measured lexically, so a correct paraphrase scores zero. Abstention is fitted per retriever but deliberately not served for BM25, whose scores are not comparable across queries.
AI Proactive Customer Operations
Repository- Problem
- Deciding what to do about an inbound customer message, auditably.
- How it works
- Fitted intent and sentiment classifiers (TF-IDF word and character n-grams into logistic regression) in front of a deterministic, auditable policy layer. Every decision names the rule that produced it. Enrichment from three other services is optional and degrades.
- Headline result
- Intent macro-F1 0.9164 across 77 real customer intents on BANKING77; 0.6476 on held-out synthetic phrasings the model has never seen.
- Validated on real data
- BANKING77 — 13,083 queries real customers sent an online bank.
- What came out worse than hoped
- A row-level split scores 1.00 on intent. That number is worthless and is published as the reason the split had to change — 0.6476 on unseen phrasings is the honest figure.
- API surface
GET /health,GET /metrics,POST /v1/decide,POST /v1/events/incident,GET /v1/proactive/outreach,GET /v1/drift— see DEMO.md.- Tests
- 92, run by CI on every push, which also regenerates the datasets and retrains to prove the committed metrics.
- Known limits
- Policy rules are hand-written and cover the enumerated cases only. No live CRM; priority is derived from synthetic account facts.
AI Incident Detection Platform
Repository- Problem
- Finding anomalous service behaviour in telemetry without drowning the on-call in alerts.
- How it works
- An
IsolationForestfitted on normal traffic only, scoring each minute of per-service metrics against that service's own healthy baseline, with a threshold calibrated to an explicit precision target. Chronological split, episode-level as well as point-level metrics. Anomalies push events to customer operations through an outbox with retries and a dead-letter queue. - Headline result
- Precision 0.7895 with all 17 held-out incidents caught and 32% fewer alerts than the z-score baseline.
- Validated on real data
- Five weeks of real server telemetry, anomalies labelled by operators from actual incident reports.
- What came out worse than hoped
- On real telemetry the fitted detector loses to a three-line z-score baseline — PR-AUC 0.1897 against 0.4348. That result is published rather than dropped, and it is the strongest single finding in the portfolio.
- API surface
GET /health,GET /metrics,POST /v1/score,GET /v1/incidents/active,GET /v1/events/outbox,GET /v1/events/dlq,GET /v1/drift— see DEMO.md.- Tests
- 77, run by CI on every push, which also regenerates the datasets and retrains to prove the committed metrics.
- Known limits
- No production telemetry ingestion, no alert routing, no incident workflow integration. The served model is fitted on synthetic telemetry; the real-data track is a parallel evaluation, not the served model.
AI Sales Intelligence Engine
Repository- Problem
- Scoring account renewal propensity with an explanation that reconstructs the score.
- How it works
- A fitted
StandardScalerintoLogisticRegressionpipeline, hyperparameters cross-validated on train only, with per-feature attribution that provably reconstructs the model's own output and segment thresholds derived from train-set score quantiles. - Headline result
- Held-out ROC-AUC 0.8614 against a measured Bayes ceiling of 0.8898 — the model recovers most of the signal the generating process actually contains.
- Validated on real data
- 41,188 actual campaign outcomes from the UCI Bank Marketing dataset.
- What came out worse than hoped
- On real data the same pipeline scores 0.7090 or 0.9364 depending only on whether the split is chronological and whether one leaky feature is dropped. Evaluation design was worth 0.2274 ROC-AUC — more than the leakage. 0.7090 is the published headline.
- API surface
GET /health,GET /metrics,POST /v1/score,GET /v1/accounts/{id}/score,GET /v1/drift— see DEMO.md.- Tests
- 50, run by CI on every push, which also regenerates the datasets and retrains to prove the committed metrics.
- Known limits
- No real CRM feed, no retraining workflow. One predictive feature (
industry) is generated but excluded from training because it is not available at serving time, and that is stated in the model card.
Autonomous Meeting Intelligence
Repository- Problem
- Pulling decisions and action items with owners out of a meeting transcript.
- How it works
- A fitted three-class sentence classifier (TF-IDF word and character n-grams into logistic regression) replaces a keyword gate, with rule-based slot extraction for owner and due date — scored against gold slots rather than assumed correct. Outputs are validated against a Pydantic schema and published into the retrieval corpus so meeting decisions become searchable.
- Headline result
- Macro-F1 0.5894 against 0.3235 for the keyword extractor it replaced, on the same held-out data.
- Validated on real data
- AMI Meeting Corpus — 137 recorded meetings with human-annotated decisions and action items.
- What came out worse than hoped
- On real meetings it performs poorly — positive-class macro-F1 0.1799, against 0.0244 for the keyword baseline. Roughly 7x the baseline on the classes that matter, and still a weak absolute result. A test asserts the slot scores stay below 0.95, because a perfect score meant the corpus was too easy.
- API surface
GET /health,GET /metrics,POST /v1/analyze,GET /v1/drift— see DEMO.md.- Tests
- 67, run by CI on every push, which also regenerates the datasets and retrains to prove the committed metrics.
- Known limits
- Owner extraction returns one owner, so "Chen and Maya will…" yields "Chen" — a schema change, not a pattern fix. Slot extraction is measured on synthetic data only, because AMI names owners in prose rather than structured fields.
Reviewer path
Three minutes, fifteen, or thirty.
3 minutes
- Read the evidence cards above.
- Scan the project matrix for headline results and real-data tracks.
- Open one decision record — they carry the alternatives that were rejected.
15 minutes
- Start with Enterprise RAG or Incident Detection — both publish a result that went against them.
- Compare each README's headline against its model card.
- Read SCALE_TEST.md for a finding that a later measurement overturned.
30 minutes, running it
docker compose upfrom this repository brings all five up.- Or clone one repo, then
pytest -q, its training script with--verify, and its evaluation script. python scripts/verify_contracts.py --localchecks the cross-service contracts against live services.
CI evidence
Every repository retrains and re-evaluates on push.
Documentation map
From overview to evidence.
Architecture decision records30 contested choices, each with the alternatives rejected and a revisit trigger.
The agentLocal models over five services, scored against a keyword router.
Load and degradationConcurrency, and a dependency killed mid-run.
Horizontal scalingWhat more workers buy, and the ceiling underneath them.
Cost modelCost per million from measured throughput and dated prices.
Portfolio READMEIndex, system map, runbook.
Architecture docsSystem landscape and shared service baseline.
API flow diagramsSequence diagrams for each API path.
Interviewer walkthroughRecruiter and technical review script.
Tradeoff notesLocal-first decisions and their costs.
What this does not claim
Worth knowing before going deep.
- Training data for every served model is synthetic and disclosed. The real-data tracks are parallel evaluations of the same pipelines, not the served models.
- Everything runs on one machine. Load, horizontal scaling and cost are measured or modelled locally; there is no second host, no network between services, no soak test and no cloud bill.
- Reported LLM metrics come from local 0.5B and 1.5B models, not frontier ones. Two points is not a scaling law.
- Every figure is a dated snapshot, not a live claim. Token prices carry the date they were read.