Marketers receive Marketing Mix Model results as a wall of coefficients and credible intervals. Juno parses that output deterministically, then a multi-agent LLM system explains what it means: a prioritized report, per-channel reads that carry explicit confidence, and a chat interface routing questions to specialized handlers. Every claim is grounded in either the parsed numbers or a cited methodology document. The backend is FastAPI streaming over SSE with ChromaDB for retrieval, Claude Sonnet 4.5 driving the agent and Opus 4.5 acting as an independent judge; the frontend is Next.js and TypeScript. The second half of the project is the real point: a benchmark suite scores the agent across six dimensions against 100 ground-truth scenarios, then validates the judge itself for reliability.
Demo sleeps when idle — first load may take ~50s to wake.
Benchmark results
- Composite
- 0.897
- Ranking accuracy
- 0.875
- Calibration error (ECE)
- 0.093
- Groundedness
- 0.903
- Failure-mode recall
- 0.940
- Judge test–retest κ
- 0.81
Scored across 100 ground-truth scenarios by an LLM-as-judge, with the judge itself validated for reliability. 58 backend tests in CI.
What I learned
The agent was under-confident, and the reliability diagram showed it
Plotting a reliability diagram surfaced something counterintuitive: Juno was labelling channels medium or low confidence while actually being correct about 88% of the time. The confidence signal was measuring the wrong thing. Redefining it as rank-certainty — how sure the model is about a channel's position in the ROI ordering, rather than raw uncertainty in the estimate — cut expected calibration error from 0.263 to 0.093.
A crash that looked like success
On a 512 MB host, loading the embedding model got the worker OOM-killed. Because that arrives as SIGKILL rather than an exception, try/except was structurally unable to catch it — and SSE reads a dropped connection as a normal end of stream, so users saw a report that stopped halfway with no error at all. The fix was to select the retrieval backend from the container's own memory limit and add a model-free BM25 retriever for constrained environments.
Juno is a solo portfolio project, not a product with real users. The public demo replays pre-computed answers; running the agent live on your own MMM upload requires your own Anthropic API key.