Running evals in production, not just in notebooks

Every team I've talked to runs evals in a Jupyter notebook with a curated dataset, ships the prompt change, and discovers in production that the curated dataset didn't reflect real user behavior. We hit this enough times that we rebuilt the pipeline. Here's how it works across all eight Nexobe products.

#Why lab evals lie

Curated eval datasets carry the bias of whoever curated them. Real users hit edge cases the curator didn't imagine, typos, mid-sentence corrections, half-finished thoughts, sarcasm. A prompt that scores 98% on the lab set might score 70% on production traffic.

#Build the harness on real traces

Every production call is logged with input, model, prompt version, and a downstream signal (did the user accept the response? edit it? abandon the session?). The eval harness samples from these real traces, not from a notebook.

• Inputs come from real production transcripts, anonymized
• Expected behaviors are written as assertions, not exact-match strings
• Models are tested in pairs, current shipping vs. proposed change
• Eval results carry a confidence interval based on sample size
• Every promotion to production attaches an eval run ID to the deploy

#Gating shipments on eval results

A prompt change cannot ship if it regresses on the production eval set. Period. Overrides exist but require a written justification on the deploy artifact. The friction is intentional, it's easier to write a better prompt than to write a justification for a regression.

#The tradeoffs nobody mentions

Production evals are expensive. Every prompt change burns tokens at scale. We mitigate with sampling, but the real answer is to accept the cost as part of the build. The alternative, shipping silent regressions to users, is more expensive in trust.

For the broader story of how we ship reliably at this cadence, read Shipping weekly without burning out.