Inside Otteri's prompt library: schemas, evals, and the long tail

Otteri is an AI command center. One of its core surfaces is a prompt library, saved, versioned, evaluated prompts that users return to. The naive version is a notes app with markdown. The actual product is a database problem wearing UI clothes.

#The schema we settled on

Every prompt has an identity that's separate from its content. A prompt entity is a stable ID; the content is a versioned artifact. Tags, evals, and usage stats hang off the entity, not the version, so renaming or rewriting doesn't lose history.

• Prompt entities are immutable IDs with mutable metadata
• Versions are append-only, every edit creates a new row
• Each version carries an inputs schema and an outputs schema
• Evals attach to the (prompt_id, model_id, version_id) triple
• Usage events are streamed, not aggregated, for replay

#Versioning is non-negotiable

Prompts are code. They have bugs, regressions, and version-dependent behavior. Treating them like notes is the fastest way to ship a quiet regression. Every version we ship is reproducible, we can pin a model, a prompt version, and an input set and get the same output six months later.

#The eval harness

Each prompt has an eval set, a list of (input, expected behavior) pairs. New versions run against the eval set before they can be promoted. Failing evals require an explicit override with a written reason, captured on the version row.

#What we'd do differently

In hindsight, we should have made the eval surface mandatory from day one rather than optional. Optional evals mean most prompts ship without them, and the few that have evals become the only ones we trust. Friction in the right place beats no friction at all.

Read more about Otteri on the product site, or see how this architecture connects to the rest of the Nexobe portfolio.