Evaluation
What we guarantee, and what we merely measure.
Two kinds of promise live in this product, and putting them in one table would be a quiet lie. So they are in two.
Reproducible-exact
deterministicThese are enforced by code that cannot be argued with. A budget's line items sum to its total, or BUDGET_SUM_MISMATCH fails the artifact. There is no p95 here and no “usually” — it is arithmetic, and it holds every time or the pack is marked failed and says so on its own page. Publishing these as a percentage would imply they could come out otherwise.
- Every artifact carries its Tribunal report
- Pass or fail, the grade ships inside the artifact. There is no path through the code that returns work without one.
- No hard check fails inside a passing pack
- A hard failure forces pass:false even when the critic loved it. No model can talk its way past arithmetic.
- Undelivered work is declared, never dropped
- An artifact the provider refused to make ships as an undelivered stub with the reason. It is never quietly removed — a thinner pack must never score better than a complete one.
Enforced by 8 hard checks: SCHEMA_INVALID, POLICY_VIOLATION, SOURCE_MISSING, BUDGET_SUM_MISMATCH, SCHEDULE_OVERLAP, DATE_INVALID, DIM_ASPECT_MISMATCH, PLACEHOLDER_TEXT. All of them are published in full.
Measured-with-variance
a model is involvedEverything below depends on a model, four external providers, and the internet. So it gets a median and a range, with the number of runs printed beside it. A single figure here would claim a precision we have not earned — and n = 2 is not a distribution, so we say that rather than dress it up as one.
| Tool | Runs | Latency (median) | Range | Passed the Tribunal |
|---|---|---|---|---|
| Write a toastoce_write_toast | 3 | 12.1s | 9.4–13.7s | 100% |
| Plan an occasionoce_plan_occasion | 3 | 105s | 96.9–107.9s | 100% (80–100%) |
| Make a moodboardoce_moodboard | 2 | 56.5s | 36.8–76.1s | 100% |
| Make a keepsakeoce_make_keepsake | 2 | 100.5s | 78.5–122.5s | 100% |
| Design an invitationoce_design_invite | 2 | 70.4s | 68.6–72.2s | 100% |
| Launch kitoce_launch_kit | 2 | 337.8s | 194.3–481.3s | 75% (50–100%) |
14 real runs, measured 2026-07-14, costing $2.61 of genuine provider spend. Reproduce with node scripts/slo.mjs.
A low pass rate is not always the critic. The launch kit renders four images in a burst, and when a provider throttles one, the pack ships that piece as a declared undelivered stub rather than dropping it — so the pack is thinner, and the artifacts that were made are still graded and still pass. That is the range working as designed: honest about what got made, never quietly shrinking the pack to flatter the number.
The pass rate used to be the least trustworthy number here
We measured the critic against itself: the same artifact, graded six times, no code changed in between. One schedule came back F P F F P F — its grounding score oscillating between 62 and 72, straddling the passing floor of 70.
A standard that scores the identical artifact differently on Tuesday than on Wednesday is not a standard. It is a mood. So it was fixed before any of the numbers above were measured — there is no point publishing a spread you have not first tried to shrink.
The critic now runs at temperature 0 against anchored scoring bands, the question that was doing all the drifting was moved out of the model and into a deterministic check, and a correctness axis may only fall below its floor if the critic can quote the exact defect. An uncited correctness failure is discarded and the score restored.
And the bar did not move. Verified against known-bad work: pure slop still fails (composition 30, grounding 30), and an invented “$49 per event, 12,000 hosts, 99.4% satisfaction” still fails on grounding 30. Stable and discriminating — a critic that never fails anything would be worse than one that varies. Reproduce with node scripts/critic-variance.mjs.
Everything on this page is reproducible from the repository, and every number was measured rather than asserted. Where a number is thin, it says so. The standard itself is published in full, generated from the same constants the engine runs.