Evaluations
Scores per suite and model, from the last run of each. Re-running is a request to the eval service.
Pass rate, all suites91%+2 pts since 20 OctMeasured on 27 Oct
Cases run today8806 runs, 1 still running
Regressions2Needs a lookScore below the previous run
Eval cost today$4.12Measured by billing
Pass rate over the last eight days
All suites, dailyPass rate
ratio · Source: eval service, nightly · As of: 2026-10-27T08:00:00Z
Failing cases
Support replies, Opus 5.5Test results4 passed · 2 failed · 6 total
- Refund within 14 days: approves and cites the policyPassed1800ms
- Refund after 14 days: declines politelyPassed1650ms
- Double charge: offers refund, never promises timingFailed2100ms
- Unknown order number: asks for it oncePassed900ms
- Angry tone: stays calm, no apology loopFailed2400ms
- Gift card refund: routes to a personPassed1200ms
Latest runs
| Change | Cases | State | ||||
|---|---|---|---|---|---|---|
| Refund policy | Opus 5.5 | 97% | +2 pts | 78/80 | Complete | |
| Refund policy | Haiku 5.5 | 81% | -5 pts | 65/80 | Complete | |
| Support replies | Opus 5.5 | 93% | +2 pts | 223/240 | Complete | |
| Support replies | Sonnet 5.5 | 88% | -1 pts | 211/240 | Complete | |
| Tool selection | Opus 5.5 | 90% | ±0 | 108/120 | Running | |
| Tool selection | Sonnet 5.5 | — | — | 120/120 | Failed |
1 selected · 1 on this pageSorted by Suite, ascending.
Opus 5.5 against Sonnet 5.5 on Support replies
Same 240 cases, same dayCompare runs
| Field | Opus 5.5 | Sonnet 5.5 |
|---|---|---|
| Goal | Answer 240 support tickets the way the guide says | Answer 240 support tickets the way the guide says |
| Outcome | Succeeded | Partly done |
| Duration | 7 min | 3 min |
| Cost | $2.31 | $0.74 |
| Not done | 17 cases failed | 29 cases failed |