What got worse, or better, when the design system, the context, the graders or the output cap changed. Each model at each context is compared with its previous version, on the tasks both ran unchanged. A change counts only when the two versions' repeats do not overlap and the means differ by more than a floor; repeats are the measure of noise.
Nov 10, 02:04 → Nov 10, 11:13, on 3 tasks. Changed: context, graders, output cap. Nothing moved beyond the noise.
| Metric | Before (repeats) | After (repeats) | Verdict |
|---|---|---|---|
| Design-system violations per run | 0.1 (0.0–0.3) | 0.0 (0.0–0.0) | same |
| Quality violations per run | 14.7 (4.0–17.3) | 9.3 (7.0–11.7) | same |
| Share of runs done | 19% (0%–33%) | 33% (33%–33%) | same |
| Cost per run | $0.0000 ($0.0000–$0.0000) | $0.0000 ($0.0000–$0.0000) | same |
typecheck/TS2304: 1.6 → 0.0 per run
lint/react(jsx-no-undef): 0.9 → 0.0 per run
lint/typescript(no-unsafe-type-assertion): 0.8 → 0.0 per run
typecheck/TS2375: 0.7 → 0.0 per run
typecheck/TS2339: 0.7 → 0.0 per run
Nov 10, 15:02 → Nov 10, 15:59, on 2 tasks. Changed: context, graders. Nothing moved beyond the noise.
| Metric | Before (repeats) | After (repeats) | Verdict |
|---|---|---|---|
| Design-system violations per run | 0.0 (0.0–0.0) | 0.0 (0.0–0.0) | unknownone batch on each side: no measure of noise |
| Quality violations per run | 6.5 (6.5–6.5) | 5.0 (5.0–5.0) | unknownone batch on each side: no measure of noise |
| Share of runs done | 50% (50%–50%) | 0% (0%–0%) |
Nov 10, 15:05 → Nov 10, 16:06, on 2 tasks. Changed: context, graders. Nothing moved beyond the noise.
| Metric | Before (repeats) | After (repeats) | Verdict |
|---|---|---|---|
| Design-system violations per run | 0.0 (0.0–0.0) | 0.0 (0.0–0.0) | unknownone batch on each side: no measure of noise |
| Quality violations per run | 0.0 (0.0–0.0) | 1.0 (1.0–1.0) | unknownone batch on each side: no measure of noise |
| Share of runs done | 0% (0%–0%) | 0% (0%–0%) |
Nov 10, 15:00 → Nov 10, 15:57, on 2 tasks. Changed: context, graders.
| Metric | Before (repeats) | After (repeats) | Verdict |
|---|---|---|---|
| Design-system violations per run | 13.5 (9.0–18.0) | 5.0 (5.0–5.0) | better |
| Quality violations per run | 13.8 (11.0–16.5) | 25.5 (25.5–25.5) | worse |
| Share of runs done | 0% (0%–0%) | 0% (0%–0%) | same |
Nov 10, 02:12 → Nov 10, 11:15, on 3 tasks. Changed: context, graders, output cap. Nothing moved beyond the noise.
| Metric | Before (repeats) | After (repeats) | Verdict |
|---|---|---|---|
| Design-system violations per run | 0.0 (0.0–0.0) | 0.0 (0.0–0.0) | same |
| Quality violations per run | 2.4 (0.7–3.3) | 3.0 (2.0–4.0) | same |
| Share of runs done | 92% (67%–100%) | 50% (33%–67%) | same |
Nov 10, 15:09 → Nov 10, 15:58, on 2 tasks. Changed: context, graders. Nothing moved beyond the noise.
| Metric | Before (repeats) | After (repeats) | Verdict |
|---|---|---|---|
| Design-system violations per run | 0.0 (0.0–0.0) | 0.0 (0.0–0.0) | unknownone batch on each side: no measure of noise |
| Quality violations per run | 2.0 (2.0–2.0) | 2.0 (2.0–2.0) | unknownone batch on each side: no measure of noise |
| Share of runs done | 100% (100%–100%) | 100% (100%–100%) |
Nov 10, 15:14 → Nov 10, 16:04, on 2 tasks. Changed: context, graders. Nothing moved beyond the noise.
| Metric | Before (repeats) | After (repeats) | Verdict |
|---|---|---|---|
| Design-system violations per run | 0.0 (0.0–0.0) | 0.0 (0.0–0.0) | unknownone batch on each side: no measure of noise |
| Quality violations per run | 0.0 (0.0–0.0) | 1.5 (1.5–1.5) | unknownone batch on each side: no measure of noise |
| Share of runs done | 100% (100%–100%) | 0% (0%–0%) |
Nov 10, 15:00 → Nov 10, 16:09, on 2 tasks. Changed: context, graders.
| Metric | Before (repeats) | After (repeats) | Verdict |
|---|---|---|---|
| Design-system violations per run | 27.3 (26.0–28.5) | 12.5 (12.5–12.5) | better |
| Quality violations per run | 10.0 (2.5–17.5) | 6.0 (6.0–6.0) | same |
| Share of runs done | 25% (0%–50%) | 50% (50%–50%) | same |
| Cost per run |
Run the same task again and the numbers move. For each model and context, in its latest version: the mean, and the mean gap between the best and the worst run of the same task. A difference smaller than that gap says nothing.
| Model | Context | Runs | Design system ± gap | Done ± gap |
|---|---|---|---|---|
| claude-haiku-5.5 | ds-mcp | 16 | 1.0 ± 0.8 | 100% ± 0% |
| claude-haiku-5.5 | ds-mcp-check | 16 | 0.0 ± 0.0 | 100% ± 0% |
| claude-haiku-5.5 | ds-mcp-check-full | 15 | 0.2 ± 0.6 |
What the runs were given and measured with, newest first: the design system, the context (the level's prompt, tools and the code behind them), the graders and the output cap. Runs are compared only within a series, version against version.
| Since | Design system | Context | Graders | Output cap | Runs | Batch labels |
|---|---|---|---|---|---|---|
| Nov 10, 16:09 | cdb5d9e7 | none e2fbfef8 | 73701208 | 20,000 | 5 | behaviour-v3 |
| Nov 10, 16:04 | cdb5d9e7 | genui-spec b177d804 | 73701208 | 20,000 | 10 |
| unknownone batch on each side: no measure of noise |
| Cost per run | $0.0000 ($0.0000–$0.0000) | $0.0000 ($0.0000–$0.0000) | unknownone batch on each side: no measure of noise |
typecheck/TS2322: 1.5 → 0.5 per run
typecheck/TS2532: 1.5 → 0.5 per run
typecheck/TS2345: 0.5 → 0.0 per run
typecheck/TS2375: 0.5 → 0.0 per run
lint/eslint(no-unused-vars): 0.5 → 0.0 per run
| Cost per run | $0.0000 ($0.0000–$0.0000) | $0.0000 ($0.0000–$0.0000) | unknownone batch on each side: no measure of noise |
behaviour/more-than-ordered: 0.0 → 0.5 per run
behaviour/shows-difference: 0.0 → 0.5 per run
| Cost per run |
| $0.0000 ($0.0000–$0.0000) |
| $0.0000 ($0.0000–$0.0000) |
| same |
typecheck/TS1351: 7.5 → 0.0 per run
typecheck/TS1005: 0.0 → 7.0 per run
typecheck/TS1003: 0.0 → 5.0 per run
outside-styling/styled-from-outside: 4.0 → 0.0 per run
typecheck/TS1128: 0.0 → 3.5 per run
| Cost per run |
| $0.0000 ($0.0000–$0.0000) |
| $0.0000 ($0.0000–$0.0000) |
| same |
a11y/landmark-one-main: 0.0 → 1.0 per run
typecheck/TS2345: 0.7 → 0.0 per run
lint/syntax-error: 0.1 → 0.5 per run
typecheck/TS1443: 0.5 → 0.2 per run
typecheck/TS1434: 0.0 → 0.3 per run
| Cost per run | $0.0000 ($0.0000–$0.0000) | $0.0000 ($0.0000–$0.0000) | unknownone batch on each side: no measure of noise |
| Cost per run | $0.0000 ($0.0000–$0.0000) | $0.0000 ($0.0000–$0.0000) | unknownone batch on each side: no measure of noise |
behaviour/more-than-ordered: 0.0 → 0.5 per run
behaviour/empty-submit: 0.0 → 0.5 per run
behaviour/shows-difference: 0.0 → 0.5 per run
| $0.0000 ($0.0000–$0.0000) |
| $0.0000 ($0.0000–$0.0000) |
| same |
raw-values/raw-length: 15.0 → 2.5 per run
lint/@pandacss(no-physical-properties): 4.3 → 0.5 per run
token-names/unknown-token: 3.5 → 1.0 per run
components/native-element: 2.0 → 3.0 per run
typecheck/TS1005: 1.3 → 0.5 per run
| 100% ± 0% |
| claude-haiku-5.5 | none | 16 | 36.2 ± 17.4 | 93% ± 20% |
| claude-haiku-5.5 | storybook-mcp | 15 | 4.3 ± 11.8 | 100% ± 0% |
| deepseek-v4.1-flash-1017 | ds-mcp | 6 | 0.2 ± 0.0 | 100% ± 0% |
| deepseek-v4.1-flash-1017 | ds-mcp-check | 5 | 0.0 | 100% |
| deepseek-v4.1-flash-1017 | ds-mcp-check-full | 5 | 0.0 | 100% |
| deepseek-v4.1-flash-1017 | none | 5 | 94.6 | 80% |
| deepseek-v4.1-flash-1017 | storybook-mcp | 5 | 0.4 | 100% |
| kimi-k3.1-pro | ds-mcp | 6 | 0.0 ± 0.0 | 100% ± 0% |
| kimi-k3.1-pro | ds-mcp-check | 5 | 0.0 | 100% |
| kimi-k3.1-pro | ds-mcp-check-full | 5 | 0.0 | 100% |
| kimi-k3.1-pro | none | 4 | 294.3 | 100% |
| kimi-k3.1-pro | storybook-mcp | 5 | 0.4 | 100% |
| nemotron-3.5-lightning-free | ds-mcp | 5 | 4.0 | 0% |
| nemotron-3.5-lightning-free | ds-mcp-check | 6 | 0.0 ± 0.0 | 33% ± 0% |
| nemotron-3.5-lightning-free | ds-mcp-check-full | 5 | 0.0 | 0% |
| nemotron-3.5-lightning-free | genui-spec | 5 | 0.0 | 20% |
| nemotron-3.5-lightning-free | genui-spec-domain | 5 | 0.0 | 40% |
| nemotron-3.5-lightning-free | none | 5 | 7.8 | 0% |
| nemotron-3.5-lightning-free | storybook-mcp | 31 | 6.0 ± 13.8 | 23% ± 40% |
| gpt-6-luna | ds-mcp | 7 | 2.2 ± 2.0 | 90% ± 50% |
| gpt-6-luna | ds-mcp-check | 5 | 0.0 | 100% |
| gpt-6-luna | ds-mcp-check-full | 5 | 0.0 | 100% |
| gpt-6-luna | none | 5 | 146.2 | 80% |
| gpt-6-luna | storybook-mcp | 5 | 36.2 | 100% |
| beam-free | ds-mcp | 5 | 0.0 | 40% |
| beam-free | ds-mcp-check | 6 | 0.0 ± 0.0 | 50% ± 33% |
| beam-free | ds-mcp-check-full | 5 | 0.0 | 60% |
| beam-free | genui-spec | 5 | 0.0 | 40% |
| beam-free | genui-spec-domain | 5 | 0.0 | 100% |
| beam-free | none | 5 | 34.6 | 40% |
| beam-free | storybook-mcp | 31 | 0.7 ± 1.4 | 87% ± 40% |
| behaviour-v3 |
| Nov 10, 15:58 | cdb5d9e7 | ds-mcp-check-full a35241be | 73701208 | 20,000 | 10 | behaviour-v3 |
| Nov 10, 15:57 | cdb5d9e7 | none c59465d2 | 73701208 | 20,000 | 5 | behaviour-v3 |
| Nov 10, 15:36 | cdb5d9e7 | genui-spec-domain 4d024867 | e43fda76 | 20,000 | 10 | warehouse-domain |
| Nov 10, 15:05 | cdb5d9e7 | genui-spec 713ea2f5 | e43fda76 | 20,000 | 10 | warehouse-free |
| Nov 10, 15:02 | cdb5d9e7 | ds-mcp-check-full f3b2e202 | e43fda76 | 20,000 | 10 | warehouse-free |
| Nov 10, 15:00 | cdb5d9e7 | ds-mcp 456af926 | e43fda76 | 20,000 | 10 | warehouse-free |
| Nov 10, 15:00 | cdb5d9e7 | none 97614f2e | e43fda76 | 20,000 | 20 | warehouse-free |
| Nov 10, 14:26 | cdb5d9e7 | genui-spec 85dccb3b | e43fda76 | 20,000 | 24 | genui-data-3 |
| Nov 10, 14:01 | cdb5d9e7 | genui-spec a69c1476 | e43fda76 | 20,000 | 24 | genui-data-2 |
| Nov 10, 13:43 | cdb5d9e7 | genui-spec 6334644c | e43fda76 | 20,000 | 24 | genui-data |
| Nov 10, 13:25 | cdb5d9e7 | genui-spec 3dff12ca | e43fda76 | 20,000 | 24 | genui-page |
| Nov 10, 12:25 | cdb5d9e7 | genui-spec 23781ed3 | e43fda76 | 20,000 | 24 | genui-forms |
| Nov 10, 11:13 | cdb5d9e7 | ds-mcp-check 8acd4811 | e43fda76 | 20,000 | 12 | graders-v2 |
| Nov 10, 11:12 | cdb5d9e7 | none e2ba161e | e43fda76 | 20,000 | 12 | graders-v2 |
| Nov 10, 07:45 | cdb5d9e7 | genui-spec a1bb6834 | a95174e0 | default | 12 | genui-spec |
| Nov 10, 02:05 | cdb5d9e7 | ds-mcp-check-full eea920bf | a95174e0 | default | 62 | free-pilot, free-rerun |
| Nov 10, 02:03 | cdb5d9e7 | ds-mcp, ds-mcp-check 735ceac5 | a95174e0 | default | 124 | free-pilot, free-rerun |
| Nov 10, 02:02 | cdb5d9e7 | none, storybook-mcp e9482b09 | a95174e0 | default | 124 | free-pilot, free-rerun |
| Nov 9, 09:14 | e02ef397 | ds-mcp-check-full 7f6f63ae | f0a118c0 | default | 10 | deepseek-5tasks, kimi-5tasks |
| Nov 9, 09:00 | e02ef397 | none, storybook-mcp 02632ab9 | f0a118c0 | default | 19 | deepseek-5tasks, kimi-5tasks |
| Nov 9, 08:59 | e02ef397 | ds-mcp, ds-mcp-check f58708b8 | f0a118c0 | default | 22 | smoke, deepseek-5tasks, kimi-5tasks |
| Nov 8, 23:32 | e02ef397 | ds-mcp, ds-mcp-check, ds-mcp-check-full dd7bf891 | d53ec290 | default | 64 | haiku-contexts, haiku-5tasks, luna-5tasks |
| Nov 8, 23:32 | e02ef397 | none, storybook-mcp 1c31b730 | d53ec290 | default | 41 | haiku-contexts, haiku-5tasks, luna-5tasks |
| Nov 8, 22:06 | – | none – | – | default | 3 | – |