What the agents are asked to build, product by product on the same design system: pages written like product tickets, each with synthetic data and a hand-written answer that follows every rule.
Hardest first, read only on the 10 series that ran every task (2 models, at each of their contexts), each weighing the same. The 44 that skipped a task are left out, or a task tried by stronger models would look easier. With 30 runs a task or more, each share done is uncertain by about ± 18 points: read close shares as a tie.
| Task | Done | Quality per run | Design system per run | Runs |
|---|---|---|---|---|
| Account details | 43% | 5.1 | 4.3 | 30 |
| Identity verification (KYC) | 47% | 12.7 | 3.6 | 30 |
| SEPA transfer form | 50% | 5.5 | 3.7 | 40 |
| Card payment dispute | 53% | 12.8 | 5.5 | 30 |
| Loan contract detail | 53% | 10.7 | 6.6 | 30 |
| Direct debit mandates | 53% | 6.7 | 8.8 | 30 |
| Transactions | 57% | 6.3 | 5.7 | 30 |
| Notification settings | 57% | 5.3 | 6.4 | 30 |
| Payment approval | 67% | 6.9 | 7.0 | 30 |
| Account statement | 83% | 4.1 | 5.5 | 30 |
One business account: where it sits among the accounts, holder, IBAN with a copy button, balances, the five latest transactions and a link to all of them.
Dispute a card payment: the payment, a 120-day notice, a reason, a description and a required confirmation with per-field errors, then a confirmation in place of the form.
Hardest first, read only on the 10 series that ran every task (2 models, at each of their contexts), each weighing the same. The 6 that skipped a task are left out, or a task tried by stronger models would look easier. With 12 runs a task or more, each share done is uncertain by about ± 29 points: read close shares as a tie.
| Task | Done | Quality per run | Design system per run | Runs |
|---|---|---|---|---|
| Goods receipt | 40% | 5.3 | 2.4 | 12 |
| Stock adjustment | 45% | 4.7 | 6.5 | 12 |
| Pick list | 55% |
A company's direct debit mandates with their last collection and status, a warning for failed collections, and cancellation after a confirmation.
Identity step of account opening: known details read-only, required personal and document fields with per-field errors, confirmation on submit.
Detail page of a personal loan: status, key terms, amount still owed, repayment schedule and signed documents.
Alerts that apply at once with a short confirmation, and a summary email whose frequency and product-news consent are saved with a button.
List of payments waiting for approval; each opens a confirmation dialog with amount, fee and total, and is marked approved after confirming.
Monthly statement of a current account: holder, IBAN, balance and a table of transactions with pending ones marked.
All transactions of a current account, newest first, in three views (all, money in, money out) and ten per page with page navigation.
Form to send a SEPA transfer from one of the user's accounts, with the available balance and an error when the amount is too high.
| 2.5 |
| 4 |
| 12 |
| Shipment detail | 70% | 4.9 | 4.0 | 12 |
| Stock levels | 80% | 2.9 | 3.7 | 12 |
Form to receive a delivery against a purchase order: the quantity received of each line, the dock, and damaged goods.
The pick list of one sales order, in walking order by location, with what is picked and how far along the picker is.
One outbound shipment: carrier and tracking, status, destination, the packages with their weights, and the tracking history.
Form to correct the quantity of one item after a count, with the difference and a required reason.
The stock of one warehouse: on hand, reserved and available for each item, with the ones at or below their reorder point marked.