Government work is highly specialized and hard. Can frontier AI finish the work government depends on? CivicArena tests how AI agents handle government and regulated work.
September results · Updated September 29, 2026. Earlier Sol attempts informed task revisions.
Astra passed 28 of 39 scored tasks. GPT-6.1 Sol passed 25 of 38. The other 5 models failed a majority of their scored assignments. These results measure this development set, not government-work accuracy as a whole.
Post on X ↗ A pass requires every required check.
Pass rates exclude unscored attempts.
Government work is a chain of decisions, calculations, and documents. Getting individual steps right isn’t enough if an error carries through to the final result. We count a task as passed only when the submission meets every required check. The breakdown below shows where models fell short, including downstream checks blocked by earlier errors.
High reasoning. A 60-minute work limit per attempt. Each model made one attempt per task. Different unscored counts produce different denominators. Choose “Common scored tasks only” to compare the same tasks.
| Model | Passed | Failed | Unscored | Pass rate |
|---|---|---|---|---|
| GPT-6 Astra | 28 | 11 | 2 | 71.8% 28/39 |
| GPT-6.1 Sol | 25 | 13 | 3 | 65.8% 25/38 |
| Claude Opus 5.5 | 17 | 23 | 1 | 42.5% 17/40 |
| GPT-5.6 Sol | 17 | 24 | 0 | 41.5% 17/41 |
| Claude Fable 5.1 | 16 | 24 | 1 | 40.0% 16/40 |
| Claude Opus 5 | 14 | 25 | 2 | 35.9% 14/39 |
| Claude Sonnet 5.5 | 12 | 25 | 4 | 32.4% 12/37 |
Only tasks with a score for every
selected model enter this chart.
The overall ranking doesn't tell you whether models fail the same assignments. Each bar counts tasks by how many selected models passed. We leave out a task if any selected model lacks a score.
31 comparable tasks. This chart excludes 10 tasks with at least one unscored attempt.
Failed / scored tasks.
Government work demands more than a simple answer. A single assignment can require reconciling conflicting rules, carrying accurate information through multiple files and steps, and producing a result that holds up to review. These results show where frontier models fall short and why Statecraft is building AI specifically for government work.
| Work family | Astra | Sol 6.1 | Opus 5.5 | Sol 5.6 | Fable 5.1 | Opus 5 | Sonnet 5.5 |
|---|---|---|---|---|---|---|---|
| Federal accounting10 tasks | 3/10 | 3/10 | 6/10 | 4/10 | 4/9 1 unscored | 4/10 | 5/9 1 unscored |
| Benefits3 tasks | 0/3 | 0/3 | 1/3 | 1/3 | 0/3 | 2/3 | 1/1 2 unscored |
| Grant eligibility1 task | 0/1 | 0/1 | 0/1 | 0/1 | 0/1 | 0/1 | 0/1 |
| Federal personnel2 tasks | 0/2 | 0/2 | 0/2 | 1/2 | 0/2 | 0/1 1 unscored | 0/2 |
| Insurance programs2 tasks | 0/2 | 0/2 | 1/2 | 0/2 | 1/2 | 1/2 | 1/2 |
| Military pay1 task | 0/1 | 0/1 | 0/1 | 0/1 | 0/1 | 0/1 | 0/1 |
| Pay and retirement3 tasks | 0/3 | 2/3 | 1/3 | 3/3 | 2/3 | 1/2 1 unscored | 2/2 1 unscored |
| Contracts, funds and FOIA14 tasks | 5/12 2 unscored | 5/12 2 unscored | 10/14 | 12/14 | 13/14 | 12/14 | 11/14 |
| Tax review4 tasks | 2/4 | 3/4 | 4/4 | 2/4 | 3/4 | 4/4 | 4/4 |
| Relocation travel1 task | 1/1 | No score 1 unscored | No score 1 unscored | 1/1 | 1/1 | 1/1 | 1/1 |
Some families contain one task. These results do not estimate performance across an entire profession.
Click a cell for check counts
or the reason we left it unscored.
Each cell shows passed checks / required checks. Green means every check passed; coral means at least one failed. Grey marks an unscored attempt. Select a cell for details.
| Task | Astra | Sol 6.1 | Opus 5.5 | Sol 5.6 | Fable 5.1 | Opus 5 | Sonnet 5.5 |
|---|---|---|---|---|---|---|---|
| gov-acct-002 | |||||||
| gov-acct-003 | |||||||
| gov-acct-004 | |||||||
| gov-acct-005 | |||||||
| gov-acct-006 | |||||||
| gov-acct-007 | |||||||
| gov-acct-008 | |||||||
| gov-acct-009 | |||||||
| gov-acct-010 | |||||||
| gov-acct-011 | |||||||
| gov-ben-001 | |||||||
| gov-ben-002 | |||||||
| gov-ben-003 | |||||||
| gov-civ-001 | |||||||
| gov-hr-001 | |||||||
| gov-hr-002 | |||||||
| gov-ins-001 | |||||||
| gov-ins-002 | |||||||
| gov-mil-001 | |||||||
| gov-pay-001 | |||||||
| gov-pay-002 | |||||||
| gov-pay-003 | |||||||
| gov-syn-001 | |||||||
| gov-syn-002 | |||||||
| gov-syn-003 | |||||||
| gov-syn-004 | |||||||
| gov-syn-005 | |||||||
| gov-syn-006 | |||||||
| gov-syn-007 | |||||||
| gov-syn-008 | |||||||
| gov-syn-009 | |||||||
| gov-syn-010 | |||||||
| gov-syn-011 | |||||||
| gov-syn-012 | |||||||
| gov-syn-013 | |||||||
| gov-syn-014 | |||||||
| gov-tax-001 | |||||||
| gov-tax-002 | |||||||
| gov-tax-003 | |||||||
| gov-tax-004 | |||||||
| gov-trv-001 |
Task-search filters affect this matrix only. The comparison controls above apply to every chart and export.
Statecraft builds these assignments around the work government teams actually handle. We use them to identify where general-purpose models fall short and guide the AI we are building specifically for government.
Government work rarely arrives as one tidy question. Models must reconcile information across files, apply the relevant rules and carry those decisions into a final submission.
One wrong decision can affect everything that follows. A task passes only when the final submission meets every required check. Partial success helps us understand a failure, but does not turn it into completed work.
An explanation can sound convincing while the underlying work is wrong. These assignments require models to produce a checkable result, such as a payment calculation, benefit determination or records response.
Get an email when we add model results.
Updates only. No marketing, easy unsubscribe.
Submit a model, propose a workflow or contact our team.
Share the model name, version and endpoint. We will contact you about an evaluation.
