Methodology

The assignments

CivicArena was built by Statecraft to test that distinction. Alex Cohen developed the first assignments from his experience running acquisition support and back-office support for the Bureau of Indian Affairs. Statecraft expanded them into 41 document-based assignments.

Each assignment ends in a checkable decision: whether to pay a contractor, recalculate a benefit or release a record. We build the fictional office files from published worked examples and official rules and data.

An AI agent accesses the files for an office assignment, reads the records and looks up the rules, then gets checked for what it can produce.

The benchmark does not equal the job

An early mistake is the Achilles' heel for everything built after it. As the work continues, context grows: every document opened and tool used creates extra burden. Errors compound if they go uncorrected. Accuracy can decrease as inputs lengthen even for equivalent tasks,¹ and models can carry mistaken assumptions through multi-turn conversations.²

In 2024, WorkArena++ found that agents performed much worse on combined office workflows than on individual software tasks.³ CivicArena tests this problem in government work by asking for the completed assignment, with the records and rules provided for the agent to work through. The results show that completing the whole job remains difficult.

Put the agent in the office

In the contractor-payment assignment, a former employee has repaid $12,000 in relocation expenses, but the contractor's payment request still includes the charge. The reviewer must reconcile that repayment, separate costs that belong in September's accounts from costs eligible for payment now, and apply the contract's fee-retention rules. The contract and relevant Federal Acquisition Regulation provisions are supplied with the records. All seven models passed this assignment.

Each case gets:

• Its own virtual computer, external networking disabled. API calls and credentials are external.
• The same Pi agent harness controls for every model: bash, read, write, edit.
• A submission tool, finalize_case, naming the answer fields and units, never their values. Travel submits a document instead.

Agents are evaluated after the run stops. The submission tool checks the record format only, then separate checkers grade the frozen submission for accuracy.

OSWorld checks the files and computer state an agent leaves behind.⁴ τ-bench checks the final database against the required outcome.⁵ Finally, CivicArena applies that principle to the submitted case.

How we score a run

A task passes only if every required check passes. Checkers can reject valid work or miss a mistake, and the expert review now under way may change both tasks and checks. Unscored runs are not counted as model failures. The travel submissions still need substantive manual review.

One run per task, high reasoning via OpenRouter. Each run has 60 minutes to work and 300 seconds to grade. The original four models ran September 16, Opus 5.5 ran September 25, and Sonnet 5.5 and GPT-6.1 Sol ran September 29. We did not retry attempts.

What the results cover

The website leaderboard covers all 41 assignments and seven models, for 287 attempts. The public repository reports a 33-task government-function subset, for 231 attempts. Those scopes have different pass rates.

The public report excludes tax preparation, insurance and pension-premium review. It also withholds college student-aid verification and hospital payroll pending review of the worker role.

The 33 assignments cover federal accounting, Social Security and SNAP, grant eligibility, federal personnel, military pay, pay and retirement, contracts, funds and FOIA, and relocation travel.

Earlier CivicArena results used different tasks and do not compare. Federal publications in the case libraries are public and may appear in training data.

Two recorded assignments

Grant assignments

A grant assignment asks which applications qualify for funding using Census poverty estimates. A high estimate would be insufficient on its own, because both survey uncertainty and overlapping service areas must be considered.

Fable 5.1 successfully reviews the Census estimates and recommends funding for one application. The recorded run took 13 tool calls and passed its checks.

Watch the recorded grant run. The replay file carries selected tool actions and submitted values, never model reasoning. The document panels are illustrations, not screenshots; the bottom captions follow the recorded calls.

FOIA disclosure review

Fable was assigned to recommend what to release in a public-records FOIA request. On high reasoning, the model flagged a sentence in the visitor directions for a classified-information referral despite already receiving a signed agency review stating that the attachment contained no classified information. Its final recommendation left the request open for that referral.

Read the signed review and submitted decision. The example concerns the classified-information referral, not whether every possible consultation would be improper. It shows an internal recommendation, not an external response or a measured delay. There is no measured human baseline for this task.

How to read the cases

The public repository includes six case packets from the reported set, with a runner that saves the work without scoring it. The other 27 reported assignments and all checkers are private. Selection follows the work assigned, not the fictional employer's name.

Each folder holds the assignment email and the records. .eml files are emails, PDFs open in any reader, and the Word and Excel files are ordinary documents. submission.schema.json defines the answer fields, and the case manifest binds all 149 starting files to their hashes. It also records the selected task version and initial-office fingerprint, tying the published packet to the evaluated assignment.

• Check a contractor's costs and payment requestgov-acct-004
• Recommend grants using uncertain poverty estimatesgov-civ-001
• Recalculate disability and survivor paymentsgov-hr-001
• Check military service records, pay and allowancesgov-mil-001
• Decide what a public-records response should release or withholdgov-syn-005
• Review the same request with different evidencegov-syn-006

Run an assignment

The six published offices can run with your own model key. They use the same Pi tools and submission format as the recorded evaluation. This edition saves the work without scoring it: private checkers and the other 27 reported tasks are not included.

You need Python 3.11+, Node.js 22+ and Docker with ARM64 support. Building the image downloads its pinned Debian packages; the worker has no external network once the run starts. The first build takes considerably longer than starting an office.

Use a new output directory for each attempt. The run keeps tool calls, messages, file versions, the frozen submission and hash-bound receipts. verify checks those records, not the answer. An incomplete or infrastructure-failed run stays labelled as such. Your key stays on the host.

Follow the setup and run instructions. Calling the model provider incurs API charges.

Methods, evidence and what's missing

Grader checks differ in difficulty and can depend on each other; they are not independent steps or tool calls. The context studies do not measure context effects in these runs. OSWorld, WorkArena++ and τ-bench provide related evaluation methods, not independent validation of CivicArena's tasks or scores.

The public report's compounding illustration uses pⁿ: independent errors, fixed per-step accuracy, no correction. It is not fitted to CivicArena results.

Recorded API cost estimates are not provider invoices; the GPT-6.1 Sol estimate uses base-tier prices. Recorded conversation spans are not full VM runtimes.

The public worker loads only the six published packets and validates their bytes and submission schemas. It does not contain private answer keys or graders. Release checks verify the file inventory, case hashes, result consistency and reproducible visuals. They do not call a model or start a VM.

Read the public results file or review the evidence and release checks.

References

1. Chroma Research. Context Rot: How Increasing Input Tokens Impacts LLM Performance.
2. Laban et al. LLMs Get Lost in Multi-Turn Conversation.
3. Boisvert et al. WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks. Sections 3 and 4.3, Table 2.
4. Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. Sections 2.2.3 and 3.2.
5. Yao et al. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Section 3, task reward.

Text adapted from the public README. Statecraft-authored documentation is CC BY 4.0. Notices and terms.