BetterOffice / evals

v3 · 2026-10-27 · real-world

Reproduce the evals

Run the arms

Harness instructions

python3 eval/formats/run.py --suite v3 --runs 3 --jobs 2 "node packages/agents/dist/cli.js" /tmp/evals-v3-basic-bo gpt-6-luna
python3 eval/formats/run.py --suite v3 --runs 3 --jobs 2 --arm generic --python /path/to/venv/bin/python /tmp/evals-v3-basic-generic gpt-6-luna
python3 eval/formats/run.py --suite v3 --tier complex --runs 3 --jobs 2 "node packages/agents/dist/cli.js" /tmp/evals-v3-complex-bo gpt-6-luna
python3 eval/formats/run.py --suite v3 --tier complex --runs 3 --jobs 2 --arm generic --python /path/to/venv/bin/python /tmp/evals-v3-complex-generic gpt-6-luna

Every v3 fixture

The v3 manifest pins 90 unmodified public corpus originals by size and SHA-256. Each file has source, publisher, license and attribution metadata. Downloads and local fixtures are verified against those pins.

The basic tier has 96 tasks and the complex tier has 33 tasks. Both tiers use pinned corpus documents; complex tasks include multi-step edits and exact JSON answers to layout questions.

metadata.json