46.7% pass rate
95% Wilson interval: 30.2% to 63.9%
n = 30 runs · 30 tasks
one run per task
v2 · 2026-10-15 · synthetic
Agent editing evals
30 synthetic fixtures (10 DOCX, 10 XLSX, 10 PPTX), one run per task, complete agent workflows with the same model: generic file tools vs the BetterOffice MCP server.
A task passes only if the requested edit is correct, the rest of the file is unchanged, and the input is untouched.
"Nothing else in the file moved" matters because a correct-looking edit can still erase formatting, formula results, paragraph ids or speaker notes. Whole-package checks catch changes a quick visual check can miss.
Results · gpt-6-luna
100.0% pass rate
95% Wilson interval: 88.6% to 100.0%
n = 30 runs · 30 tasks
one run per task
60.0% pass rate
95% Wilson interval: 31.3% to 83.2%
n = 10 runs · 10 tasks
one run per task
100.0% pass rate
95% Wilson interval: 72.2% to 100.0%
n = 10 runs · 10 tasks
one run per task
0.0% pass rate
95% Wilson interval: 0.0% to 27.8%
n = 10 runs · 10 tasks
one run per task
100.0% pass rate
95% Wilson interval: 72.2% to 100.0%
n = 10 runs · 10 tasks
one run per task
80.0% pass rate
95% Wilson interval: 49.0% to 94.3%
n = 10 runs · 10 tasks
one run per task
100.0% pass rate
95% Wilson interval: 72.2% to 100.0%
n = 10 runs · 10 tasks
one run per task
Intervals describe uncertainty in this small sample. They do not establish performance on other documents. Method and limits
Suites
| Version | Date | Fixtures | Tasks | Runs per task per model | Models | Arms and pass rates | JSON |
|---|---|---|---|---|---|---|---|
| v2 | 2026-10-15 | synthetic | 30 | one run per task | gpt-6-luna | gpt-6-luna · Generic file tools: 14/30 (46.7%) gpt-6-luna · BetterOffice MCP: 30/30 (100.0%) | Download JSON |