Complex: 36.4% pass rate · 95% Wilson interval: 27.6% to 46.2% · n = 99 runs · 33 tasks · all runs pass: 5/33 tasks (15.2%)
Basic: 61.1% pass rate · 95% Wilson interval: 55.4% to 66.6% · n = 288 runs · 96 tasks · all runs pass: 40/96 tasks (41.7%)
v3 · 2026-10-27 · real-world
129 tasks (43 DOCX, 43 XLSX, 43 PPTX) on 90 real-world fixtures, 3 runs per task, complete agent workflows with the same model: generic file tools vs the BetterOffice MCP server.
A task passes only if the requested edit is correct, the rest of the file is unchanged, and the input is untouched.
"Nothing else in the file moved" matters because a correct-looking edit can still erase formatting, formula results, paragraph ids or speaker notes. Whole-package checks catch changes a quick visual check can miss.
| Format | Arm | Passes / runs | Pass rate |
|---|---|---|---|
| DOCX | Generic file tools · Complex | 2/33 | 6.1% |
| DOCX | BetterOffice MCP · Complex | 19/33 | 57.6% |
| DOCX | Generic file tools · Basic | 43/96 | 44.8% |
| DOCX | BetterOffice MCP · Basic | 76/96 | 79.2% |
| XLSX | Generic file tools · Complex | 13/33 | 39.4% |
| XLSX | BetterOffice MCP · Complex | 31/33 | 93.9% |
| XLSX | Generic file tools · Basic | 49/96 | 51.0% |
| XLSX | BetterOffice MCP · Basic | 94/96 | 97.9% |
| PPTX | Generic file tools · Complex | 21/33 | 63.6% |
| PPTX | BetterOffice MCP · Complex | 31/33 | 93.9% |
| PPTX | Generic file tools · Basic | 84/96 | 87.5% |
| PPTX | BetterOffice MCP · Basic | 92/96 | 95.8% |
| Total | Generic file tools · Complex | 36/99 | 36.4% |
| Total | BetterOffice MCP · Complex | 81/99 | 81.8% |
| Total | Generic file tools · Basic | 176/288 | 61.1% |
| Total | BetterOffice MCP · Basic | 262/288 | 91.0% |
Complex: 36.4% pass rate · 95% Wilson interval: 27.6% to 46.2% · n = 99 runs · 33 tasks · all runs pass: 5/33 tasks (15.2%)
Basic: 61.1% pass rate · 95% Wilson interval: 55.4% to 66.6% · n = 288 runs · 96 tasks · all runs pass: 40/96 tasks (41.7%)
Complex: 81.8% pass rate · 95% Wilson interval: 73.1% to 88.2% · n = 99 runs · 33 tasks · all runs pass: 25/33 tasks (75.8%)
Basic: 91.0% pass rate · 95% Wilson interval: 87.1% to 93.8% · n = 288 runs · 96 tasks · all runs pass: 83/96 tasks (86.5%)
Complex: 6.1% pass rate · 95% Wilson interval: 1.7% to 19.6% · n = 33 runs · 11 tasks · all runs pass: 0/11 tasks (0.0%)
Basic: 44.8% pass rate · 95% Wilson interval: 35.2% to 54.7% · n = 96 runs · 32 tasks · all runs pass: 8/32 tasks (25.0%)
Complex: 57.6% pass rate · 95% Wilson interval: 40.8% to 72.8% · n = 33 runs · 11 tasks · all runs pass: 5/11 tasks (45.5%)
Basic: 79.2% pass rate · 95% Wilson interval: 70.0% to 86.1% · n = 96 runs · 32 tasks · all runs pass: 24/32 tasks (75.0%)
Complex: 39.4% pass rate · 95% Wilson interval: 24.7% to 56.3% · n = 33 runs · 11 tasks · all runs pass: 0/11 tasks (0.0%)
Basic: 51.0% pass rate · 95% Wilson interval: 41.2% to 60.8% · n = 96 runs · 32 tasks · all runs pass: 8/32 tasks (25.0%)
Complex: 93.9% pass rate · 95% Wilson interval: 80.4% to 98.3% · n = 33 runs · 11 tasks · all runs pass: 10/11 tasks (90.9%)
Basic: 97.9% pass rate · 95% Wilson interval: 92.7% to 99.4% · n = 96 runs · 32 tasks · all runs pass: 30/32 tasks (93.8%)
Complex: 63.6% pass rate · 95% Wilson interval: 46.6% to 77.8% · n = 33 runs · 11 tasks · all runs pass: 5/11 tasks (45.5%)
Basic: 87.5% pass rate · 95% Wilson interval: 79.4% to 92.7% · n = 96 runs · 32 tasks · all runs pass: 24/32 tasks (75.0%)
Complex: 93.9% pass rate · 95% Wilson interval: 80.4% to 98.3% · n = 33 runs · 11 tasks · all runs pass: 10/11 tasks (90.9%)
Basic: 95.8% pass rate · 95% Wilson interval: 89.8% to 98.4% · n = 96 runs · 32 tasks · all runs pass: 29/32 tasks (90.6%)
Intervals describe uncertainty in this small sample. They do not establish performance on other documents. Method and limits
| Version | Date | Fixtures | Tasks | Runs per task per model | Models | Arms and pass rates | JSON |
|---|---|---|---|---|---|---|---|
| v3 | 2026-10-27 | real-world | 129 | 3 runs per task | gpt-6-luna | gpt-6-luna · Basic · Generic file tools: 176/288 (61.1%) gpt-6-luna · Basic · BetterOffice MCP: 262/288 (91.0%) gpt-6-luna · Complex · Generic file tools: 36/99 (36.4%) gpt-6-luna · Complex · BetterOffice MCP: 81/99 (81.8%) | Download JSON |
| v2 | 2026-10-15 | synthetic | 30 | one run per task | gpt-6-luna | gpt-6-luna · Generic file tools: 14/30 (46.7%) gpt-6-luna · BetterOffice MCP: 30/30 (100.0%) | Download JSON |