BetterOffice / evals

v3 · 2026-10-27 · real-world

Agent editing evals

129 tasks (43 DOCX, 43 XLSX, 43 PPTX) on 90 real-world fixtures, 3 runs per task, complete agent workflows with the same model: generic file tools vs the BetterOffice MCP server.

A task passes only if the requested edit is correct, the rest of the file is unchanged, and the input is untouched.

"Nothing else in the file moved" matters because a correct-looking edit can still erase formatting, formula results, paragraph ids or speaker notes. Whole-package checks catch changes a quick visual check can miss.

Results · gpt-6-luna

Pass rate
  • Generic file tools · Complex
  • BetterOffice MCP · Complex
  • Generic file tools · Basic
  • BetterOffice MCP · Basic
Pass rates for gpt-6-luna, shared scale from 0% to 100%
FormatArmPasses / runsPass rate
DOCXGeneric file tools · Complex2/336.1%
DOCXBetterOffice MCP · Complex19/3357.6%
DOCXGeneric file tools · Basic43/9644.8%
DOCXBetterOffice MCP · Basic76/9679.2%
XLSXGeneric file tools · Complex13/3339.4%
XLSXBetterOffice MCP · Complex31/3393.9%
XLSXGeneric file tools · Basic49/9651.0%
XLSXBetterOffice MCP · Basic94/9697.9%
PPTXGeneric file tools · Complex21/3363.6%
PPTXBetterOffice MCP · Complex31/3393.9%
PPTXGeneric file tools · Basic84/9687.5%
PPTXBetterOffice MCP · Basic92/9695.8%
TotalGeneric file tools · Complex36/9936.4%
TotalBetterOffice MCP · Complex81/9981.8%
TotalGeneric file tools · Basic176/28861.1%
TotalBetterOffice MCP · Basic262/28891.0%
Overall · Generic file toolscomplex 36/99 · basic 176/288

Complex: 36.4% pass rate · 95% Wilson interval: 27.6% to 46.2% · n = 99 runs · 33 tasks · all runs pass: 5/33 tasks (15.2%)

Basic: 61.1% pass rate · 95% Wilson interval: 55.4% to 66.6% · n = 288 runs · 96 tasks · all runs pass: 40/96 tasks (41.7%)

Overall · BetterOffice MCPcomplex 81/99 · basic 262/288

Complex: 81.8% pass rate · 95% Wilson interval: 73.1% to 88.2% · n = 99 runs · 33 tasks · all runs pass: 25/33 tasks (75.8%)

Basic: 91.0% pass rate · 95% Wilson interval: 87.1% to 93.8% · n = 288 runs · 96 tasks · all runs pass: 83/96 tasks (86.5%)

DOCX · Generic file toolscomplex 2/33 · basic 43/96

Complex: 6.1% pass rate · 95% Wilson interval: 1.7% to 19.6% · n = 33 runs · 11 tasks · all runs pass: 0/11 tasks (0.0%)

Basic: 44.8% pass rate · 95% Wilson interval: 35.2% to 54.7% · n = 96 runs · 32 tasks · all runs pass: 8/32 tasks (25.0%)

DOCX · BetterOffice MCPcomplex 19/33 · basic 76/96

Complex: 57.6% pass rate · 95% Wilson interval: 40.8% to 72.8% · n = 33 runs · 11 tasks · all runs pass: 5/11 tasks (45.5%)

Basic: 79.2% pass rate · 95% Wilson interval: 70.0% to 86.1% · n = 96 runs · 32 tasks · all runs pass: 24/32 tasks (75.0%)

XLSX · Generic file toolscomplex 13/33 · basic 49/96

Complex: 39.4% pass rate · 95% Wilson interval: 24.7% to 56.3% · n = 33 runs · 11 tasks · all runs pass: 0/11 tasks (0.0%)

Basic: 51.0% pass rate · 95% Wilson interval: 41.2% to 60.8% · n = 96 runs · 32 tasks · all runs pass: 8/32 tasks (25.0%)

XLSX · BetterOffice MCPcomplex 31/33 · basic 94/96

Complex: 93.9% pass rate · 95% Wilson interval: 80.4% to 98.3% · n = 33 runs · 11 tasks · all runs pass: 10/11 tasks (90.9%)

Basic: 97.9% pass rate · 95% Wilson interval: 92.7% to 99.4% · n = 96 runs · 32 tasks · all runs pass: 30/32 tasks (93.8%)

PPTX · Generic file toolscomplex 21/33 · basic 84/96

Complex: 63.6% pass rate · 95% Wilson interval: 46.6% to 77.8% · n = 33 runs · 11 tasks · all runs pass: 5/11 tasks (45.5%)

Basic: 87.5% pass rate · 95% Wilson interval: 79.4% to 92.7% · n = 96 runs · 32 tasks · all runs pass: 24/32 tasks (75.0%)

PPTX · BetterOffice MCPcomplex 31/33 · basic 92/96

Complex: 93.9% pass rate · 95% Wilson interval: 80.4% to 98.3% · n = 33 runs · 11 tasks · all runs pass: 10/11 tasks (90.9%)

Basic: 95.8% pass rate · 95% Wilson interval: 89.8% to 98.4% · n = 96 runs · 32 tasks · all runs pass: 29/32 tasks (90.6%)

Intervals describe uncertainty in this small sample. They do not establish performance on other documents. Method and limits

Suites

Every published evaluation suite
VersionDateFixturesTasksRuns per task per modelModelsArms and pass ratesJSON
v32026-10-27real-world1293 runs per taskgpt-6-luna

gpt-6-luna · Basic · Generic file tools: 176/288 (61.1%)

gpt-6-luna · Basic · BetterOffice MCP: 262/288 (91.0%)

gpt-6-luna · Complex · Generic file tools: 36/99 (36.4%)

gpt-6-luna · Complex · BetterOffice MCP: 81/99 (81.8%)

Download JSON
v22026-10-15synthetic30one run per taskgpt-6-luna

gpt-6-luna · Generic file tools: 14/30 (46.7%)

gpt-6-luna · BetterOffice MCP: 30/30 (100.0%)

Download JSON