BetterOffice / evals

v3 · 2026-10-27 · real-world

Method and limits

Complete agent workflows

129 tasks (43 DOCX, 43 XLSX, 43 PPTX) on 90 real-world fixtures, 3 runs per task, complete agent workflows with the same model: generic file tools vs the BetterOffice MCP server.

Models: gpt-6-luna via codex exec. Each task starts in a fresh temporary directory containing only its input. User configuration, rules and persisted sessions are skipped. Runs have a time limit and no approval prompts.

Generic file tools

Shell and Python with python-docx, openpyxl and python-pptx, without network access.

The agent uses shell commands and Python with python-docx, openpyxl and python-pptx in a workspace-write sandbox, with no network or MCP servers. Logged commands are audited for access to the repository or task metadata. Reads are not isolated by the filesystem sandbox.

Shell commands and file edits; one command can run a whole script.

BetterOffice MCP

Only the BetterOffice MCP server for document access.

The agent runs in a read-only sandbox. The required BetterOffice MCP server is rooted at the task directory and is the only writer. Other document access tools are disabled. Code mode can call tools but has no file or network access. Server startup and required tools are checked before tasks run.

BetterOffice MCP calls.

Whole-package checkers

Checkers derive the expected document from the input and the task specification, then compare the whole package. They check content, formatting, formulas and cached results, order, notes, metadata and unrelated parts. The runner also checks tool access, completion and that the input bytes are unchanged.

Equivalent serialization, relationship ids and resolved style indexes are normalized. Calculation chains and the DOCX save timestamp have documented exceptions. Exact scoring rules

Checker validation: All 209 correct outputs accepted, all 140 untouched inputs rejected and all 1267 corrupted outputs rejected.

Pinned real-world fixtures

The v3 manifest pins 90 unmodified public corpus originals by size and SHA-256. Each file has source, publisher, license and attribution metadata. Downloads and local fixtures are verified against those pins.

The basic tier has 96 tasks and the complex tier has 33 tasks. Both tiers use pinned corpus documents; complex tasks include multi-step edits and exact JSON answers to layout questions.

Limits of v3

The v3 results use gpt-6-luna with three independent runs per task in each arm. They measure the agent and tools together on selected corpus tasks. They do not establish performance on other documents or models. Checkers validate defined edits and answers rather than general visual rendering quality.

Each task starts in a fresh temporary directory. Tasks have a 600-second time limit; MCP startup has a separate 60-second preflight limit. The commands run at most two task sessions concurrently.

Tool counts differ by arm. Timings cannot support a speed comparison across arms. Wilson intervals do not account for task selection or dependence between repeated runs, and do not predict results on other document sets.

Tool calls count different things across arms.

  • Generic file tools: Shell commands and file edits; one command can run a whole script.
  • BetterOffice MCP: BetterOffice MCP calls.

Each arm ran in separate batches (Generic file tools: 2026-10-27; BetterOffice MCP: 2026-10-27). Times are not comparable across arms.