Method and limits

30 synthetic fixtures (10 DOCX, 10 XLSX, 10 PPTX), one run per task, complete agent workflows with the same model: generic file tools vs the BetterOffice MCP server.

Models: gpt-6-luna via codex exec. Each task starts in a fresh temporary directory containing only its input. User configuration, rules and persisted sessions are skipped. Runs have a time limit and no approval prompts.

Shell and Python with python-docx, openpyxl and python-pptx, without network access.

The agent uses shell commands and Python with python-docx, openpyxl and python-pptx in a workspace-write sandbox, with no network or MCP servers. Logged commands are audited for access to the repository or task metadata. Reads are not isolated by the filesystem sandbox.

Shell commands and file edits; one command can run a whole script.

Only the BetterOffice MCP server for document access.

The agent runs in a read-only sandbox. The required BetterOffice MCP server is rooted at the task directory and is the only writer. Other document access tools are disabled. Code mode can call tools but has no file or network access. Server startup and required tools are checked before tasks run.

BetterOffice MCP calls.

Checkers derive the expected document from the input and the task specification, then compare the whole package. They check content, formatting, formulas and cached results, order, notes, metadata and unrelated parts. The runner also checks tool access, completion and that the input bytes are unchanged.

Equivalent serialization, relationship ids and resolved style indexes are normalized. Calculation chains and the DOCX save timestamp have documented exceptions. Exact scoring rules

Checker validation: All 65 correct outputs accepted, all 30 untouched inputs rejected and all 157 corrupted outputs rejected.

The v2 generator creates synthetic files with fixed timestamps. DOCX fixtures start as hand-written OOXML, are opened and saved by BetterOffice, and carry paragraph ids. The manifest pins each published file by size and SHA-256. Downloads and local fixtures are verified against those pins.

Regenerated bytes can change when engine output changes. A changed fixture set needs a new version prefix and updated pins. Fixed timestamps do not guarantee identical bytes across engine versions.

The v2 results cover small synthetic fixtures, one run per task and one model. They measure the agent and tools together. They do not establish performance on real-world documents, other models or repeated runs. The checkers validate defined edits rather than visual rendering quality.

Tool counts differ by arm. Timings cannot support a speed comparison across arms. Wilson intervals do not account for task selection or predict results on other document sets.