BetterOffice / evals

v2 · 2026-10-15 · synthetic

DOCX tasks

gpt-6-luna

DOCX results for gpt-6-luna
TaskGeneric file toolsBetterOffice MCP
Fix a misspelled heading

docx-01-fix-heading

Task prompt

In input.docx, the heading Sumary is misspelled. Change it to Summary. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Pass

Run 1 · 2026-10-15

2 tool calls · 9.35 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Pass

Run 1 · 2026-10-15

6 tool calls · 21.50 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Rename a project everywhere

docx-02-replace-term

Task prompt

In input.docx, replace every occurrence of the project name Atlas with Beacon, in the title, the body text, the list and the table. The match is case-sensitive, so leave the lowercase word atlas unchanged. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Pass

Run 1 · 2026-10-15

3 tool calls · 10.10 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Pass

Run 1 · 2026-10-15

7 tool calls · 16.89 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Correct a table value

docx-03-correct-cell

Task prompt

In input.docx, correct the Budget table: the Actual amount for Supplies is 130, not 100, so the Actual total is 660, not 630. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-15

2 tool calls · 9.01 seconds

python-docx cell.text replaced the edited cell paragraphs and dropped their paragraph ids, right alignment, run size and total bold formatting. Cause: library.

Checker message

document.body[11].rows[3].cells[4].blocks[1].attributes: fields differ

Pass

Run 1 · 2026-10-15

12 tool calls · 26.46 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Add a table row above the total

docx-04-add-row

Task prompt

In input.docx, add a row to the Budget table directly above the Total row: Item Training, Owner Avery, Planned 80, Actual 75. Format it like the other expense rows, not like the bold Total row. Then update the Total row to Planned 700 and Actual 705. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-15

2 tool calls · 14.98 seconds

The model copied a template row with its paragraph ids intact, duplicating them, and added an empty run to the Total row. Cause: model.

Checker message

document.body[11].rows[5].cells[1].blocks[1].attributes: paraId reuses an input paragraph id

Pass

Run 1 · 2026-10-15

10 tool calls · 22.08 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Delete a table row

docx-05-delete-row

Task prompt

In input.docx, the Supplies line was cancelled. Delete its row from the Budget table and update the Total row to Planned 500 and Actual 530. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-15

2 tool calls · 9.10 seconds

python-docx cell.text replaced the total cell paragraphs and dropped their paragraph ids, right alignment, run size and bold formatting. Cause: library.

Checker message

document.body[11].rows[4].cells[3].blocks[1].attributes: fields differ

Pass

Run 1 · 2026-10-15

12 tool calls · 31.31 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Promote a paragraph to a heading

docx-06-heading-style

Task prompt

In input.docx, make the paragraph Risks and mitigations a Heading 2, a subsection heading under Next steps, using the document's Heading 2 style. Keep its text. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Pass

Run 1 · 2026-10-15

2 tool calls · 7.71 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Pass

Run 1 · 2026-10-15

5 tool calls · 12.61 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Bold a phrase

docx-07-bold-phrase

Task prompt

In input.docx, in the paragraph that begins The pilot started, make only the phrase ten participants bold. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-15

2 tool calls · 16.81 seconds

The model created a suffix run starting with a space without xml:space="preserve", so that space was no longer preserved. Cause: model.

Checker message

document.body[4].content[3].content: count differs (1 expected, 2 found)

Pass

Run 1 · 2026-10-15

5 tool calls · 11.02 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Insert a paragraph below a heading

docx-08-insert-paragraph

Task prompt

In input.docx, insert a new paragraph directly below the heading Next steps with the text The steering group meets on 20 October. Use the normal body text style. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Pass

Run 1 · 2026-10-15

2 tool calls · 10.76 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Pass

Run 1 · 2026-10-15

5 tool calls · 11.33 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Delete a draft note

docx-09-delete-note

Task prompt

In input.docx, delete the paragraph that starts with Draft note. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Pass

Run 1 · 2026-10-15

2 tool calls · 8.01 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Pass

Run 1 · 2026-10-15

5 tool calls · 9.36 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Finalize a report

docx-10-finalize-report

Task prompt

In input.docx, finalize the report: replace the placeholder [DATE] with 14 October 2026, delete the line that only says DRAFT, and make the first line, Pilot status report, a Heading 1. Edit the document directly, without tracked changes. Save the result as output.docx, leaving input.docx unchanged and preserving everything else.

Pass

Run 1 · 2026-10-15

2 tool calls · 7.05 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Pass

Run 1 · 2026-10-15

11 tool calls · 26.39 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Reading these results

Tool calls count different things across arms.

  • Generic file tools: Shell commands and file edits; one command can run a whole script.
  • BetterOffice MCP: BetterOffice MCP calls.

Each arm ran in separate batches (Generic file tools: 2026-10-15; BetterOffice MCP: 2026-10-15). Times are not comparable across arms.