BetterOffice / evals

v2 · 2026-10-15 · synthetic

XLSX tasks

gpt-6-luna

XLSX results for gpt-6-luna
TaskGeneric file toolsBetterOffice MCP
Correct a budget amount

xlsx-01-correct-value

Task prompt

In input.xlsx, change the planned Supplies amount in Budget!B3 to 180. Keep the totals as formulas and update their calculated results. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

3 tool calls · 33.27 seconds

openpyxl saved extra theme and document property parts, page margins and a calcId, changing unrelated workbook state. Cause: library.

Checker message

document.sheets[1].metadata: count differs (2 expected, 4 found)

Pass

Run 1 · 2026-10-14

7 tool calls · 20.83 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Repair a total formula

xlsx-02-fix-total

Task prompt

In input.xlsx, the Actual total in Budget!C5 leaves out Travel. Replace it with =SUM(C2:C4) and update its calculated result. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

5 tool calls · 43.24 seconds

openpyxl cleared the existing Budget!B5 formula cache on save while the model restored only the requested C5 result. Cause: library.

Checker message

Budget!B5: empty cached value

Pass

Run 1 · 2026-10-14

6 tool calls · 19.21 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Rename a product across sheets

xlsx-03-replace-product

Task prompt

In input.xlsx, replace every occurrence of the case-sensitive product name Trail Mix with Forest Mix on both Catalog and Orders. Keep the rest of each cell intact. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

2 tool calls · 10.59 seconds

openpyxl saved extra theme and document property parts, page margins and a calcId while replacing product text. Cause: library.

Checker message

document.sheets[1].metadata: count differs (2 expected, 4 found)

Pass

Run 1 · 2026-10-14

6 tool calls · 26.80 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Format a header

xlsx-04-style-header

Task prompt

In input.xlsx, make Sales!A1:C1 bold and give those three header cells a solid light blue fill, color #D9EAF7. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

1 tool calls · 11.18 seconds

openpyxl saved extra theme and document property parts, page margins and a calcId while styling the header. Cause: library.

Checker message

document.sheets[1].metadata: count differs (2 expected, 4 found)

Pass

Run 1 · 2026-10-14

6 tool calls · 20.68 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Sort an inventory table

xlsx-05-sort-inventory

Task prompt

In input.xlsx, sort Inventory!A2:C5 by Stock in column C, smallest to largest. Move each complete row together and leave the header in row 1. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

2 tool calls · 8.84 seconds

openpyxl saved extra theme and document property parts, page margins and a calcId while sorting the inventory. Cause: library.

Checker message

document.sheets[1].metadata: count differs (2 expected, 4 found)

Pass

Run 1 · 2026-10-14

6 tool calls · 19.90 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Add a currency total row

xlsx-06-add-totals

Task prompt

In input.xlsx, add a total row on Sales at row 5: A5 should say Total, B5 should contain =SUM(B2:B4), and C5 should contain =SUM(C2:C4). Format B5:C5 as dollars with two decimal places and a thousands separator, using $#,##0.00, and save calculated results. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

3 tool calls · 24.88 seconds

The model wrote total formulas but left their calculated results empty, relying on a later recalculation instead of saving the requested values. Cause: model.

Checker message

Sales!B5: empty cached value

Pass

Run 1 · 2026-10-14

7 tool calls · 24.40 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Insert a growth column

xlsx-07-growth-column

Task prompt

In input.xlsx, insert a column immediately after Revenue column C, moving the existing Status column to E. Name D1 Growth %. In D2:D4 use =(C2-B2)/B2, =(C3-B3)/B3, and =(C4-B4)/B4 respectively. Format D2:D4 as percentages with one decimal place, using 0.0%, and save calculated results. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

5 tool calls · 29.46 seconds

openpyxl saved extra theme and document property parts, page margins and a calcId while inserting the growth column. Cause: library.

Checker message

document.sheets[1].metadata: count differs (2 expected, 4 found)

Pass

Run 1 · 2026-10-14

8 tool calls · 26.31 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Rename and reorder sheets

xlsx-08-organize-sheets

Task prompt

In input.xlsx, rename Forecast to Outlook. Arrange the tabs in this order: Actuals, Outlook, Assumptions. Keep the formulas on Assumptions working by changing its B2 reference to ='Outlook'!B2; keep all other formulas and values intact. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

2 tool calls · 20.48 seconds

openpyxl cleared the Assumptions!B2 formula cache and renumbered sheet ids on save while the model renamed and reordered sheets. Cause: library.

Checker message

Assumptions!B2: empty cached value

Pass

Run 1 · 2026-10-14

9 tool calls · 20.42 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Build a linked budget summary

xlsx-09-budget-summary

Task prompt

In input.xlsx, add an Expenses total row at row 5 with A5 Total, B5 =SUM(B2:B4), and C5 =SUM(C2:C4). On Summary, set B2 to ='Expenses'!B5, B3 to ='Expenses'!C5, and B4 to =B3-B2. Format Expenses!B5:C5 and Summary!B2:B4 as dollars using $#,##0.00 and save calculated results. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

4 tool calls · 42.96 seconds

openpyxl saved extra theme and document property parts, page margins and a calcId while building the budget summary. Cause: library.

Checker message

document.sheets[1].metadata: count differs (2 expected, 4 found)

Pass

Run 1 · 2026-10-14

8 tool calls · 22.48 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Prepare a quarterly report

xlsx-10-quarter-report

Task prompt

In input.xlsx on Quarterly, insert Growth % in a new column D, moving Owner to E. Sort the three data rows by Q2 descending, keeping each owner with the product. Set D2:D4 to =(C2-B2)/B2, =(C3-B3)/B3, and =(C4-B4)/B4. Add row 5 with A5 Total, B5 =SUM(B2:B4), C5 =SUM(C2:C4), and D5 =(C5-B5)/B5. Use 0.0% in D2:D5 and $#,##0.00 in B5:C5. Make A1:E1 bold with solid #D9EAF7 fill. Save calculated results. Save the result as output.xlsx, leaving input.xlsx unchanged and preserving everything else.

Fail

Run 1 · 2026-10-13

5 tool calls · 53.01 seconds

The model changed a shared default fill color through a shallow copy, changing the base fill outside the requested header. Cause: model.

Checker message

document.sheets[1].cells.A1.style.base.fill: expected None, found 'FFD9EAF7'

Pass

Run 1 · 2026-10-14

10 tool calls · 30.37 seconds

Checker message

exact requested content, formulas, formatting, order, notes, and preserved content verified

Reading these results

Tool calls count different things across arms.

  • Generic file tools: Shell commands and file edits; one command can run a whole script.
  • BetterOffice MCP: BetterOffice MCP calls.

Each arm ran in separate batches (Generic file tools: 2026-10-13; BetterOffice MCP: 2026-10-14). Times are not comparable across arms.