Reproduce the evals

Harness instructions

for p in fonts docx xlsx pptx agents; do bun run --filter "@betteroffice/$p" build; done
eval/formats/run.sh "node packages/agents/dist/cli.js" /tmp/format-eval gpt-6-luna

eval/formats/run.sh --arm generic --python /path/to/venv/bin/python /tmp/format-eval-generic gpt-6-luna

python3 eval/formats/check.py xlsx-04-style-header /tmp/format-eval/xlsx-04-style-header/output.xlsx
python3 eval/formats/validate-checkers.py
python3 eval/formats/test-runner.py
bun eval/formats/generate-fixtures.ts /tmp/format-fixtures

The v2 generator creates synthetic files with fixed timestamps. DOCX fixtures start as hand-written OOXML, are opened and saved by BetterOffice, and carry paragraph ids. The manifest pins each published file by size and SHA-256. Downloads and local fixtures are verified against those pins.

Regenerated bytes can change when engine output changes. A changed fixture set needs a new version prefix and updated pins. Fixed timestamps do not guarantee identical bytes across engine versions.

metadata.json

• docx-01-fix-heading.docx · docx-01-fix-heading

  4651 bytes · SHA-256 6c812dc00b94a1c33aa83910d0e1bf12e0ea8936b1a176ec617a51fcd71024e9

• docx-02-replace-term.docx · docx-02-replace-term

  4684 bytes · SHA-256 a9a126e5d154bd1b228df0b2d876c78b3a17dd9324867dc94b4e47cd3687dd1e

• docx-03-correct-cell.docx · docx-03-correct-cell

  4652 bytes · SHA-256 68c84d53acedffa2a2db3d28224ff54a2f9445cfa3c08a76b79f2f119c97ecb3

• docx-04-add-row.docx · docx-04-add-row

  4652 bytes · SHA-256 68c84d53acedffa2a2db3d28224ff54a2f9445cfa3c08a76b79f2f119c97ecb3

• docx-05-delete-row.docx · docx-05-delete-row

  4652 bytes · SHA-256 68c84d53acedffa2a2db3d28224ff54a2f9445cfa3c08a76b79f2f119c97ecb3

• docx-06-heading-style.docx · docx-06-heading-style

  4726 bytes · SHA-256 b293864ebb97cf0ba02effc092cf4a45864d661ea3bf8e508a1ed167797edda7

• docx-07-bold-phrase.docx · docx-07-bold-phrase

  4652 bytes · SHA-256 68c84d53acedffa2a2db3d28224ff54a2f9445cfa3c08a76b79f2f119c97ecb3

• docx-08-insert-paragraph.docx · docx-08-insert-paragraph

  4652 bytes · SHA-256 68c84d53acedffa2a2db3d28224ff54a2f9445cfa3c08a76b79f2f119c97ecb3

• docx-09-delete-note.docx · docx-09-delete-note

  4710 bytes · SHA-256 ad66af5f6b56c6730070b3647d2a0b37f55980f43e1da86a5adf8ccf5e82a76f

• docx-10-finalize-report.docx · docx-10-finalize-report

  4670 bytes · SHA-256 e0456aef604fea54dab0b5bcef7c67e0ec129304d4d37e3b4b6efc6785c09387

• pptx-01-change-title.pptx · pptx-01-change-title

  8728 bytes · SHA-256 5eed523d96b75aff660002a0a4ec46150b36a1f7ac80d1c36e44b8e607ec372e

• pptx-02-update-revenue.pptx · pptx-02-update-revenue

  8739 bytes · SHA-256 c7ac976d205fea42a39927de648eceb3907843fccfab779e4aaa849ec247d999

• pptx-03-replace-bullets.pptx · pptx-03-replace-bullets

  8739 bytes · SHA-256 7e04f86313b452c12145790521497790df98ff18ac62b406b1754acc80593482

• pptx-04-replace-term.pptx · pptx-04-replace-term

  8752 bytes · SHA-256 d91755dc608dee0b5ce4afffdd79381bcd5dca6613d00f824caa95c6775cdc26

• pptx-05-finalize-titles.pptx · pptx-05-finalize-titles

  8754 bytes · SHA-256 0603a7585e1767dd6f3e61eb738c3829fed63bfb223af6006e7bb07a8308959d

• pptx-06-delete-appendix.pptx · pptx-06-delete-appendix

  10687 bytes · SHA-256 0908ae052c951bd591562b240a812add351e0eafe08c8fdcf09ea2798ee11d1b

• pptx-07-reorder-slides.pptx · pptx-07-reorder-slides

  10681 bytes · SHA-256 45956dea07f2aef6e8ef5411f4669ccf6877019d363bfcf210c77ce21ff4adeb

• pptx-08-speaker-notes.pptx · pptx-08-speaker-notes

  8716 bytes · SHA-256 2528fdfb1ee7f72e5aed5f0441b3df99fcfa5a996c1225809f07c2b274fe9fee

• pptx-09-position-textbox.pptx · pptx-09-position-textbox

  8719 bytes · SHA-256 a877e4275184d507f05cb3087c05aa38bb15ebeb04b93749e22de7bdf9a39bf2

• pptx-10-insert-slide.pptx · pptx-10-insert-slide

  10682 bytes · SHA-256 51e7cde1b8c55a63a3745c80326a9832306bd68d2483b29174ff6b5cf0fc8f9d

• xlsx-01-correct-value.xlsx · xlsx-01-correct-value

  2195 bytes · SHA-256 00d9aafc2a10e7c02553342d71952be66a6e65626d7a34b902d2c7b47cf39b3b

• xlsx-02-fix-total.xlsx · xlsx-02-fix-total

  2196 bytes · SHA-256 99c7d9fefd3e60d0f6338cb00f9db4e6fe9ac19d4c8790cf1effecdda2b27a46

• xlsx-03-replace-product.xlsx · xlsx-03-replace-product

  2535 bytes · SHA-256 f95454bc34bc618e590cd34b25a6178f02d91c6dd21cf92aab2f9403acbfd554

• xlsx-04-style-header.xlsx · xlsx-04-style-header

  2145 bytes · SHA-256 5f0714d386cff1fa13071bf6450db38e367f2ca10c63c80df4fef0039d6fc762

• xlsx-05-sort-inventory.xlsx · xlsx-05-sort-inventory

  2116 bytes · SHA-256 0923ec7e724f2b794a7e0e3d72956a0baea34cd3469a20345a4e9d9f3c0432fd

• xlsx-06-add-totals.xlsx · xlsx-06-add-totals

  2145 bytes · SHA-256 5f0714d386cff1fa13071bf6450db38e367f2ca10c63c80df4fef0039d6fc762

• xlsx-07-growth-column.xlsx · xlsx-07-growth-column

  2175 bytes · SHA-256 8a5a914abb3a1fb8bd054e267145ed5d3038bd678c05c36aeab2752823f43360

• xlsx-08-organize-sheets.xlsx · xlsx-08-organize-sheets

  2853 bytes · SHA-256 b0cdefaed16ca5f9845bfffd5732a8bfcebbab560242f663c3f80e5fbab896c1

• xlsx-09-budget-summary.xlsx · xlsx-09-budget-summary

  2576 bytes · SHA-256 0ae172cfa1b2b8a6aceacf660b0446f75548de191fd1762aa730532b9c80141e

• xlsx-10-quarter-report.xlsx · xlsx-10-quarter-report

  2178 bytes · SHA-256 e682a4b942f7908234f3152281d65ee35ed348ebdcf6d4e027dfe1b377b8413e