Skip to content

Preserve batched Excel writes across recipe batches - #1112

Merged
ebhills merged 5 commits into
devfrom
recovery/excel-batch-writes
Aug 2, 2026
Merged

Preserve batched Excel writes across recipe batches#1112
ebhills merged 5 commits into
devfrom
recovery/excel-batch-writes

Conversation

@ebhills

@ebhills ebhills commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • prevent recipe batching from writing partial intermediate results
  • accumulate repeated Excel sheet writes for append and overwrite modes
  • switch subsequent overwrite batches to append so earlier output is retained
  • align dynamic or reordered batch columns by column name instead of position
  • add focused connector and recipe regression coverage

Why

Recipe batches can produce multiple writes to the same Excel sheet. Previously, intermediate recipe batches could be written independently, overwrite mode could discard earlier batches, and changing dictionary keys could shift values into the wrong columns when batches were combined positionally.

Impact

Repeated Excel writes now preserve all accumulated rows. Columns are unioned in first-seen order and row values remain aligned with their column names, including dynamic dictionary outputs.

Validation

  • 9 passed using pytest 9.0.2
  • covered Excel append and overwrite accumulation
  • covered reordered and dynamic columns
  • covered prevention of partial recipe-batch writes
  • focused run emitted eight existing pandas SettingWithCopyWarning warnings and no failures

Workflow

This PR targets the dev staging destination for live validation. After staging acceptance, the same recovery/excel-batch-writes branch should be proposed separately to main; dev should not be merged into this branch or into main.

mborodii-prog and others added 5 commits August 2, 2026 09:55
Handle repeated `excel.sheet.write` batches with changing or reordered columns by aligning rows on column names instead of position. This preserves accumulated Excel output for dynamic dictionary splits and keeps overwrite batches in the original column order, with tests covering both connector-level and recipe batch behavior.
@ebhills
ebhills marked this pull request as ready for review August 2, 2026 22:32
@ebhills
ebhills merged commit fd0862f into dev Aug 2, 2026
4 checks passed
@ebhills ebhills added the in-dev PR branch merged into dev for integration testing label Aug 2, 2026
@ebhills ebhills self-assigned this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

in-dev PR branch merged into dev for integration testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants