Incremental persistence: single MERGE per fact table + fix stale-row & ordering bugs - #66
Merged
Merged
Conversation
…se `merge_incremental` - Updated `persist_facts_incremental` to consolidate the handling of changed and unchanged DataFrames into a single `merge_incremental` call, improving efficiency and clarity. - Introduced new parameters for processed and updated container IDs to scope delete operations correctly, ensuring stale rows are pruned during incremental updates. - Enhanced documentation to reflect the new behavior and added tests to validate the changes, ensuring correct functionality across various scenarios. - Modified related components in the `Report` class to accommodate the new persistence logic, streamlining the overall data handling process.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #66 +/- ##
==========================================
+ Coverage 88.66% 88.72% +0.06%
==========================================
Files 59 59
Lines 4791 4826 +35
Branches 547 552 +5
==========================================
+ Hits 4248 4282 +34
Misses 445 445
- Partials 98 99 +1
🚀 New features to boost your workflow:
|
…ntal` and `Report` class - Changed the parameter `processed_container_ids` to `has_processed_containers` in `persist_facts_incremental` to simplify the logic for determining if any containers were recomputed during the run. - Updated the `Report` class to reflect this change, ensuring consistent handling of processed container states across various persistence calls. - Adjusted related tests to use the new boolean flag, enhancing clarity and maintainability in the test suite.
- Updated the documentation in `index.md` to clarify the persistence strategy for unchanged and changed definitions, emphasizing the atomic nature of Delta `MERGE` operations. - Improved the `test_incremental_modified_container_fewer_rows_deletes_stale` test by utilizing a new helper function to streamline the setup of modified containers for incremental tests. - Added a new test, `test_incremental_modified_container_multi_family_shrink_deletes_stale`, to validate that modified containers producing fewer rows under unchanged definitions correctly delete stale rows across all relevant fact tables while preserving the integrity of non-reprocessed containers. - Refactored the `clone_silver_with_shrunk_container` helper to enhance clarity and maintainability in test setups.
- Updated documentation in `index.md` to clarify the behavior of the `MERGE` operation with schema evolution, detailing how it handles added and dropped columns during incremental runs. - Modified the `UnityCatalogSink` class in `report_storage.py` to implement `withSchemaEvolution()`, allowing for seamless integration of schema changes without breaking incremental processing. - Added a new test case in `histogram_bin_change_test.py` to validate that a fact-schema change (such as adding a new column) correctly evolves the gold table during an incremental MERGE, ensuring data integrity and consistency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Collapses the incremental fact-write from two Delta transactions (replaceWhere + N× upsert) into a single MERGE per fact table, and fixes two bugs in the incremental workflow: stale gold rows left by shrinking containers, and a persist-ordering bug that silently skipped reprocessing of updated containers' calculated channels.
Changes
Test Plan
Checklist