From 21c4401ebee995aaf33df1b402a6b42c7bd2ab60 Mon Sep 17 00:00:00 2001 From: Kevin Costner <120246174+kevincostner17@users.noreply.github.com> Date: Tue, 15 Sep 2026 15:13:18 +0530 Subject: [PATCH] docs: changelog for the round-5 bug-hunt fixes Add Unreleased Added, Changed and Fixed entries for merged PRs #351, #352, #353, #354, #356, #357, #358, #360, #361, #362, #363, #364, #365, #367, #368, #369, #370, #371, #372, #374, #375, #377, #378 and #379. --- CHANGELOG.md | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4769afd0..fb924ca5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,14 @@ adheres to [Semantic Versioning](https://semver.org/). tool orderings while keeping PyJanitor optional. - A dependency-optional Great Expectations recipe demonstrating the repair-then-validate workflow with an in-memory checkpoint. +- `TimeSeriesCleanConfig.timestamp_unit` (`"s"`, `"ms"`, `"us"` or `"ns"`) sets + the epoch unit for numeric timestamp and event-time columns in time-series + streaming. Without it the unit is inferred from the values (#227). +- `apply_review_decisions()` accepts a keyword-only `queue=` (the + `ReviewQueueReport` the reviewer worked from), so decisions that carry only + an `item_id` can be resolved to a record pair (#267). +- The `FRESHDATA_MODEL_TIMEOUT` environment variable sets the network timeout, + in seconds, for `fd.models.pull` downloads (default 60) (#341). ### Changed - `explain_clean()` now profiles only post-clean columns that can contribute a @@ -34,6 +42,99 @@ adheres to [Semantic Versioning](https://semver.org/). zero-column frame that keeps the row count, as the pandas pipeline does. Polars and Arrow output cannot hold rows without columns; the report records that difference (#201). +- `freshdata validate` exits 2 ("could not load rules") instead of 1 when a + `--suite` or `--contract` file is not an object, and + `ValidationSuite.from_dict` raises `ValueError` for non-mappings (#289). +- `dbt-gate` no longer passes when nothing was gated. A file without a `nodes` + mapping (such as `run_results.json`) raises `ValueError`, `all_passed` is + false when no models were processed, and `--fail` exits 1. Ephemeral and + disabled models are skipped and listed under a new `skipped` summary key + instead of counting as failures (#296, #249). +- Semantic repairs that could rewrite a valid value are now suggested for + review instead of auto-applied: fuzzy cleaning-memory matches, percent values + in `rate`/`ratio` columns whose scale is fractional or unknown, and shape + alignments of unseparated values. Shape alignments whose groups do not match + the template are no longer proposed (#252, #253, #254). +- The EIDR check character (MD-C002) now uses the hybrid ISO 7064 MOD 37,36 + system from the EIDR ID Format spec, so published EIDR IDs validate. IDs + whose check character came from the old MOD 37-2 code are flagged, and a `*` + check character is always rejected (#259). +- The finance FIN-003 guard leaves ambiguous DD/MM vs MM/DD dates unresolved + even when a time follows the date, instead of reading them month-first + (#260). +- HIPAA Safe Harbor reports now rest on column evidence. `fd.clean` and + `fd.apply_plan` record the input columns, so identifier columns that + cleaning did not touch are detected without `dataframe=` and reports that + used to pass can fail. Reports with no full column list (synthetic reports, + native backends, streaming and multi-file runs) set + `coverage_verifiable: False`, add a warning and do not pass (#245). +- HIPAA identifier hints match whole column-name tokens, so `ip` no longer + matches `description` or `ship_address`. Hints of four characters or fewer + no longer match inside run-together lowercase names such as `visitdate`; + separated and camelCase forms still match (#283). +- Blocking rules the pandas entity-resolution backend cannot evaluate (`OR`, + comparison operators, literals, arithmetic, `BETWEEN`, parenthesised + predicates, unquoted names with spaces or hyphens) now raise + `EntityResolutionError` instead of returning zero or wrong pairs. Quote such + names, for example `l."first name" = r."first name"` (#237). +- Entity resolution treats NaT and `pd.NA` as missing, so they no longer count + as agreement and records previously merged on missing datetimes can split + (#238). +- `clean_enterprise` raises `ValueError` when `EnterpriseConfig.anonymization` + is non-empty, instead of silently ignoring a setting it does not apply. + `EnterpriseConfig` still accepts the field, and the CLI prints a one-line + error and exits 1 (#247). +- `TrustScoreWeights` rejects NaN and infinite weights with `ValueError` + instead of producing a `nan` trust score (#277). +- Time-series streaming reads numeric timestamp and event-time columns as + epochs instead of 1970 dates, recording the unit in a + `timeseries_timestamp_parse` action. Unparseable timestamps keep their row + and are reported in `coerced_cells`/`coerced_rows` with a warning, and + mixed-offset batches become a UTC-aware column (#227, #250). +- `TimeSeriesCleanConfig(anomaly_window_size=1)` is rejected when the config + is built (#290). +- `cdc_profile` measures freshness against the current UTC time by default, + and naive `now=` and `watermark=` values are read as UTC (#233, part 4). +- A constant baseline column now produces `drift.ks` findings when current + values move away from the constant. Saved baseline JSON stores statistics at + full precision instead of six decimal places; existing files load unchanged. + Some point-mass shifts score lower than before, because the higher scores + came from the tie handling fixed in #234 (#235, #275). +- `load_review_decisions` reads CSV ids as strings (`"007"` stays `"007"`), and + `apply_review_decisions` raises `ValueError` for a decision it cannot + resolve to a pair instead of dropping it. `feedback_summary` gains an + `n_unmatched` count, and clusters created by an apply get ids numbered past + `n_records` (#239, #267, #268). +- Learned clean values in cleaning-memory JSON are plain JSON types: integers + stay numbers, and Timestamps and Decimals are strings (#256). +- With `apply_plan(allow_drift=True)`, actions whose raw value is no longer in + the column are recorded as skipped (`frame drift`) with count 0, and applied + actions record the observed cell count instead of the plan-time + `n_affected` (#258). +- `Parser.read_text` defaults to `encoding="utf-8-sig"` and strips a leading + UTF-8 BOM from text input (#314). +- `mostly` thresholds are inclusive: a rule with exactly the allowed share of + violating rows (for example 1 of 10 under `mostly=0.9`) warns and passes + instead of failing (#307). +- `RepairPlan.decisions_hash` and `to_json` change for plans whose params hold + sets (members are now sorted) or numpy scalars (now JSON numbers and bools), + so the value no longer depends on `PYTHONHASHSEED` or the numpy version. + Plans without such params keep their hash (#312). +- The context compiler no longer splits allowed values or dedup keys on `/`, + and a bare number is no longer read as a confidence gate (`only if 3 + neighbours agree`); such phrases stay unparsed and raise under `strict` + (#301, #303). +- `fd.clean(..., policy=..., strict=True)` with a schema-free policy now + raises `protection_conflict`, as the `columns=` and `context=` flows do + (#304). +- Phone validation rejects numbers with more than 15 digits (E.164) (#318). +- Quality-debt `duplicates` now counts duplicate rows detected in the cleaned + output, not only rows removed, so frames with duplicates can warn or fail the + gate under default options. `pii_risk` counts distinct PII columns instead of + matching cells, so scores drop for tall PII columns (#264, #286). +- Network plugins registered without `allow_network=True` re-read + `FRESHDATA_ALLOW_NETWORK_PLUGINS` at call time, so setting the variable after + registration activates them and unsetting it deactivates them again (#299). ### Fixed - Trust-gate integrations now validate `on_low_score` policies at configuration @@ -98,6 +199,104 @@ adheres to [Semantic Versioning](https://semver.org/). - The missing-pyarrow error names the feature that needs it (for example Arrow output), and Parquet metadata reads no longer fail with `AttributeError` in a fresh process (#215). +- `freshdata clean --config` reports invalid YAML or JSON, non-object + sections and unknown keys as a one-line error naming the file (exit 1) + instead of a traceback, and `dbt-gate` does the same for malformed manifests + and directory paths (#289). +- `freshdata clean` and `freshdata validate` no longer exit 1 after a passed + gate when stdout cannot encode UTF-8 (#295). +- Semantic memory replay looks up the expert that learned a repair, so learned + Unicode normalization and shape-alignment repairs replay instead of being + checked as dates (#300). +- Retail GTIN checks read float-loaded integral cells as integer text, so a + blank cell no longer causes a valid GTIN to be rewritten into a different + one (#229). +- Healthcare date checks compare offset-aware FHIR `dateTime` values with naive + dates or mixed offsets in UTC instead of raising `TypeError` (#233, part 2). +- The GDPR Article 30 report lists only the measures the run actually applied + (#287). +- The DuckDB entity-resolution backend accepts non-equality blocking SQL such + as `jaro_winkler_similarity(l.name, r.name) > 0.8` (#236). +- `fd.link(backend="duckdb")` works on keys containing spaces or hyphens + (#266). +- `link_entities` and external `fd.link` reports record their thresholds, so + `build_review_queue` orders items around the configured midpoint (#271). +- `StreamingCleaner(global_duplicates=True)` keeps a bounded window of recent + rows instead of the first `window_size` rows forever, so duplicates of recent + rows are removed for the whole stream (#292). +- Streaming cross-batch deduplication no longer misses duplicates when a + column flips between integer and float dtypes (#293). +- Streaming distribution drift fires for a column that had been constant and + then changes (#294). +- MAD anomaly detection no longer flags the minority value of two-valued or + sparse series (#291), and anomaly columns stay stable when a batch's dtype + changes, with text cells never scored or capped (#248, part). +- CSV review queues round-trip: the formula guard added on export is removed + from ids on load, blank decision cells are skipped, and applying decisions + keeps existing cluster ids and canonical records (#239, #240, #268). +- A frame no longer fails drift checks against its own baseline when values + tie across stored quantiles (#234). +- `pd.ArrowDtype` columns (double, decimal, large string, dictionary) are + recognised by contracts and baselines, and Arrow decimals no longer crash + numeric profiling (#241). +- Semantic cross-field checks no longer raise or miss findings on frames with + duplicate row labels (#231, part 1), integer column labels no longer raise + `KeyError` in semantic repair (#232, part 1), and date-ordering checks + compare tz-aware and naive values instead of raising `TypeError` (#233, + part 1). +- `save_profile` no longer fails with `TypeError` on profiles learned from + numpy or pandas clean values (#256), and `LearningProfile.merge` no longer + modifies either parent's memory (#257). +- The test suite runs from an unpacked sdist and in isolation, and the + streaming docs describe `rolling_trust_score` as an unweighted mean + (#347, #348, #349). +- The FHIR parser records malformed resources (unexpected list or object + shapes, non-string `resourceType`) as per-resource warnings instead of + raising (#313). +- HL7v2, FHIR and EDIFACT parsers handle input that starts with a UTF-8 BOM + (#314). +- The GPX parser skips points with NaN, infinite or out-of-range coordinates + with a warning (#320). +- The context compiler keeps quoted values and values such as + `Trinidad and Tobago` whole (#301), and dotted column names such as + `file.name` no longer split the sentence (#302). +- `clean_text_value` is idempotent and returns text in the configured Unicode + normal form (#316). +- `lint_text_encoding` no longer reports Japanese text such as `コーヒー`, or + `Nº 5`, as mixed script, or uppercase Portuguese such as `MANHÃ DE SOL` as + mojibake (#317). +- `validate_fields` accepts international phone numbers such as + `+49 (0) 30 12345678` and punycode email TLDs (#318). +- `insight_report` issue ids are unique for columns whose names slug the same + (#329). +- HTML report filter boxes work; the generated script was a JavaScript syntax + error (#336). +- `stakeholder_summary` and the per-column view no longer describe preserved + missing values as changes (#337), and no longer claim 100% completeness when + every column was dropped (#339). +- `export_dbt_tests` quotes YAML scalars that would change type or lose + characters on load (dates, `0x1F`, trailing newlines), and floats such as + NaN and infinity load back as floats (#342). +- `OnnxEncoder` no longer fails when several threads trigger the lazy model + load at once (#340). +- `fd.models.pull` times out stalled downloads and rejects a truncated file + instead of installing it (#341). +- The FreshCore backend falls back to pandas for configurations its native + kernels got wrong: `impute="missforest"`, per-column `impute_strategy`, + outlier detection on float columns holding infinity, and mode imputation of + nullable boolean columns. Under `fallback_policy="error"` these runs raise + `FallbackError` (#322, #334, #335). +- The FreshCore adapter reports native duplicate detections and applies + `duplicate_ratio_action` to native drop counts, as the pandas pipeline does + (#323, part). +- The Spark engine renames columns without collisions, honours + `duplicate_keep` and input order when deduplicating, reads float `NaN` as + null with outlier fences from finite values only, and no longer treats + interval columns as numeric (#330, #331, #332, #333). +- A malformed plugin proposal or entry point is dropped or skipped with a log + message instead of crashing the clean or stopping registration (#297). +- Reusing a plugin name within one kind logs a warning naming the replaced + plugin (#298). ## [2.0.0] - 2026-07-20