Add functionality to track FCA Call Report Schedule Metadata - #30
Open
RNKuhns wants to merge 11 commits into
Open
Add functionality to track FCA Call Report Schedule Metadata#30RNKuhns wants to merge 11 commits into
RNKuhns wants to merge 11 commits into
Conversation
…ation Real FCA data shows a field's definition can change while it stays continuously present (RCB's INV_CODE embeds a code list that grows over time with no presence gap) -- the previous model (one shared dtype/ definition across a field's whole history) couldn't represent that. FieldAttributes now holds one or more FieldVersions, each a span where dtype/definition held constant; adjacent versions are allowed only when their content actually differs, so FieldSchema.as_of can surface the version genuinely active at a given quarter instead of always the latest. Also add is_equal/compare (with check_order) on FieldSchema and FileMetadata for inspecting drift between two snapshots, and to_json/ from_json -- the format canonical FCA schedule metadata will ship in, chosen over the existing dataframe round-trip because a field's versions nest naturally under its name. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
FCA layout files only ever say "Numeric"/"Alphanum." plus a decimal position -- nothing in the codebase previously translated that into a narwhals dtype (fca/reader.py's value-level casting infers Python int/ float/str ad hoc per raw value, never producing or consulting a stored dtype). The schedule-metadata generation script needs this translation to build FieldVersion.dtype, so it's added here as its own small, tested, doctested function rather than inlined into the script. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Maintainer tool (not part of the package's runtime or its coverage gate): scans every real, checked-in FCA archive to incrementally build each schedule's canonical FieldMetadata, applies a small hand-maintained overrides file for SME corrections, and writes the merged, authoritative result to src/call_report/fca/data/ for the package to ship. Incremental by default -- FCA's archived quarters never change once published, so only newly added periods need reprocessing; --full forces a from-scratch rebuild and audits it against the previous base via the new FieldSchema.compare, catching any drift a bug would introduce. Also add a codespell ignore-words-list entry for "SME" (subject-matter expert), a real recurring term here that reads as a typo for "some". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
get_fca_file_metadata/get_institutions_file_metadata/all_fca_file_metadata read the JSON files scripts/generate_fca_schedule_metadata.py ships under src/call_report/fca/data/schedules/ via importlib.resources (new to this codebase -- no prior use of it), cached per schedule so a file parses at most once per process. Verified against a real built wheel installed into an isolated venv, not just the editable dev install. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
api_reference.rst was missing entries for FieldAttributes/FieldSchema/
FileMetadata entirely (a pre-existing gap) -- fixed while adding the new
FieldVersion/FieldChange/FieldSchemaDiff/FileMetadataDiff and fca schedule-
metadata loaders alongside them. CLAUDE.md's repo layout now documents
the new src/call_report/fca/data/schedules/ (shipped), data/fca-schedule-
metadata/{base,overrides}/ (not shipped, generation working data), and
scripts/generate_fca_schedule_metadata.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Shows the new schedule-metadata API (get_fca_file_metadata) alongside the existing data-loading quickstart, using RCF1 specifically because its LOANSTATUS field demonstrates both a real code column (its metadata definition documents what the loaded integer codes mean) and real cross-time drift (FCA revised its code list in 2015 with no presence gap, so it has two versions -- as_of recovers whichever definition applied at a given quarter). Every value shown is verified against a real run, not illustrative placeholder output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RNKuhns
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #30 +/- ##
==========================================
Coverage 100.00% 100.00%
==========================================
Files 19 20 +1
Lines 1146 1269 +123
Branches 125 137 +12
==========================================
+ Hits 1146 1269 +123 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Melts each loaded schedule to a long intermediate, tags code-bearing fields distinctly from plain ones, then pivots on (UNINUM, period) so every schedule's variables land in one row per institution per quarter. Works on every configured backend, including pyarrow -- core._backend.pivot falls back to a manual filter-and-join reshape where narwhals has no native pivot support for it. Part 1 of 3 for v0.2's wide/long/mini-dataset stacking goal; long-format and shared-dimension mini-datasets are follow-on work.
…schedule_metadata
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| """Pivot takes no positional arguments.""" | ||
| frame = _build_long_frame(backend="pandas") | ||
| with pytest.raises(TypeError): | ||
| pivot(frame, "key", ["UNINUM", "period"], "value") # type: ignore[call-arg] |
| with config_context(dataframe_backend="pandas"): | ||
| frame = build_frame(data={"UNINUM": [1], "period": ["2026-03-31"], "A": [1]}) | ||
| with pytest.raises(TypeError): | ||
| melt_schedule_frame(frame, "RC", None) # type: ignore[call-arg] |
That directory holds raw, uncorrected per-schedule text extracted verbatim from FCA's own layout files (working data for scripts/generate_fca_schedule_metadata.py); genuine source typos like "similiar" and "Recieved" belong in the overrides pipeline, not codespell suppression. pre-commit's own codespell hook already skips it (scoped to python/markdown types), but the standalone codespell-annotations job scans the whole repo unrestricted.
…data FCACallReport.to_wide_format used to collect each schedule immediately after loading it, before any melt/concat/pivot work started. With lazy=True configured, every schedule now stays a narwhals LazyFrame through melt, concat, and column-key computation -- the only place collection genuinely happens is core._backend.pivot, since a pivoted frame's schema depends on the "on" column's distinct values and can't be determined lazily. FileMetadata.to_dataframe had the same avoidable early collect; fixed the same way, matching the fresh file-level frame's laziness to the (possibly already-lazy) field-level frame before concatenating rather than collecting one of them upfront. concat() gained a proper overload ladder (eager-in/eager-out, lazy-in/lazy-out, and a generic fallback for callers that don't statically know which) so precise callers keep precise return types, and its schema comparisons switched from `.columns` to `.collect_schema()` to avoid a PerformanceWarning on lazy input.
| with config_context(dataframe_backend="pandas"): | ||
| rc = build_frame(data={"UNINUM": [1], "period": ["2026-03-31"], "A": [1]}) | ||
| with pytest.raises(TypeError): | ||
| to_wide_format({"RC": rc}, {"RC": None}, {"RC": ()}) # type: ignore[call-arg] |
| ) -> nw.DataFrame[Any]: | ||
| """Stack multiple eager narwhals frames according to a schema policy. | ||
| ) -> nw.DataFrame[Any]: # numpydoc ignore=GL08 | ||
| ... # pragma: no cover |
| def concat( | ||
| *, frames: Sequence[nw.LazyFrame[Any]], how: SchemaPolicy | ||
| ) -> nw.LazyFrame[Any]: # numpydoc ignore=GL08 | ||
| ... # pragma: no cover |
| def concat( | ||
| *, frames: Sequence[FrameOrLazy], how: SchemaPolicy | ||
| ) -> FrameOrLazy: # numpydoc ignore=GL08 | ||
| ... # pragma: no cover |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.