Skip to content

Add value counts aggregation to select.group_by - #1114

Open
ebhills wants to merge 1 commit into
devfrom
recovery/select-group-by-counts
Open

Add value counts aggregation to select.group_by#1114
ebhills wants to merge 1 commit into
devfrom
recovery/select-group-by-counts

Conversation

@ebhills

@ebhills ebhills commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a counts aggregation to select.group_by
  • return each distinct value and its frequency as a dictionary
  • convert dictionary keys to JSON-safe strings
  • represent missing values as null and booleans as lowercase true or false
  • support existing per-column output renaming behavior

Why

Grouped recipes sometimes need the distribution of values within each group rather than only a scalar count or a list of the original values. Without this aggregation, recipes need additional transformations to construct a value-to-frequency mapping.

Impact

Recipes can now request value counts directly:

wrangles:
  - select.group_by:
      by: Group
      counts:
        - Selection: Selection Counts
      auto_rename_columns: false

The resulting Selection Counts value is a JSON-safe dictionary such as {"Primary": 2, "None": 1}.

Validation

  • complete TestGroupBy class: 30 passed
  • grouped smoke coverage for strings, nulls, and booleans passed
  • git diff --check passed
  • two existing Python invalid-escape SyntaxWarning warnings were emitted; no test failures
  • branch differs from current main by one commit and two files

Workflow

This is the first pull request for the feature and targets the dev staging destination. After staging validation, open a second pull request from the same recovery/select-group-by-counts branch into main. Do not delete the branch after the dev merge.

@ebhills ebhills self-assigned this Aug 3, 2026
@ebhills
ebhills requested a review from Copilot August 3, 2026 02:22
@ebhills
ebhills marked this pull request as ready for review August 3, 2026 02:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new counts aggregation option to select.group_by so grouped recipes can return per-value frequency distributions (as JSON-safe dictionaries) instead of only scalar aggregates.

Changes:

  • Extend select.group_by to support a counts aggregation that returns {value_as_string: frequency} per group.
  • Normalize keys for JSON-safety (string conversion, lowercase boolean strings, and a "null" key for missing values).
  • Add a unit test validating the new aggregation and per-column output renaming with auto_rename_columns: false.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
wrangles/recipe_wrangles/select.py Adds schema documentation and implements the counts aggregation using value_counts(dropna=False) with JSON-safe key normalization.
tests/recipes/wrangles/test_select.py Adds a test covering counts output structure and column renaming behavior.

Comment on lines +325 to +327
description: >-
Return a dictionary containing the count of each distinct value for
these column(s). Keys are converted to JSON-safe strings.
Comment on lines +1953 to +1962
dataframe=pd.DataFrame({
"Selection": ["Primary", "Primary", "None"],
"Review": [False, False, True],
}),
)

assert df.to_dict(orient="records") == [{
"Selection Counts": {"Primary": 2, "None": 1},
"Review Counts": {"false": 2, "true": 1},
}]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants