Summary
Object-dtype categorical channels fall back to a Python pass over every row in _factorize_categories, even when the column is a low-cardinality repeated category column that could use the existing native factorizer. This creates an avoidable backend performance cliff for common pandas object/string columns.
Current behavior
In python/xy/channels.py, _factorize_categories() uses the native fixed-width factorizer only for U, S, and boolean NumPy dtypes. Object arrays always execute:
labels = [category_label(value) for value in arr.astype(object)]
categories = sorted(set(labels))
For a large object column with a small repeated category set, this materializes an N-entry Python label list and performs Python-level mapping despite the data being suitable for native equality/factorization after a safe normalization step.
The performance audit identifies this as a significant cost for common object-backed pandas string/category columns: the current path performs multiple Python-level passes and retains an N-entry label list during payload construction.
Proposed direction
Add a semantics-preserving object-array fast path for values that can be normalized to a fixed-width native representation, then route those values through the existing native factorizer. Retain the current Python fallback for mixed objects, null-like values, custom category_label cases, and any values whose canonical display labels can collide or change ordering during normalization.
Acceptance criteria
- Object-dtype categorical columns with safely normalizable repeated string/category values use the native factorizer without changing category labels, sorted palette order, or code assignments.
- Mixed-object and null-containing columns retain the existing fallback behavior and exact label semantics.
- Existing categorical color and palette parity tests remain unchanged and pass.
- Add a regression/performance-oriented test proving the eligible object path avoids the N-entry Python label materialization, using a spy or equivalent deterministic assertion rather than a timing threshold.
- Large object categorical channels do not regress memory usage or introduce an additional full-size copy when normalization is unnecessary.
- Document the fallback conditions so future changes do not broaden the fast path without parity coverage.
Summary
Object-dtype categorical channels fall back to a Python pass over every row in
_factorize_categories, even when the column is a low-cardinality repeated category column that could use the existing native factorizer. This creates an avoidable backend performance cliff for common pandas object/string columns.Current behavior
In
python/xy/channels.py,_factorize_categories()uses the native fixed-width factorizer only forU,S, and boolean NumPy dtypes. Object arrays always execute:For a large object column with a small repeated category set, this materializes an N-entry Python label list and performs Python-level mapping despite the data being suitable for native equality/factorization after a safe normalization step.
The performance audit identifies this as a significant cost for common object-backed pandas string/category columns: the current path performs multiple Python-level passes and retains an N-entry label list during payload construction.
Proposed direction
Add a semantics-preserving object-array fast path for values that can be normalized to a fixed-width native representation, then route those values through the existing native factorizer. Retain the current Python fallback for mixed objects, null-like values, custom
category_labelcases, and any values whose canonical display labels can collide or change ordering during normalization.Acceptance criteria