Skip to content

Use the native factorizer for low-cardinality object categorical channels #471

Description

@harsh21234i

Summary

Object-dtype categorical channels fall back to a Python pass over every row in _factorize_categories, even when the column is a low-cardinality repeated category column that could use the existing native factorizer. This creates an avoidable backend performance cliff for common pandas object/string columns.

Current behavior

In python/xy/channels.py, _factorize_categories() uses the native fixed-width factorizer only for U, S, and boolean NumPy dtypes. Object arrays always execute:

labels = [category_label(value) for value in arr.astype(object)]
categories = sorted(set(labels))

For a large object column with a small repeated category set, this materializes an N-entry Python label list and performs Python-level mapping despite the data being suitable for native equality/factorization after a safe normalization step.

The performance audit identifies this as a significant cost for common object-backed pandas string/category columns: the current path performs multiple Python-level passes and retains an N-entry label list during payload construction.

Proposed direction

Add a semantics-preserving object-array fast path for values that can be normalized to a fixed-width native representation, then route those values through the existing native factorizer. Retain the current Python fallback for mixed objects, null-like values, custom category_label cases, and any values whose canonical display labels can collide or change ordering during normalization.

Acceptance criteria

  • Object-dtype categorical columns with safely normalizable repeated string/category values use the native factorizer without changing category labels, sorted palette order, or code assignments.
  • Mixed-object and null-containing columns retain the existing fallback behavior and exact label semantics.
  • Existing categorical color and palette parity tests remain unchanged and pass.
  • Add a regression/performance-oriented test proving the eligible object path avoids the N-entry Python label materialization, using a spy or equivalent deterministic assertion rather than a timing threshold.
  • Large object categorical channels do not regress memory usage or introduce an additional full-size copy when normalization is unnecessary.
  • Document the fallback conditions so future changes do not broaden the fast path without parity coverage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions