Construction of arrays with `object` dtype very slow when Pandas `future.infer_string` is enabled

[y4n9squared]

What happened?

Constructing a Variable (or Dataset/DataArray) from an object-dtype numpy array of strings takes ~0.1ms per 10M elements under pandas' default settings, but ~500–620ms with pd.options.future.infer_string = True — the setting that becomes the default in pandas 3.0.

Why:

as_compatible_data routes every object array through _possibly_convert_objects,

xarray/xarray/core/variable.py

Lines 316 to 317 in adc8005

	if data.dtype.kind in "OMm":
	data = _possibly_convert_objects(data)

which round-trips the data through pd.Series to convert datetime/timedelta objects to datetime64/timedelta64.

Under legacy (Pandas 2.x) inference, that probe was effectively free — pandas' datetime scan bails on the first non-datetime element. Under string (Pandas 3.x) inference, the same constructor's contract now includes instantiating the string dtype object, so pandas performs a full scan and materializes an Arrow string array of the entire input — which _possibly_convert_objects immediately converts back to numpy and discards:

as_compatible_data
└─ _possibly_convert_objects
   └─ pd.Series(values)                      # probe only
      └─ maybe_infer_to_datetimelike
         └─ ArrowStringArrayNumpySemantics._from_sequence   # full Arrow build
   └─ np.asarray(series)                     # Arrow array discarded

Minimal Complete Verifiable Example

import time

import numpy as np
import pandas as pd
import xarray as xr

rng = np.random.default_rng(0)
labels = np.array([f"label_{i:04d}" for i in range(500)], dtype=object)
arr = labels[rng.integers(0, len(labels), 10_000_000)]

for infer_string in (False, True):
    pd.options.future.infer_string = infer_string
    best = float("inf")
    for _ in range(5):
        t0 = time.perf_counter()
        xr.Variable(("x",), arr)
        best = min(best, time.perf_counter() - t0)
    print(f"infer_string={infer_string}: {best * 1000:.1f}ms")

Output:

infer_string=False:   0.1ms
infer_string=True:  606.3ms

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Environment

Details

INSTALLED VERSIONS

commit: None
python: 3.14.5 (main, May 10 2026, 19:28:16) [Clang 22.1.3 ]
python-bits: 64
OS: Linux
OS-release: 6.17.0-1019-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('C', 'UTF-8')
libhdf5: None
libnetcdf: None

xarray: 2026.7.0
pandas: 2.3.3
numpy: 2.4.4
scipy: 1.17.1
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
zarr: 2.18.7
cftime: None
nc_time_axis: None
iris: None
bottleneck: 1.6.0
dask: 2025.12.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2026.4.0
cupy: None
pint: None
sparse: 0.18.0
flox: None
numpy_groupies: None
setuptools: 80.9.0
pip: None
conda: None
pytest: 9.1.0
mypy: None
IPython: 9.12.0
sphinx: None

[JJFlorian]

I could work on this issue. What do you guys think of a fast path in the _possibly_convert_objects, that finds the first non-None/non-NaT element and returns the value immediately if it is a string. Otherwise, the existing pd.Series probe will still be used.

e.g.:

def _possibly_convert_objects(values):
    ...
    flat = values.ravel()
    first_non_null = next(
        (x for x in flat if x is not None and x is not pd.NaT),
        None,
    )
    if isinstance(first_non_null, str):
        return values

    as_series = pd.Series(values.ravel(), copy=False)
    result = np.asarray(as_series).reshape(values.shape)

Pandas only converts object arrays when every non-null element is a datetime or timedelta, so a leading string means datetime conversion is impossible.

The usage of next might be slow if there are millions of None/NaT values before the first string, but I don't think that is a realistic scenario.

Happy to open a PR if you think if this direction looks good to you, or if you can guide me in the correct direction.

[dcherian]

Can you control the input instead? Object dtype tells us nothing so it really would be best if you explicitly cast before you send it in to Xarray. Any auto-guessing is going to be fragile.

[y4n9squared]

Can you control the input instead? Object dtype tells us nothing so it really would be best if you explicitly cast before you send it in to Xarray. Any auto-guessing is going to be fragile.

This is essentially what I'm doing in client code right now as a work-around.

def build_variable(
    dims: tuple[Hashable, ...],
    arr: np.ndarray,
    source_dtype: np.dtype,
) -> xr.Variable | tuple[tuple[Hashable, ...], np.ndarray]:
    if np.issubdtype(source_dtype, np.datetime64) or np.issubdtype(
        source_dtype, np.timedelta64
    ):
        return dims, arr
    return xr.Variable(dims, arr, fastpath=True)

fastpath=True causes me to skip the problematic as_compatible_data() branch.

But since the normal usage of XArray happens most commonly with the xr.Dataset and xr.DataArray initializers, the actual numpy array is actually being constructed internally. i.e. if you just pass a normal Python list, you'll see the performance regression. Folks will be manually forced to construct DataArrays only from Variable objects to avoid the issue.

Seems to me like the change can be entirely local to as_compatible_data() using a heuristic like what @JJFlorian proposed (but we'd probably want to limit the scan). Even if you called `pd.apis.types.infer_dtype which does a scan across the entire array, it will be 10x faster than what we currently do, since you still avoid the re-allocation of the individual objects that is necessary to return the temporary Series object.

[dcherian]

To clarify, can we call pd.apis.types.infer_dtype + a cast instead of roundtripping ? That sounds like a great improvement. If that's right, can you open a PR please

[JJFlorian]

To clarify, can we call pd.api.types.infer_dtype + a cast instead of roundtripping ? That sounds like a great improvement. If that's right, can you open a PR please

I tried replacing the pd.Series roundtrip with a pd.api.types.infer_dtype+ cast. It works well for string, datetime, and timedelta. However, I can't find a way to cast datetime64/timedelta64 while preserving their units. We could fall back on the roundtrip for those cases, but that would only add a pd.api.types.infer_dtype to the existing route.

Also, running the example with the infer + cast gives:

infer_string=False: 55.5ms
infer_string=True: 57.8ms

So indeed, a 10x speedup for infer_string=True, but infer_string=False shows the cost of pd.api.types.infer_dtype. Of course, this effect is relatively large with the 10m elements example, but still I wonder if that is something to worry about?

Edit: for reference, my changes after experimenting in a draft PR: #11484