A Python package implementing an improved nearest neighbor method for estimating differential entropy for continuous variables. This is a port of the Julia EntropyInvariant package -- both packages are kept in sync, including the KSG/Frenzel-Pompe estimator described below.

• Invariant under change of variables: Scale and translation invariant entropy, mutual information, and conditional mutual information
• Positive in practice: Solves Edwin Thompson Jaynes' limiting density of discrete points problem. Unlike raw differential entropy, which is negative whenever the data spread is small in the chosen units, the invariant entropy is positive for common distribution families; it goes negative only in extreme cases (see Sign of the entropy)
• Bias-cancelling MI/CMI by default: mutual_information and conditional_mutual_information (and everything built on them) default to method="inv_ksg" -- invariant-measure normalization combined with a KSG/Frenzel-Pompe shared-radius estimator, which is both more accurate and more outlier-robust than the naive plug-in differencing (method="inv", also available)
• Multiple methods: entropy supports invariant (default), k-NN, and histogram methods

pip install entropy-invariant

Or install from source:

pip install -e .

import numpy as np
from entropy_invariant import entropy, mutual_information

# Generate random data
n = 1000
x = np.random.rand(n)
y = 2 * x + np.random.rand(n)

# Compute entropy (invariant method, default)
h = entropy(x)
print(f"Entropy: {h}")

# Entropy is invariant under scaling and translation
h_scaled = entropy(1e5 * x - 123.456)
print(f"Entropy (scaled): {h_scaled}")  # Same value!

# Mutual information (method="inv_ksg" by default)
mi = mutual_information(x, y)
print(f"Mutual Information: {mi}")

# Different entropy methods
h_knn = entropy(x, method="knn")
h_hist = entropy(x, method="histogram", nbins=20)

# The plug-in ("inv") estimator is still available directly for MI/CMI too,
# e.g. when you need the individual entropy terms rather than just their difference
mi_plugin = mutual_information(x, y, method="inv")

See examples/tutorial_getting_started.ipynb for a hands-on walkthrough, including why inv_ksg is the default and where it has a measurable edge over std-based normalization (a small number of extreme outliers). examples/ also has deep-dive comparison notebooks against ennemi and sklearn.

The invariant measure is the median nearest-neighbor distance, and a value that occurs more than once has a distance of 0 to its copy. So every exactly repeated value -- the zeros of sparse data, a saturation level, a fill value -- is set aside when computing the scale, wherever it sits. It still counts in the entropy and mutual information themselves: only the unit of measure ignores it.

Only exact repeats are recognized. If readings below the detection limit come back as tiny noise (1e-12 rather than 0) and make up most of a column, the scale collapses to the spacing of that noise, the entropy grows without bound as the noise shrinks, and mutual information with the rest of the data is underestimated. Snap those readings to one exact value first:

# 80% of readings are below the detection limit, recorded as noise around 0
x = np.concatenate([np.random.rand(100) * 10 + 1, 1e-12 * np.random.randn(400)])

detection_limit = 1e-6  # in the units of x: rescale it with the data
x_snapped = np.where(np.abs(x) < detection_limit, 0.0, x)

entropy(x_snapped)

A dimension left with fewer than two values that occur once has no scale, and every quantity involving it is NaN.

The invariant entropy is positive for common distribution families (uniform, normal, exponential, Cauchy and others) but not for every distribution. It goes negative only in extreme cases: a minority of the data packed into a region roughly a thousand times narrower than the rest, or more. The narrow part pulls the differential entropy down, while the scale, set by the median spacing, still follows the majority. For example:

rng = np.random.default_rng(0)
n = 100_000
x = np.where(rng.random(n) < 0.3, rng.random(n) * 1e-6, rng.random(n))  # 30% inside [0, 1e-6]
entropy(x)  # about -2.1

redundancy / unique / synergy decompose I({X,Y}; Z) into four terms and are fixed at two sources. pid_lattice gives the general N-source decomposition of Williams & Beer (2010): I({X_1,...,X_N}; Z) splits into a lattice of partial-information atoms — 4 nodes for 2 sources, 18 for 3, 166 for 4.

import numpy as np
from entropy_invariant import pid_lattice

# continuous data, three sources
atoms = pid_lattice([x1, x2, x3], z, names=["a", "b", "c"])
atoms["{a}{b}{c}"]   # redundancy shared by all three individually
atoms["{ab}{c}"]     # shared by the pair {a,b} and by c alone
atoms["{abc}"]       # the top synergy atom

# discrete data, from an explicit joint distribution (target in the last axis).
# Two-input AND, reproducing Williams & Beer's published values in bits:
pmf = np.zeros((2, 2, 2))
pmf[0, 0, 0] = pmf[0, 1, 0] = pmf[1, 0, 0] = 0.25
pmf[1, 1, 1] = 0.25
a = pid_lattice(pmf, names=["X", "Y"])
a["{X}{Y}"]   # 0.3113  (redundancy)
a["{XY}"]     # 0.5     (synergy)

Three redundancy measures are available:

• "mmi" (default, continuous data) — min over coalitions of I(X_A; Z). Works with any of the package's estimators and reduces exactly to redundancy / unique at two sources.
• "imin" (discrete data) — Williams & Beer's original measure, using specific information per target outcome. Guarantees non-negative atoms, which "mmi" does not for three or more sources. Over-credits redundancy: on two-bit COPY it reports two independent bits as fully redundant.
• "iccs" (discrete data) — Ince's (2017) pointwise common change in surprisal. Fixes the COPY case (R = 0, U_X = U_Y = 1, Syn = 0) and, unlike "mmi", lets both unique atoms be positive at once. Atoms may be negative; Ince argues these are meaningful.

# the same distribution under two measures -- Z = (X, Y), independent bits
pmf = np.zeros((2, 2, 4))
pmf[0, 0, 0] = pmf[0, 1, 1] = pmf[1, 0, 2] = pmf[1, 1, 3] = 0.25
pid_lattice(pmf, measure="imin", names=["X", "Y"])["{X}{Y}"]   # 1.0  -- over-credited
pid_lattice(pmf, measure="iccs", names=["X", "Y"])["{X}{Y}"]   # 0.0  -- correct

Estimated coalition MIs need repairing first. True mutual information satisfies I(X_A; Z) <= I(X_B; Z) whenever A is a subset of B, but finite-sample kNN estimates measurably do not — adding a source that is nearly redundant with the existing ones lowers the estimate. Both the min in "mmi" and the Möbius inversion assume that ordering, so unrepaired estimates produce negative unique and synergy atoms, which are impossible for true information. pid_lattice therefore applies isotonic_repair by default (repair="isotonic", or "majorant" / "none"). The repair deliberately does not clamp to zero: an estimate of -0.02 where the truth is near zero is ordinary symmetric noise, and truncating it would bias every low-signal region upward.

"mmi"'s unique atoms are winner-take-all. At two sources they are max(0, I_X - I_Y) and max(0, I_Y - I_X), so exactly one is nonzero by construction. That is appropriate for asking how information is divided, but it cannot express "partly one source, partly the other" and will not move gradually as an underlying relationship shifts. For how much each source contributes, use conditional_mutual_information or "iccs".

Note also that sum(atoms) == I({all sources}; Z) holds identically for any redundancy measure — it is a property of the Möbius inversion, not a check that the decomposition is right. Correctness is established in the test suite against published atom values on discrete toy distributions, the Williams & Beer non-negativity theorem, and exact agreement with redundancy / unique at two sources.

• entropy(X, method="inv", k=3, base=e, ...) - Unified entropy interface
• entropy_inv(X, ...) - Invariant method (default)
• entropy_knn(X, ...) - k-NN method
• entropy_hist(X, ...) - Histogram method

All default to method="inv_ksg" (invariant-measure normalization + KSG/Frenzel-Pompe shared-radius estimator); method="inv" (plug-in, entropy differencing), "knn", and "histogram" remain available.

• conditional_entropy(X, Y, ...) - H(Y|X)
• mutual_information(X, Y, ...) - I(X;Y)
• conditional_mutual_information(X, Y, Z, ...) - I(X;Y|Z)
• normalized_mutual_information(X, Y, ...) - NMI
• interaction_information(X, Y, Z, ...) - Three-way interaction
• information_quality_ratio(X, Y, ...) - IQR
• mutual_information_ksg(X, Y, ...) - The KSG estimator directly
• conditional_mutual_information_ksg(X, Y, Z, ...) - The Frenzel-Pompe estimator directly

• redundancy(X, Y, Z, ...) - Shared information
• unique(X, Y, Z, ...) - Unique information
• synergy(X, Y, Z, ...) - Synergistic information

• MI(X, method="inv_ksg", n_jobs=1, ...) - Pairwise mutual information matrix
• CMI(X, Z, method="inv_ksg", n_jobs=1, ...) - Pairwise conditional MI matrix

For method="inv_ksg" (the default), each dimension's own k-NN tree is built once and reused across all pairs, and n_jobs parallelizes the remaining O(m²) per-pair work across processes (n_jobs=-1 for all CPUs, scikit-learn convention) -- useful for large matrices (hundreds of dimensions), e.g. mass- spec or sensor-array data.

• Felix Truong
• Alexandre Giuliani

MIT

If you use this code or data, please cite:

An Invariant Measure for Differential Entropy: From Kullback–Leibler Divergence to Scale-Invariant Information Theory DOI: 10.3390/e28030301