Skip to content

Performance regression: v5 is 2–7× slower than v3.8.0 for local evaluation #198

Description

@gagantrivedi

Benchmarked flagsmith 5.0.1 vs 3.8.0 against the same environment (edge API, enable_local_evaluation=True, 262 features, single-threaded, after 200-iter warmup).

p50 per call

Operation v3.8.0 v5.0.1 Regression
get_identity_flags 0.39 ms 0.82 ms 2.1×
get_environment_flags 0.11 ms 0.79 ms 7.4×
is_feature_enabled (cached Flags) 0.08 µs 0.08 µs —

Throughput drops from ~2,570 → ~1,215 identity evals/sec per core. Latency variance also grew (stdev ~10 µs → ~220 µs).

Repro

from time import perf_counter
from flagsmith import Flagsmith

client = Flagsmith(
    environment_key="<server-side-key>",
    api_url="https://edge.api.flagsmith.com/api/v1/",
    enable_local_evaluation=True,
    environment_refresh_interval_seconds=3600,
)
client.update_environment()

# warmup
for _ in range(200):
    client.get_identity_flags(identifier="anonymous", traits={"venue_id": "12345"})

for n in (100, 1_000, 10_000):
    t0 = perf_counter()
    for _ in range(n):
        client.get_identity_flags(identifier="anonymous", traits={"venue_id": "12345"})
    elapsed = perf_counter() - t0
    print(f"{n:>6} iters: {elapsed*1000:8.1f} ms ({elapsed/n*1e6:7.2f} µs/call)")

Environment: flag-engine 10.0.1, Python 3.12, macOS.

Activity

  1. assigned and unassigned on Apr 24, 2026
  2. khvn26 commented on Apr 29, 2026

    @khvn26
    Member

    We've released #200 to mitigate this, but we still need to address the issue described here. The rough plan is to profile the repro benchmark to find the exact bottleneck in evaluation. Keep in mind we need to add .all_flags() now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions