[po-et]

What does this PR do?

Blocking synchronous experiment tasks execute serially on the event loop despite max_concurrency. Add run_sync_tasks_in_threads=True to opt into concurrent synchronous tasks, while preserving the default execution context for existing tasks and caller-owned SQLite connections.

result = langfuse.run_experiment(
    name="blocking-io-evaluation",
    data=items,
    task=my_thread_safe_task,
    max_concurrency=50,
    run_sync_tasks_in_threads=True,
)

• Each experiment uses a dedicated pool with up to max_concurrency workers; a smaller asyncio default executor does not cap its capacity.
• Copy each item's context into its worker so traces keep the correct parent and experiment attributes. Async functions, decorated async wrappers, partials and async callable objects stay on the experiment loop and invoke the original callable.
• Expose the option through Langfuse, DatasetClient and RunnerContext. Document resource ownership, item-error handling and cancellation in the regular API reference. Remove the earlier breaking-change README and standalone migration note.
• Close native coroutine results that arrive after cancellation prevents awaiting the worker result. Shutdown cancels queued calls without blocking the event loop. Running synchronous calls cannot be stopped and can continue after cancellation or delay Python process exit; tasks should use finite I/O timeouts.

This targets prepare-v5-release and integrates its abbc6cb24d2af237ffec029fa44b017d295af4ff head with a normal merge. Existing branch history is preserved. A retarget to main after v5 publication remains subject to the maintainer's release plan.

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor
• Documentation update
• Tooling, CI, or repo maintenance

Verification

Python 3.12.12 on macOS arm64, dependencies installed with uv sync --locked --python 3.12:

# Use the workflow's dummy test configuration for the unit suite.
LANGFUSE_BASE_URL=http://localhost:3000 LANGFUSE_PUBLIC_KEY=pk-lf-test \
LANGFUSE_SECRET_KEY=sk-lf-test OPENAI_API_KEY=test-openai-key \
uv run --frozen --no-sync --python 3.12 pytest -n 4 --dist worksteal -q tests/unit
uv run --frozen --no-sync --python 3.12 ruff check .
uv run --frozen --no-sync --python 3.12 mypy langfuse --no-error-summary
uv run --frozen --no-sync --python 3.12 ruff format --check \
  langfuse/experiment.py langfuse/_client/client.py \
  langfuse/_client/datasets.py tests/unit/test_experiment.py

• Complete unit suite on the updated v5 base: 958 passed, 2 skipped, 17 warnings, exit 0. Includes real SQLite compatibility through all three public entry points, opt-in task overlap and trace parenting, capacity with a one-worker default executor, and cancellation while a synchronous call remains running.
• Independent automated review found a late-coroutine cancellation cleanup issue; it was reproduced, fixed, and independently rechecked. The smoke confirms the late coroutine is closed without executing it, queued calls remain cancelled, and the dedicated worker exits after release.
• Ruff lint, mypy and formatting of the four changed Python files pass. Generated API code and unrelated files match the v5 target.
• No Langfuse-server E2E, live-provider calls, other Python versions or other platforms were exercised. Local verification does not imply upstream CI success or maintainer approval.

Checklist

• I self-reviewed the diff using code_review.md.
• I added or updated tests for behavior changes.
• I updated the API documentation for the opt-in execution context.
• Relative to the target, generated API files are unchanged.
• I did not commit secrets or credentials.

AI assistance was used for implementation, tests, description and automated review.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ po-et
❌ niklassemmler
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[chrikrah]

@po-et this fixes the serialization: sync tasks overlap now, and two of your new cases plus the concurrency case fail on main. Your docs note already says thread-affine tasks should move to async. I'd ship it in v5 and not in a patch, because the cost of missing that note is every item.

$ pytest -q tests/unit/test_experiment.py   # uv sync --locked, Python 3.12
36 passed                  # head 47c4ee4
3 failed, 33 passed        # langfuse/experiment.py from bf11ec1, your tests kept
FAILED ...::TestExperimentObservationTree::test_sync_tasks_run_concurrently_with_trace_context

$ python lf_sync_task_sqlite.py   # sync task reads a sqlite3 connection opened by the caller, max_concurrency=2
outputs: ['A', 'B'] items: 2   # bf11ec1
Item 0 failed: SQLite objects created in a thread can only be used in that same thread. ...
outputs: [] items: 0           # head 47c4ee4
# not run: the e2e suites against a Langfuse server

non-blocking: called from a plain script, run_experiment returns zero item results for that task and raises nothing. The failure shows only as an ERROR log line and an error span in the trace. In a notebook, run_async_safely already moves the loop off the caller's thread, so scripts are what changes. With the bug-fix box ticked it would ship with no migration note. I'd retarget this to prepare-v5-release with a ! in the title and tick "Breaking change". #1934 landed experiment.py changes there, so expect a rebase.

@hassiebp does this belong on prepare-v5-release?

[cursor]

Thanks a lot for this, the implementation looks solid. Context propagation into the worker threads and the async detection both hold up well in review.

We've decided not to include it in v5.0, which ships tomorrow. Changing the default execution context the day before a major release is too risky for us. A few things also need more design work first:

• Tasks that hold thread-bound resources, such as sqlite connections or some client sessions, would start failing, and those items are only logged.
• The default executor caps effective concurrency at min(32, cpus + 4). For example, max_concurrency=50 on 8 CPUs ran only 12 tasks at once. A dedicated ThreadPoolExecutor(max_workers=max_concurrency) would avoid that.
• Ctrl-C can't stop tasks that are already running in threads.

We'd be happy to ship this in a v5.x minor release behind an opt-in flag, so the default doesn't change. Once v5 is out, could you retarget this PR to main and add the flag? The migration notes can then go into the regular docs, instead of the MIGRATION_V5_SYNC_TASKS.md and README additions.