Skip to content

Support async sources - #3248

Open
timcassell wants to merge 2 commits into
masterfrom
feature/async-enumerable-params-source
Open

Support async sources#3248
timcassell wants to merge 2 commits into
masterfrom
feature/async-enumerable-params-source

Conversation

@timcassell

@timcassell timcassell commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Adding support for async sources revealed several issues with the system. Discovery and codegen were refactored here to be able to support it, and a few smaller issues were also fixed here. Other issues will be addressed separately.

Resolves #3120

Core changes:

  1. [ParamsSource]/[ArgumentsSource] may return IAsyncEnumerable<T>, which BenchmarkDotNet awaits when reading the values.
    • The source method may take an optional [EnumeratorCancellation] CancellationToken.
  2. Added async discovery overloadsBenchmarkConverter.TypeToBenchmarksAsync/MethodsToBenchmarksAsync, TypeFilter.FilterAsync.
  3. Non-constant parameter values obtained in the child process strategy changed
    • An instance source is read in the constructor and passed back through an out parameter to be assigned in the object initializer, so required and init members are respected and await works.
  4. Params are now language-agnostic
    • IParam replaced by a closed ParameterValue hierarchy. A toolchain reads ParameterInstance.ParameterValue and emits whatever its language needs, instead of a param returning a C# string.
    • ParameterDefinition is metadata only. ParameterValues pairs a definition with the values it ranges over.

Behavior changes:

  1. A base type's static [Params]/[ParamsSource]/[ParamsAllValues] member is now a parameter.
    • Reflection withholds inherited statics without FlattenHierarchy and discovery never asked, so such a member was silently no parameter at all and the benchmark ran once on the default value.
  2. A base type's static [BenchmarkCancellation] member is now assigned. Same cause; cancellation silently did nothing.
  3. Arguments from one [ArgumentsSource] row share a single read in the child process (no change to in-process).
    • The source is obtained and enumerated once. Previously it was obtained and enumerated multiple times (once per argument).
  4. ExecutionValidator and ReturnValueValidator skip a benchmark whose by-ref-like shapes reflection cannot handle, instead of refusing it.
    • Generated code executes just fine, only reflection is restricted. They now yield a non-critical error instead of a critical exception.
  5. IAsyncDisposable parameters are now disposed asynchronously.
  6. required members not annotated with [Params*] or [BenchmarkCancellation] now cause the benchmark type to be rejected
    • Previously codegen cheated and lied by adding [SetsRequiredMembers] to the runnable ctor. Since the codegen changed back to object initializer syntax, that hack was no longer necessary.

Breaking changes:

  1. Removed IParam, ArrayParam<T>, EnumParam — replaced by ParameterValue.
  2. Removed ParameterInstance.ToSourceCode() — read ParameterValue instead. The object-taking constructor is now internal (for tests only).
  3. Removed ParameterDefinition.Values, and its constructor's values parameter with it — a definition is metadata, and the values live on the new ParameterValues.
  4. Removed ParameterDefinitions
  5. ParameterInstances no longer implements IEquatable<ParameterInstances> and no longer overrides Equals/GetHashCode. Value comparison lives in ParameterEqualityComparer.
  6. ParameterExtractor.GetParameter<T> replaced by GetParameterAsync<T> (codegen-only, [EditorBrowsable(Never)]).
  7. CorrectionsSuggester is now internal — used only by console-argument plumbing for --filter typo suggestions.
  8. Removed DisassemblerConstants — its single constant value was moved to the internal RunnableConstants.

Other Changes:

  1. Fixed a parameter member being looked up by name alone. A [Params] member hiding an inherited one threw AmbiguousMatchException on master under both in-process toolchains; the lookup now matches the declared type and picks the most derived member. Relatedly, when both declarations carry the attribute the pair no longer becomes two parameters of one name - reflection reports a hidden field twice where it collapses a hidden property - which multiplied the cases against themselves and named the member twice in the runnable's object initializer.
  2. Fixed ExecutionValidator and ReturnValueValidator reporting a spurious error for any benchmark taking [Arguments]no property or field <name> found on master, because an argument was looked up as though it were a member.
  3. Fixed two ways DynamicAwaitHelper crashed the process while awaiting a user's awaitable, both from asking it for an interface map it cannot supply. An awaiter implementing only INotifyCompletion - legal, ICriticalNotifyCompletion is optional in the pattern - threw an unhandled ArgumentException: Interface not found; an awaiter whose declared type is an interface - the pattern binds on the declared type - threw ArgumentException: 'this' type cannot be an interface itself. Both landed on AwaitUnsafeOnCompleted, where they were rethrown onto the thread pool with nothing to catch them.
  4. Generated member renames, freeing those names for user members.
    • Run -> __Run, FieldsContainer -> __FieldsContainer
  5. Dropped System.Linq.AsyncEnumerable dependency, re-implemented operations inline to use the BenchmarkSynchronizationContext instead of the ambient SynchronizationContext.
  6. Updated docs and samples.
  7. New analyzer rules and validators.
  8. BDN1206 analyzer rule removed
    • init-only [Params] members are supported via assignment through an object initializer (in-process toolchains already supported them).
    • Code-gen min lang version set to C# 9 due to compiler requirements (CS8400).
  9. New and expanded tests.

Note for anyone implementing IValidator/IDiagnoser outside this repo on a pre-net10 target: IAsyncEnumerable<T> itself flows from the Microsoft.Bcl.AsyncInterfaces reference, so an async iterator needs nothing extra - only the LINQ helpers (AsyncEnumerable.Empty<T>(), ToAsyncEnumerable()) require adding the System.Linq.AsyncEnumerable package, which is in the framework from .NET 10.

Refactored parameter discovery and codegen.
Added/removed analyzer rules.
Added tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adapts the branch to the toolchain/runtime refactor (#3232):

- `InProcessEmitToolchain`'s constructor is now private behind the `From`
  factory; `AsyncEnumerableParamsSourceTests` updated accordingly.
- The branch's `9.0` LangVersion floor is re-applied in master's renamed
  `CsProjNativeAotGenerator`; `MonoAOTLLVMCsProj.txt` is gone with its toolchain.
- `BenchmarkCase` keeps both master's `GetToolchain()` and the branch's
  `DisposeAsync()`.
- `MockInProcessDiagnoser` is a new netstandard2.0 satellite that bound the
  branch's internal `System.Linq.AsyncEnumerable` polyfill, which is compiled out
  of BenchmarkDotNet's .NET 10 asset. It now iterates locally, matching
  `EtwProfiler` and `FinalizerBlockerDiagnoser`.

The analyzer resource files are restored to the repository's CRLF+BOM encoding.
The branch had rewritten both to LF, which made every line conflict.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Params/ArgumentsSource support IAsyncEnumerable

1 participant