Implement local-first mode and CLI for test run reporting - #110
Merged
Conversation
Proves the SDK can collect test data with no API key configured, which is the
prerequisite for a local-first standalone experience.
Introduces XpingMode (Auto/LocalOnly/Connected/Disabled), resolved once at
startup. The blocker was never the uploader: [Required] on ApiKey/ProjectId made
ValidateDataAnnotations() throw before any mode could be resolved, and the
orchestrator's recovery path then swapped in NoOpTestExecutionCollector - so a
missing key meant nothing was collected at all, not merely "not uploaded".
- Remove [Required] from ApiKey/ProjectId; Validate() is now mode-aware
- Strict mode still forces Connected, so CI keeps failing fast on a missing key
- LocalOnly skips HttpClient, retry pipeline and circuit breaker entirely, and
suppresses network metrics (a DNS lookup plus 4 pings with a fixed 300ms of
inter-ping delay - pure waste when nothing will be uploaded)
- Split IsHealthy into IsCollecting/IsUploading so collection survives without
credentials
- Fix the finalize drain loop, which terminated on the uploader's acknowledged
record count. NoOpXpingUploader always reports zero, so any non-uploading mode
drained a single batch and silently discarded the rest of the buffer.
- Add LocalRunReportWriter: end-of-run summary on the raw stdout handle,
verified to survive vstest output capture on NUnit, xUnit and MSTest
Tests: 637 pass in Core.Tests. 15 previously encoded the old contract
("no ApiKey" == invalid) and were repointed at conditions that are invalid in
every mode; new coverage for mode resolution, local-only collection and the
drain-loop regression.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Zero-config `dotnet test` now stores each run locally and prints a summary with cross-run flakiness, with no account and no API key. Storage is one gzipped JSONL file per run, no database and no new package references. Concurrent test assemblies write disjoint files, so nothing locks; a run truncated by a killed test host costs that run alone, because the reader drops the torn trailing line and keeps everything before it. Filenames carry the timestamp sort key, which removes the need for a shared index. Records are a slim projection of TestExecution - fingerprint, name, outcome, duration, retry attempt, error hash - and are therefore not re-uploadable; the outbox tier for recovering failed uploads is separate and not in this change. Fingerprints are stored in full so local history can join against cloud identity. Analysis covers four signals: intra-run retry flake, cross-run pass/fail flip, newly failing, and consistently failing. Intra-run flake is what keeps the very first run from rendering an empty report. The platform's six-factor confidence score is deliberately not reproduced locally - local history cannot contain CI runs, other machines or teammates, so a local number would look like the dashboard's and disagree with it. Two problems found while building this: - Output written to the test host's stdout is discarded by vstest at the default console verbosity (minimal); it only surfaces at normal or above. Bypassing Console.SetOut, as RawConsoleSink does, reaches the pipe but not the developer. The report is now written to the controlling terminal (/dev/tty, CONOUT$) and falls back to stdout when none is attached. - All test projects in a solution share one store, so an unfiltered window mixed unrelated suites - MSTest's report listed xUnit's tests and counted their runs. Analysis is now scoped by assembly, taken from TestIdentity rather than GetEntryAssembly(), which under vstest resolves to the test host. Measured on a 2,000-test suite: 1.75ms to write a run, 0.30ms to project executions, 32ms to read and analyse a full 12-run window, 69 bytes per record on disk. Nothing is added to the per-test hot path. Tests: 699 pass in Core.Tests (+62), covering retention, torn files, forward schema compatibility, concurrent writers, assembly scoping, glyph fallback, layout alignment, and CTA throttling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…g.Cli
Xping.Sdk.Core now only records: it uploads for connected users and writes the
local store otherwise. Interpreting that store - flakiness analysis, report
rendering, the cloud invitation - moves to a new `Xping.Cli` tool, invoked as
`dotnet xping report` from a tool manifest or `xping report` when installed
globally.
The split is what the Phase 1 findings argued for. Printing from inside a test
host meant fighting vstest for the terminal: bypassing Console.SetOut only
reaches the test host's pipe, which the runner then discards at default console
verbosity. That fragility is inherent to the location, not to the
implementation, and a tool that owns its own stdout simply does not have it.
Scoping analysis by assembly stops being a constraint imposed by the in-process
report and becomes a CLI flag.
The SDK keeps one line, printed only when tests flaked on retry during the run
just observed:
[Xping] 2 tests flaked on retry this run - run `dotnet xping report` ...
That fact needs no store read and no cross-run comparison, so it costs nothing,
and without it a developer with no API key would see nothing after a test run
and have no reason to learn the tool exists. It stays silent on clean runs and
in CI.
- Core exposes LocalRunStore.Create as the supported entry point for reading the
store, keeping JsonLinesRunStore internal. Preferred over InternalsVisibleTo,
which would couple a shipping tool to Core's private surface.
- ILocalRunReporter is replaced by the narrower ILocalRunWriter.
- LocalAnalysis, UnstableTest and InstabilityKind stop being public Core types.
Free now because Phase 1 is unreleased; a breaking change after any release.
- RawConsoleReportSink is deleted; its stdout fallback is inlined into
TerminalWriter, which stays in Core for the retry hint and for the pending fix
to the SDK's own log lines.
- The analysis benchmark leaves Xping.Sdk.Benchmarks with the code it measured.
Two bugs found by running the tool against a real store: the summary said
"1 tests", and an unscoped report drew headline counts from one run while its
history spanned every assembly in the store. Reports now scope to the newest
assembly by default and say so when the store holds others.
Tests: 689 in Core.Tests, 51 in the new Xping.Cli.Tests. Moved tests were
relocated unchanged. Verified `dotnet xping report` at parity with the removed
in-process block, from a packed tool installed into a manifest.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes the CLI surface the split in fe794a7 made possible. `--all` reports across every assembly in the store. Each assembly is analysed against its own runs and the findings are merged afterwards, rather than pooling every run into one analysis: pooling would describe a suite seen in 12 of 36 runs against a 36-run window it never took part in, and "last N runs" would mean something different for every line in the block. For the same reason the aggregate header deliberately drops the pass/fail counts of the scoped report. Those describe a single run, and across assemblies there is no single latest run they could honestly describe. Findings carry their assembly so two suites with similarly named tests stay distinguishable. `--json` emits a versioned document for scripting and CI, carrying the run metadata and per-finding history rather than only the rendered conclusions. `where` answers "where is my data", which is otherwise unanswerable by inspection because the store location is discovered by walking up for a repository root. It also breaks down runs per assembly, which is the first thing to check when a report looks wrong. `clear` deletes runs, optionally scoped to one assembly. Local history is not recoverable - not in version control, not uploaded in local-only mode - so it confirms first, and refuses outright when stdin is redirected rather than assuming consent. Deletion is a store operation rather than file handling in the caller, so the on-disk layout stays an implementation detail; ILocalRunStore gains Delete and RunsPath for this. Two layout bugs found by running --all against a real multi-assembly store: the unstable-section header claimed "last 9 local runs" above findings reading 2/3, and the consistent-failure list truncated mid-name so the third name vanished with no indication it existed. Both now adapt to aggregate mode. Tests: 77 in Xping.Cli.Tests (+26), 689 in Core.Tests. Aggregation is covered for per-assembly window scoping, assembly tagging, and runs with no recorded assembly; clear is covered for the redirected-stdin refusal and scoped deletion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Until now the docs described a product that always required an account. Since Phase 0 the SDK has run local-only without credentials, and upgrading silently changed behaviour - missing credentials went from "SDK disabled" to "collecting locally" - with nothing explaining it. New pages: - getting-started/local-first.md - the no-account path end to end, including how to read the sparkline and what the four flakiness signals mean. - cli/command-reference.md - every command and option, exit codes, and the --json schema with a worked example. - configuration/local-store.md - location resolution, the self-ignoring .gitignore, retention, measured performance, and what is deliberately not stored. Updates: - The configuration reference gains an Operating Mode section covering the four modes and how Auto resolves, and its table gains Mode, XPING_LOCAL_STORE and XPING_NO_BANNER. ApiKey and ProjectId are no longer documented as unconditionally required, which had been wrong since Phase 0. - The three quickstarts no longer list an Xping account as a prerequisite. - The README leads its quick start with the no-account path and shows real report output. Two things stated explicitly because they are the questions users will actually have: strict mode still forces Connected, so a missing key in CI remains a hard error rather than a silent downgrade; and local-only deliberately shows no confidence score, because several of its six factors need data that cannot exist on one developer's machine. Verified with docfx: builds with no new warnings, all three pages render, CLI appears in the nav, and every relative link and toc href resolves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The docs now tell users to `dotnet tool install Xping.Cli`, so the package has to exist. Solution-level `dotnet pack` already picked the project up; what was missing was everything around it. - The release workflow pinned `dotnet-version: 8.x`, which cannot build the six net10.0 projects in this solution, including the CLI. It has been working only because the runner image happens to ship a newer SDK and nothing pins one. Both 8.0.x and 10.0.x are now installed explicitly. - Xping.Cli is uploaded to the GitHub release alongside the SDK packages. - The release build now installs the packed tool into a manifest and runs it before anything can be published. A tool that does not install is a broken release and nothing else in the pipeline would have caught it. - Xping.Cli.Tests was not run by CI at all, so its 77 tests never executed on any push or pull request. It is now part of the test job. - The CLI ships its own NuGet readme. The shared one describes installing SDK adapters, which is not what someone landing on the Xping.Cli page wants. Verified locally: solution pack produces all five packages at an injected version, the packed tool installs into a manifest and reports that version, and the package carries the DotnetTool package type, MIT license, icon and its own readme. Dependencies are bundled rather than declared, which is correct for a tool: 1.5 MB compressed, 4.2 MB unpacked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The five test projects multi-targeted net8.0;net10.0 and now target net10.0 only. CI, release, and package-publish workflows all install 10.0.x, and the CI matrix is a single .NET 10 leg. Documentation examples follow. The CI matrix previously read ['8.0.x', '10.0.x'], which implied the solution was verified on both. It was not: the solution contains net10.0 projects that a .NET 8 SDK cannot build, so that leg silently used whatever newer SDK the runner image provided. Removing it makes the real requirement explicit rather than dependent on the image. Locally this also removes the "You must install or update .NET" testhost aborts that appeared on every multi-targeted project, since only the .NET 10 runtime is installed on a current developer machine. Note the coverage this gives up: the Xping.Sdk.* packages target netstandard2.0 and are unchanged, but they are now exercised only on a .NET 10 runtime. Nothing verifies they still load and run on the older runtimes netstandard2.0 implies. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI hit EnforcesMaxAgeRetention failing with an empty store. The test was timing dependent, but the underlying defect is in the product. ApplyRetention computes `cutoff = UtcNow - MaxAge` after the incoming run has been written. With a short MaxAge, a write that takes longer than the window - routine on a loaded CI runner - leaves the just-written file already older than the cutoff, so retention deletes it and the store ends up empty. The same holds for MaxBytes when a single run exceeds the cap. Retention exists to bound history, not to discard the thing it was called to keep, so the newest run is no longer a deletion candidate. This was masked rather than caught: EnforcesMaxBytesRetention asserted `Count <= 1`, which tolerated exactly the empty store this produces. That assertion is now `Single`, and two tests pin the invariant directly - one with every limit set to zero, one where every run is over age. Reverting the one-line product change fails three tests, so they guard the behaviour rather than describing it. EnforcesMaxAgeRetention no longer sleeps. It backdates the file's write time explicitly, so it tests retention rather than how long a write happens to take. 15 consecutive runs, no failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds local-first test collection and a .NET CLI for analyzing stored test-run reliability without an account.
Changes:
- Introduces operating modes and a compressed local run store.
- Adds reporting, JSON output, aggregation, store inspection, and cleanup commands.
- Expands tests, documentation, packaging, and .NET 10 CI support.
Reviewed changes
Copilot reviewed 74 out of 75 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
Xping.Sdk.sln |
Adds CLI projects. |
tests/Xping.Sdk.XUnit.Tests/Xping.Sdk.XUnit.Tests.csproj |
Targets .NET 10. |
tests/Xping.Sdk.NUnit.Tests/Xping.Sdk.NUnit.Tests.csproj |
Targets .NET 10. |
tests/Xping.Sdk.MSTest.Tests/Xping.Sdk.MSTest.Tests.csproj |
Targets .NET 10. |
tests/Xping.Sdk.Integration.Tests/Xping.Sdk.Integration.Tests.csproj |
Targets .NET 10. |
tests/Xping.Sdk.Core.Tests/Xping.Sdk.Core.Tests.csproj |
Targets .NET 10. |
tests/Xping.Sdk.Core.Tests/Reporting/RetryFlakeHintTests.cs |
Tests retry hints. |
tests/Xping.Sdk.Core.Tests/Orchestration/XpingContextOrchestratorTests.cs |
Tests operating modes. |
tests/Xping.Sdk.Core.Tests/LocalStore/LocalTestRecordTests.cs |
Tests local projection. |
tests/Xping.Sdk.Core.Tests/LocalStore/LocalStorePerformanceTests.cs |
Adds performance budgets. |
tests/Xping.Sdk.Core.Tests/LocalStore/LocalStorePathResolverTests.cs |
Tests store resolution. |
tests/Xping.Sdk.Core.Tests/LocalStore/LocalRunWriterTests.cs |
Tests run persistence. |
tests/Xping.Sdk.Core.Tests/Helpers/ServiceHelper.cs |
Updates configuration fixtures. |
tests/Xping.Sdk.Core.Tests/Extensions/XpingServiceCollectionExtensionsTests.cs |
Tests mode-aware DI. |
tests/Xping.Sdk.Core.Tests/Configuration/XpingConfigurationTests.cs |
Tests mode resolution. |
tests/Xping.Sdk.Core.Tests/Configuration/XpingConfigurationBuilderTests.cs |
Tests builder modes. |
tests/Xping.Sdk.Benchmarks/LocalStoreBenchmarks.cs |
Benchmarks local storage. |
tests/Xping.Cli.Tests/Xping.Cli.Tests.csproj |
Defines CLI tests. |
tests/Xping.Cli.Tests/SequentialCollection.cs |
Serializes stateful tests. |
tests/Xping.Cli.Tests/ProgramTests.cs |
Tests CLI entry point. |
tests/Xping.Cli.Tests/Commands/ReportCommandTests.cs |
Tests report behavior. |
tests/Xping.Cli.Tests/Commands/CtaThrottleTests.cs |
Tests CTA throttling. |
tests/Xping.Cli.Tests/Commands/CliSurfaceTests.cs |
Tests CLI commands and JSON. |
tests/Xping.Cli.Tests/Analysis/LocalFlakinessAnalyzerTests.cs |
Tests local analysis. |
tests/Xping.Cli.Tests/Analysis/AggregateAnalyzerTests.cs |
Tests assembly aggregation. |
src/Xping.Sdk.Core/Services/Reporting/Internals/TerminalWriter.cs |
Writes terminal hints. |
src/Xping.Sdk.Core/Services/Reporting/Internals/RetryFlakeHint.cs |
Builds retry-flake hints. |
src/Xping.Sdk.Core/Services/LocalStore/LocalStoreOptions.cs |
Defines retention settings. |
src/Xping.Sdk.Core/Services/LocalStore/LocalRunStore.cs |
Exposes the store factory. |
src/Xping.Sdk.Core/Services/LocalStore/Internals/LocalStorePathResolver.cs |
Resolves store paths. |
src/Xping.Sdk.Core/Services/LocalStore/Internals/LocalRunWriter.cs |
Persists completed runs. |
src/Xping.Sdk.Core/Services/LocalStore/Internals/JsonLinesRunStore.cs |
Implements compressed storage. |
src/Xping.Sdk.Core/Services/LocalStore/ILocalRunWriter.cs |
Defines writer contract. |
src/Xping.Sdk.Core/Services/LocalStore/ILocalRunStore.cs |
Defines store contract. |
src/Xping.Sdk.Core/Models/Local/LocalTestRecord.cs |
Defines compact test records. |
src/Xping.Sdk.Core/Models/Local/LocalRun.cs |
Defines stored runs. |
src/Xping.Sdk.Core/Extensions/XpingServiceCollectionExtensions.cs |
Registers mode-aware services. |
src/Xping.Sdk.Core/Configuration/XpingMode.cs |
Defines operating modes. |
src/Xping.Sdk.Core/Configuration/XpingConfigurationBuilder.cs |
Adds mode configuration. |
src/Xping.Sdk.Core/Configuration/XpingConfiguration.cs |
Resolves and validates modes. |
src/Xping.Cli/Xping.Cli.csproj |
Packages the .NET tool. |
src/Xping.Cli/Reporting/ReportGlyphs.cs |
Defines terminal glyph sets. |
src/Xping.Cli/Reporting/JsonReportWriter.cs |
Emits versioned JSON. |
src/Xping.Cli/Program.cs |
Dispatches CLI commands. |
src/Xping.Cli/Commands/WhereCommand.cs |
Reports store location. |
src/Xping.Cli/Commands/ReportOptions.cs |
Parses report options. |
src/Xping.Cli/Commands/ReportCommand.cs |
Generates reports. |
src/Xping.Cli/Commands/CtaThrottle.cs |
Rate-limits invitations. |
src/Xping.Cli/Commands/ClearCommand.cs |
Deletes local history. |
src/Xping.Cli/Analysis/LocalFlakinessAnalyzer.cs |
Classifies instability. |
src/Xping.Cli/Analysis/LocalAnalysis.cs |
Defines analysis results. |
src/Xping.Cli/Analysis/AggregateAnalyzer.cs |
Merges assembly analyses. |
README.md |
Documents local-first usage. |
nuspec/README.Cli.md |
Adds CLI package documentation. |
docs/toc.yml |
Adds CLI navigation. |
docs/guides/self-hosted-testing.md |
Updates .NET version. |
docs/getting-started/toc.yml |
Links local-first guide. |
docs/getting-started/quickstart-xunit.md |
Makes credentials optional. |
docs/getting-started/quickstart-nunit.md |
Makes credentials optional. |
docs/getting-started/quickstart-mstest.md |
Makes credentials optional. |
docs/getting-started/local-first.md |
Adds local-first guide. |
docs/getting-started/ci-cd-setup.md |
Updates CI to .NET 10. |
docs/configuration/toc.yml |
Links store documentation. |
docs/configuration/local-store.md |
Documents local storage. |
docs/configuration/configuration-reference.md |
Documents operating modes. |
docs/cli/toc.yml |
Adds CLI documentation navigation. |
docs/cli/command-reference.md |
Documents CLI commands. |
.gitignore |
Ignores macOS metadata. |
.github/workflows/release.yml |
Packages and verifies the CLI. |
.github/workflows/publish-github-packages.yml |
Uses .NET 10. |
.github/workflows/ci.yml |
Runs .NET 10 and CLI tests. |
Suppressed comments (1)
docs/getting-started/local-first.md:104
- These two rows reverse the legend and describe a state the analyzer does not produce:
○○●…shows old failures followed by recovery, whileNewlyFailingis implemented as exactly one latest failure after prior passes. Remove the unsupported “recently broken” row and align the newly-failing pattern with the implementation.
| `●●○●●●○●●●●○` | Flaky — passes and fails without a clear cause |
| `○○●●●●●●●●●●` | Recently broken — was fine, now consistently failing |
| `●●●●●●●●●●○○` | Newly failing — a regression, not flakiness |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
All sixteen comments were valid. Grouped by what they were actually about: Correctness of the store - A malformed run header threw JsonException past TryReadFile's catch, so one corrupt file aborted ReadRecent and Delete entirely instead of costing a single run. Record deserialization was already guarded; the header was not. - Path resolution returned the repository root without checking it was writable, so a read-only checkout never reached the profile fallback the docs promise and silently lost every run. Resolution now probes with a real file write. - An out-of-range Mode (Xping:Mode=99, or a cast) was honoured as an implicit mode, selecting a no-op uploader while leaving local-only network suppression off - a state no configuration should be able to produce. Validate() reports it and resolution falls through to Auto. Destructive command safety - `xping clear --force --assembly` deleted every run, because scanning for flags cannot distinguish a value from a missing one. clear and where now parse strictly and reject unknown options and missing values before deleting. - Parse errors advertised `xping <verb> --help`, which was itself rejected as an unknown option. Subcommand help is now handled before parsing. Analysis correctness - `--all --last N` read N x assemblies globally, so one busy suite owning the newest runs could push another suite out of the window entirely. Each assembly now gets its own window. - A retry flake was tracked as a boolean, so a test that flaked once and then stopped running kept reporting "flaked inside this run". It now records which run the flake happened in. - Cross-run classification applied the three-run minimum to the window but not to how many times the test was actually observed, so two observations were enough to call a test flaky. - Aggregate HasSufficientHistory was derived from the total across assemblies, so three assemblies with one run each read as sufficient. It now uses the largest single-assembly window, which is what the computed-but-discarded maxWindow was for. - The CTA passed isConnected: false unconditionally, pitching signup at existing customers. The run header now records connected status and the CLI derives it. - ReportCommand bypassed ReportGlyphs.Detect() and always emitted Unicode, so a console on a legacy code page got mojibake despite the documented `auto` default. Documentation and benchmarks - Every sparkline example labelled "newly failing" was reversed: with an oldest-to-newest legend and ● meaning pass, they depicted a recovered test. The legend table was wrong in both directions. Corrected in the README, the CLI package readme, and the local-first guide to match the analyzer. - The CLI targets net10.0 and cannot launch on older runtimes; installation instructions now say so, and note that the netstandard2.0 SDK packages are unaffected. - The store benchmark built fingerprints from one repeated character, which gzip compresses away, understating the write path it exists to measure. Tests: 704 in Core.Tests (+13), 92 in Xping.Cli.Tests (+15). Each fix has coverage, including the clear misparse asserting no runs were deleted and the aggregation window asserting a rarely-run suite still appears. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 78 out of 79 changed files in this pull request and generated no new comments.
Suppressed comments (11)
src/Xping.Sdk.Core/Services/LocalStore/Internals/LocalStorePathResolver.cs:71
- When a repository is found but is read-only, the fallback is keyed by
origin. The SDK's origin is itsbin/...directory while the CLI passes the repository working directory, so they resolve different profile stores and reports appear empty. Key this fallback by the discovered repository root; useoriginonly when no root was found.
src/Xping.Cli/Analysis/LocalFlakinessAnalyzer.cs:176 latestFailedmeans “latest observation failed,” not “the test failed in the latest run.” If a test passes, then fails, then is absent from newer runs, it is still labeledNewlyFailingeven though the documented current run did not execute it. Track the global index of the latest observation and require it to equaltotalRuns - 1for this classification.
// Failing now after an unbroken run of passes reads as a regression, not as flakiness.
if (latestFailed && passes == runCount - 1)
return Build(InstabilityKind.NewlyFailing);
src/Xping.Cli/Reporting/LocalRunReportWriter.cs:240
- This scoped headline uses the suite's total run count even though consistent-failure classification uses only runs where each test appeared. A test that failed in 3 runs and was absent from 9 is reported as having “failed in all 12 runs.” Use observation-based wording because multiple findings can also have different
RunCountvalues.
src/Xping.Sdk.Core/Services/LocalStore/Internals/LocalStorePathResolver.cs:145 GetEntryAssembly()is the test host underdotnet test, not the test project (the orchestrator already documents this atXpingContextOrchestrator.cs:591-593). Starting discovery fromtesthost.dllcan miss the repository and write to a profile fallback thatdotnet xpingcannot find. Prefer the SDK assembly location, which is copied beside the test assembly, and use the entry assembly only as a fallback.
src/Xping.Cli/Commands/ReportCommand.cs:195- A run can contain multiple records for one fingerprint when retries are recorded (as covered by
CollapsesRetryAttemptsWithinASingleRun). Counting every record makes the headline report show one retried test as multiple tests and retain the failed intermediate attempt. Collapse each fingerprint to its highest attempt before computing totals and outcomes.
foreach (LocalTestRecord record in run.Records)
src/Xping.Cli/Commands/ReportCommand.cs:222
- Aggregate findings are scoped by assembly, but detail lookup matches only the fingerprint. Fingerprints are generated from fully qualified name and parameters without the assembly (
TestIdentityGenerator.cs:80-92), so identical test names in two projects cause each detail section to include the other assembly's runs. Apply the finding's assembly filter here as well.
LocalTestRecord? record = run.Records
.Where(r => string.Equals(r.Fingerprint, test.Fingerprint, StringComparison.Ordinal))
.OrderByDescending(r => r.Attempt)
.FirstOrDefault();
src/Xping.Cli/Reporting/JsonReportWriter.cs:60
TestCountcounts execution records rather than tests, so a test with multiple retry attempts is over-counted in the versioned JSON contract. Count distinct fingerprints (or otherwise collapse retries to the final attempt) to keep this field consistent with its name.
src/Xping.Sdk.Core/Services/LocalStore/LocalStoreOptions.cs:34- This public option is never read: the CLI assigns it but uses
ReportOptions.Lastdirectly, and the store does not consult it. Consumers can setAnalysisWindowwith no behavioral effect. Either wire it into the reporting API or remove it from the public retention options.
README.md:130 - This command only installs into a tool manifest if one already exists; in a fresh repository it fails despite the comment. Add the manifest creation step used by the CLI package documentation before installing.
dotnet tool install Xping.Cli # into a tool manifest
dotnet xping report
src/Xping.Cli/Reporting/LocalRunReportWriter.cs:230
- This aggregate wording is incorrect when a consistently failing test was absent from some runs. The analyzer classifies from observations only (
RunCountomits absent runs), so “every analysed run” can claim failures in runs where the test did not execute. Describe it as failing every time observed, or render each finding's own count.
This issue also appears on line 236 of the same file.
src/Xping.Cli/Reporting/LocalRunReportWriter.cs:289
- For aggregate reports this displays total runs across assemblies. With three assemblies having one run each,
HasSufficientHistoryis correctly false but the message says “3 of 3 runs,” implying the threshold was reached. DisplayLargestAssemblyWindowin aggregate mode so progress reflects the per-assembly history requirement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce a local-only operating mode that allows the SDK to collect test data without an API key, enabling a standalone experience. Implement a CLI for reporting and analyzing test run results, including cross-assembly aggregation and JSON output. Update documentation to reflect these changes and package the CLI as a .NET tool. Standardize on .NET 10.0 for builds and tests.