Skip to content

[CHORE] Choose an outlier-resistant statistic for benchmark charts #160

Description

@lukekarrys

Context

Registry charts publish the mean of a small number of runs (three for registry variations). One outlier can dominate the daily point: a single 294-second run produced a roughly 100-second mean during the incident. Standard deviation and median are already present in the raw data.

This is a methodology decision rather than a one-line display change. Options include:

  • plot median;
  • retain mean and add error bands/confidence information;
  • separate run 0/cold behavior from steady-state runs.

This should compose with the partial-failure work so timeout survivors are not treated as complete samples.

Acceptance criteria

  • The canonical statistic for benchmark charts is chosen and documented.
  • The UI exposes enough variability/sample information to interpret small-run results.
  • Cold/run-0 behavior is either intentionally included or intentionally separated.
  • Processing, latest results, history, summaries, and leaderboard calculations use compatible semantics.
  • Tests cover a sample containing one extreme outlier.

Source

Performance investigation: item 7 — mean-of-three amplifies outliers

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions