[alexshamrai]

Closes #278

Adds a timeline-report input. It adds a Timeline section to the job summary with three parts:

• a Mermaid Gantt chart of when each test ran on each thread or worker;
• a table of busy time, idle time and utilization per thread;
• a collapsible table of test timings.

- uses: ctrf-io/github-test-reporter@v1
  with:
    report-path: './ctrf/*.json'
    timeline-report: true

It answers questions a list of results can't:

• how evenly work is spread across workers;
• which long test keeps one worker busy while the others wait;
• what actually ran in parallel, and where workers sat idle;
• when failures and retries happened.

Examples

Rendered by GitHub, from the preview folder:

• Small E2E run on 3 workers
• 20,000 tests on 16 threads
• Two Gradle forks that share a thread name
• A shard re-run two hours later
• A real JUnit run with Gradle test-retry

Job summaries from real workflow runs:

• 16 fixtures
• JUnit tests on 2 Gradle forks with retries

Large runs

This answers the question on the draft. There is no maximum test count: the report scales the chart down instead.

• The Mermaid text stays under 20,000 characters, well inside GitHub's limit of 50,000.
• Tests too short to tell apart are merged into one bar labelled with their count.
• Failures are never merged into passes. A failure too short to see is drawn as a diamond or a thin sliver.
• When the threads need more than 40 rows, rows are inferred from start and stop times instead.
• If more than 40 rows remain, or the chart is still too large, the chart draws the rows with failures and the busiest ones. The timings table keeps the failures of the rows it doesn't draw.
• The timings table lists up to 100 tests. With more, it keeps failures, numbered bars, retried failures and the slowest tests.

What else it handles

• Rows: a row comes from threadId, or from the pytest-xdist worker in extra.worker.
  • A thread whose tests overlap in time is split into several rows, like the "Test worker" thread of every Gradle fork.
  • Without thread IDs, or when the threads need more than 40 rows, rows are inferred from start and stop times.
• Names: names are written on bars wide enough to hold them. GitHub's Mermaid has no hover text. So a bar too narrow for its name carries the number of its row in the timings table (the # column).
• Retries: runs before a retry (retryAttempts) are drawn on their test's thread where it was free, otherwise in a row of their own. A failed run before a retry is yellow with a dashed outline. CTRF has no field for the thread an earlier attempt ran on, so this placement is a best guess; Proposal: Add optional threadId to retryAttempts entries ctrf#74 proposes adding one.
• Several runs in one report (re-run jobs, merged shards):
  • the wait between runs is cut out of the chart at a gray line;
  • earlier runs that a later run repeated are left out, as when a reporter keeps appending to one file.
• Unusual data:
  • times in seconds, microseconds or nanoseconds are converted;
  • with whole-second times, the chart shows the order of tests;
  • Go parent tests are left out;
  • names that look like Mermaid or Markdown syntax are escaped.
• Custom templates: a timelineReport Handlebars helper exposes the same data (see docs/build-your-own-report.md).

Compatibility

• Opt-in (false by default). Other reports are unchanged.
• Needs Mermaid 10.4 or later, which GitHub.com has. On older GitHub Enterprise Server the chart may not draw, but the tables still work.
• Mermaid before 11.7 doesn't draw the gray line at a cut.
• In PR comments, the chart renders where GitHub renders Mermaid. Notification emails show the Mermaid source.

How to review

All the logic is in src/ctrf/timeline/:

File	What it does
types.ts	Data model and limits
collect.ts	Reads tests, splits runs, assigns rows
gantt.ts	Writes the Mermaid chart within the size budget
report.ts	Headline, notes and tables
text.ts	Escaping and text width
heap.ts	A small binary heap that keeps packing rows fast

The template is src/reports/timeline.hbs. The rest is wiring: the input, the report order and the helper. dist/ is rebuilt with npm run package.

Testing

• These all pass:
  • npm test (321 tests, including runs of 20,000 and 40,000 tests);
  • lint:check, format:check and build:check.
• npm run package reproduces dist/ from a clean npm ci.
• Rendered 1,460 generated and real reports with Mermaid 12.2.0, the version GitHub's viewer uses, in headless Chromium. There were no warnings, no chart over the size limit, and no label outside its bar.
• A new self-test job runs the report on ctrf-reports/timeline/ctrf-report.json.

[Ma11hewThomas]

I think this is a great idea, the report itself and using a mermaid gant chart. Design looks nice too!

How would it behave for large test runs? Should we add a max test count?

[alexshamrai]

I think this is a great idea, the report itself and using a mermaid gant chart. Design looks nice too!

How would it behave for large test runs? Should we add a max test count?

It should be tested for different setups, which I haven't done, yet: Long, short, rerun flaky tests etc. This is still a draft, that's why i created an issue #278 , to collect some feedback, and enhance requirements.

Not using mermaid there is an obvious issue, when test is failed on initialization and has very short duration, it is difficult to visualize.

For now, let's proceed with this design. Once I test more edge cases, I'll mark this PR as ready for review and tag you. Thank you!

[alexshamrai]

@Ma11hewThomas this is ready for review. I rebuilt it on top of current main and tested it on many setups: large and long runs, re-runs, retries, runs without thread IDs, Gradle forks and Go reports.

On your question about large runs: there is no max test count. The chart stays under 20,000 characters by merging tests too short to see into one bar labelled with their count. Failures are never merged. With more than 40 rows, it infers fewer rows from start and stop times, and if that is not enough, it draws the rows with failures and the busiest ones. Here is 20,000 tests on 16 threads.

The description has the details and more examples. The CI and self-test workflows are waiting for your approval to run.

[Ma11hewThomas]

Thank you @alexshamrai, the report is great! To be released shortly