Repository navigation
Conversation
…cuments. Adds a workflow which compares how the HTML API parses real-world HTML with and without a pull request. It fetches a fresh batch of documents from the latest CommonCrawl crawl, replays the same batch through the target branch and through the pull request merged into it, and leaves a report as a comment on the pull request. The report covers: - Tag Processor and HTML Processor parse times, with totals, per-document deltas, and percentage changes for all, small, medium, and large documents. - A chart of median parse time by document size for both checkouts, leaving out sizes at either end with too few documents for a meaningful median. - How many documents the HTML Processor does not support with each checkout, grouped by the reason it stopped and the line it stopped on. - Which documents became newly supported or unsupported. The analysis takes a while, so it only runs when the `html-api-commoncrawl` label is added to a pull request which changes the HTML API, or when the workflow is run manually with a pull request number. The label is removed as soon as it is added, so it can be added again to request another analysis. How many documents to fetch, how long to spend fetching them, and how many timing rounds to run are configurable. Documents are fetched and replayed with a vendored build of cc-analyzer, which is built without its default HTTP identity. The analysis scripts load the HTML API, and what `wp_kses()` needs, from an explicit list of files without loading WordPress, presenting a UTF-8 site at example.com. The label removal and the comment are handled by a separate `pull_request_target`/`workflow_run` workflow, because workflows triggered by pull requests from forks cannot write to them. It replaces the report from any earlier analysis, searching every page of comments for it. That workflow only runs from the default branch, so it can only be tested after this is merged, or on a fork's default branch. See #66430. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds `wp_kses( $html, 'post' )` as a third measurement beside the Tag Processor and the HTML Processor. Each document's time is recorded for both checkouts, and the report gains a section for it with the same per-size-group table, chart, and largest slowdowns as the parsers. Only the files `wp_kses()` needs are loaded, without WordPress's default filters such as those on `pre_kses`, so its times show how a change affects `wp_kses()` itself rather than what a full site would measure. The report notes this. Since `wp_kses()` is now measured, pull requests which change `kses.php` can also request the analysis. See #66430. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| name: HTML API CommonCrawl Comments | ||
|
|
||
| on: | ||
| pull_request_target: |
| - 'tools/html-api-commoncrawl/**' | ||
| - '.github/workflows/html-api-commoncrawl.yml' | ||
| - '.github/workflows/html-api-commoncrawl-comment.yml' | ||
| workflow_run: |
The first run of the CommonCrawl analysis spent half an hour fetching one WARC file: cc-analyzer re-summed its whole cache before storing each document, and read every record of the file, though only about a third are HTML responses. Rebuild cc-analyzer.phar from a cc-analyzer branch which keeps a running total of cached bytes, can sample captures from random blocks of a crawl's URL index, and downloads those records concurrently, retrying throttled transfers without holding up the others. The fetch now samples successful `text/html` captures from many hosts across the whole crawl, downloads only those records four at a time, and retries transfers which CommonCrawl throttles. See #66430. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Built with Claude code.
Trac ticket: Core-66430