Skip to content

enhance(build): validate built data.json against schemas - #27685

Merged
caugner merged 8 commits into
mainfrom
27564-validate-data-against-schema
Nov 5, 2025
Merged

caugner merged 8 commits into
mainfrom
27564-validate-data-against-schema

Conversation

@caugner

@caugner caugner commented Aug 27, 2025 •

Copy link
Copy Markdown
Contributor

Summary

  1. Validates the built data.json against the schemas.
  2. Removes the maxProperties requirement for the browsers object from the browsers schema.

Test results and supporting details

This revealed two minor issues:

  1. The browsers schema was requiring a maximum of one browser defined in browsers. This requirement only applies to the individual browser/*.json files, not the built data.json, so this requirement was removed.
  2. The __meta object in the built data.json is not currently covered by the schema, so we cannot validate it.

Related issues

Fixes #27564.

@github-actions github-actions Bot added schema Issue or pull request concerning the JSON schema. infra Issue concerning project infrastructure, such as npm, GitHub Actions, or releases. linter Issue or pull request concerning JSON tests or linting. scripts Issue or pull request concerning scripts in . labels Aug 27, 2025
@caugner caugner changed the title enhance(build): validate data against schemas enhance(build): validate built data.json against schemas Aug 27, 2025
@caugner
caugner requested review from Elchi3 and queengooborg August 27, 2025 09:23
@github-actions github-actions Bot added the size:m Pull request changing 25-100 lines of code. label Aug 27, 2025
@github-actions

github-actions Bot commented Aug 27, 2025 •

Copy link
Copy Markdown
Contributor

Tip: Review these changes grouped by change (recommended for most PRs), or grouped by feature (for large PRs).

Comment thread scripts/build/index.ts
const writeData = async () => {
const dest = new URL('data.json', targetdir);
const data = await createDataBundle();
validate(data);

@caugner caugner Aug 27, 2025 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

If validation fails, the validation errors are logged via console.error, but we don't bail out, to avoid disruption for contributors in the unlikely event that this ever happens on main. In either case, these errors won't get unnoticed, because they will appear on every npm install (via the prepare script).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that a PR that breaks schema validation won't fail the build? I can kinda see the case for never failing during the prepare script, but it'd be nice if there were a test that does fail. I won't block on this though, as it would be a step up from what we have.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about checking process.env.CI and failing if we're in CI?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would help for local development, but not pull request checks. Maybe check if GITHUB_REF is refs/heads/main and, if true, convert errors to warnings, else do nothing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In pull request checks, we always have CI=true in the environment, so we would notice it in a PR before merging. (In fact, the PR check would fail, preventing a merge.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's tackle this afterwards.

@ddbeck

ddbeck commented Aug 28, 2025

Copy link
Copy Markdown
Contributor
  • The __meta object in the built data.json is not currently covered by the schema, so we cannot validate it.

FWIW, I raised this as a concern in #17574 and got shot down on doing anything about it.

@caugner

caugner commented Aug 28, 2025

Copy link
Copy Markdown
Contributor Author

FWIW, I raised this as a concern in #17574 and got shot down on doing anything about it.

FWIW, here's the meta schema (isolated, like the other two):

{
  "$schema": "http://json-schema.org/schema",

  "definitions": {
    "meta_block": {
      "type": "object",
      "properties": {
        "timestamp": {
          "type": "string",
          "format": "date-time"
        },
        "version": {
          "type": "string"
        }
      },
      "required": ["timestamp", "version"],
      "additionalProperties": false
    }
  },

  "properties": {
    "__meta": {
      "$ref": "#/definitions/meta_block"
    }
  },

  "title": "MetaData",
  "type": "object",
  "required": ["__meta"],
  "additionalProperties": false,
  "maxProperties": 1,
  "minProperties": 1
}

Unfortunately, ajv gives me an error when validating the data:

TYPE must be string

  2 |   "__meta": {
  3 |     "version": "7.1.0",
> 4 |     "timestamp": "2025-08-28T14:30:46.924Z"
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^ 👈🏽  type must be string
  5 |   }
  6 | }

That error doesn't make sense, but given this is not a priority, I would declare the scope of this PR to validate the built data against the existing schemas.

Comment thread utils/ajv.ts
Comment thread schemas/browsers.schema.json
@caugner
caugner requested a review from a team as a code owner September 16, 2025 13:38
@caugner
caugner requested review from queengooborg and removed request for Elchi3 September 16, 2025 13:43
Comment thread scripts/build/index.ts
@caugner
caugner force-pushed the 27564-validate-data-against-schema branch from b884975 to c046688 Compare October 7, 2025 12:32
@github-actions github-actions Bot added data:api Compatibility data for Web API features. https://developer.mozilla.org/docs/Web/API size:l Pull request changing 101-1,000 lines of code. and removed size:m Pull request changing 25-100 lines of code. labels Oct 7, 2025
@caugner
caugner force-pushed the 27564-validate-data-against-schema branch from c046688 to fabb4f2 Compare October 14, 2025 13:11
@github-actions github-actions Bot added size:m Pull request changing 25-100 lines of code. and removed data:api Compatibility data for Web API features. https://developer.mozilla.org/docs/Web/API size:l Pull request changing 101-1,000 lines of code. labels Oct 14, 2025
@caugner

caugner commented Oct 17, 2025

Copy link
Copy Markdown
Contributor Author

@queengooborg If we can get this in, we can split up the schema into internal/public, and validate separately. Currently, it's a mix of both.

@Elchi3 Elchi3 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the schema change 👍

@caugner
caugner requested a review from a team as a code owner November 4, 2025 11:05
@caugner
caugner requested a review from argl November 4, 2025 11:05

@ddbeck ddbeck left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reviewed the code. Thank you!

Comment thread scripts/build/index.ts

for (const [key, value] of Object.entries(data)) {
if (key === '__meta') {
// Not covered by the schema.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also 😒 #17574

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See also: #27685 (comment)

Comment thread scripts/build/index.ts
const writeData = async () => {
const dest = new URL('data.json', targetdir);
const data = await createDataBundle();
validate(data);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that a PR that breaks schema validation won't fail the build? I can kinda see the case for never failing during the prepare script, but it'd be nice if there were a test that does fail. I won't block on this though, as it would be a step up from what we have.

@caugner
caugner merged commit 0dde3fe into main Nov 5, 2025
12 checks passed
@caugner
caugner deleted the 27564-validate-data-against-schema branch November 5, 2025 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infra Issue concerning project infrastructure, such as npm, GitHub Actions, or releases. linter Issue or pull request concerning JSON tests or linting. schema Issue or pull request concerning the JSON schema. scripts Issue or pull request concerning scripts in . size:m Pull request changing 25-100 lines of code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The built package is not validated against the schema

6 participants