CLDR-19641 Add JSON representation of XPath coverage levels to cldr-json by sffc · Pull Request #5909 · unicode-org/cldr

sffc · 2026-07-14T22:02:40Z

CLDR-19641

This PR adds a JSON representation of API-computed path-specific coverage levels to cldr-json tool output as coverageByXPath.json in cldr-misc-full.

As explained in the issue, this allows ICU4X and other consumers to inspect path-specific coverage requirements by region/locale for improved data slicing.

Here is coverageByXPath.json sample output generated from this PR: https://gist.github.com/sffc-bot/e1e2415c9ce1a35084cea1c0aed2d937

Implementation

Instead of raw XML translation, this implementation uses the CLDR Java CoverageLevel2 / SupplementalDataInfo APIs (cov.getLevel(xpath)) across locales to compute and serialize effective coverage levels (core, basic, moderate, modern) into cldr-misc-full.

Output layout:

Root Coverage Baseline: cldr-misc-full/coverageByXPath.json ({ "coverageByXPath": { "root": { ... } } })
Locale-Specific Overrides: cldr-misc-full/coverageByXPath/<bcp47loc>.json (e.g., it-CH.json, und.json containing { "coverageByXPath": { "<bcp47loc>": { ... } } })

🤖 Antigravity helped generate this change.

srl295

see comments in ticket:

need clear description of what data is requested and how it is to be used.

xpath doesn't make sense in a non-xml environment

srl295 · 2026-07-14T22:57:28Z

        }
    }

+    public void writeCoverageLevelsByXPath(String outputDir) throws IOException {


+1 this is the right way to hook this in

sffc-bot · 2026-07-14T23:39:17Z

🤖 This comment is written by Antigravity, an AI agent working with @sffc.

Size Analysis: Including `comprehensive` in `coverageByXPath.json`

Currently, coverageByXPath.json excludes comprehensive and undetermined levels, yielding a root baseline file of 693 KB (38.5 KB gzipped, 8,545 XPaths across basic, core, moderate, and modern).

If comprehensive level XPaths were included in coverageByXPath.json:

undetermined: Adds 0 XPaths (0 bytes).
comprehensive: Adds 16,517 XPaths to the root baseline.

Baseline Configuration	Total Root XPaths	Uncompressed Size	Gzipped Size
Excluding `comprehensive` (Current)	8,545	693 KB	38.5 KB
Including `comprehensive`	25,062	2.9 MB	145 KB

Note: Per-locale override files (e.g. coverageByXPath/it-CH.json) are unaffected and remain small (~2.3 KB).

Should comprehensive coverage level XPaths be included in the root coverageByXPath.json output file?

… cldr-misc-full Co-authored-by: Gemini <176961590+gemini-code-assist[bot]@users.noreply.github.com>

robertbastian · 2026-07-15T15:11:32Z

Shouldn't this generate JSON paths instead of XML paths?

sffc · 2026-07-15T18:03:42Z

XPaths are well-documented in the spec, and they are more stable. JSON paths are the opposite: not well-documented, and not as stable (they are subject to change when we find bugs). Maybe we want to have a JSON version of this in the future, though, which is why I named these files explicitly coverageByXPath.json.

…rce files Co-authored-by: Gemini <176961590+gemini-code-assist[bot]@users.noreply.github.com>

… against root coverage Co-authored-by: Gemini <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…aluated against root coverage Co-authored-by: Gemini <176961590+gemini-code-assist[bot]@users.noreply.github.com>

sffc-bot · 2026-07-16T21:03:53Z

Updated PR #5909: Harvest baseline XPaths across all available locales rather than relying solely on en.xml. This ensures XPaths present in non-English locales (such as //ldml/localeDisplayNames/territories/territory[@type="SA"][@alt="short"] ) are included in the root baseline coverageByXPath.json file.

… root coverageByXPath Co-authored-by: Gemini <176961590+gemini-code-assist[bot]@users.noreply.github.com>

jira-pull-request-webhook · 2026-07-16T21:12:50Z

Notice: the branch changed across the force-push!

tools/cldr-code/src/main/java/org/unicode/cldr/json/Ldml2JsonConverter.java is different

View Diff Across Force-Push

~ Your Friendly Jira-GitHub PR Checker Bot

Co-authored-by: Gemini <176961590+gemini-code-assist[bot]@users.noreply.github.com>

srl295 · 2026-07-21T12:59:19Z

+            } catch (NoSourceDirectoryException e) {
+                // Skip legacy metadata locale aliases that lack XML source files.
+            }


this sounds like it's some other bug, can you give an example?

srl295 · 2026-07-21T13:02:45Z

+        // Get a union of all XPaths in all locales of CLDR in main.
+        // Then, use root to determine the baseline coverage level for each XPath.
+        // We can't pull XPaths from only root.xml since that file contains only a
+        // subset of all data.


this isn't quite optimal, as fullIterable is the extra xpaths. Can be optimized later.

srl295 · 2026-07-21T13:04:01Z

+                throw new IllegalStateException(
+                        "Unexpected coverage level " + l + " for XPath: " + x);
+            }
+            // Skip COMPREHENSIVE level as it represents the catch-all max coverage tier.


Not sure about skipping comprehensive.

srl295 · 2026-07-21T13:04:51Z

+                            + miscDir.getAbsolutePath()
+                            + File.separator
+                            + "coverageByXPath.json using CoverageLevel2 API");
+            outf.println(gson.toJson(rootCoverageObj));


coverageByXPath.json is the root coverage. I still think it ought to be the und file alongside the others.

I still disagree about putting the root coverage in und.json. CLDR JSON has no inheritance structure. All the files are fully resolved. Since this is infeasible for these coverage levels, I am defining this structure to be "the diff between 'that coverage file over there' and this locale's coverage". That "coverage file over there" does not necessarily need to be root, and the algorithm for generating it is different from the algorithm that would be used for und.json: for example, it includes XPaths that are not in root.xml.

If we were to put the coverage levels in und.json, something needs to break:

There is no inheritance in CLDR JSON (but now there would be: but only for coverage levels)

The und locale is exported just like any other locale (but would not be: it would contain xpaths that don't exist in und.xml, for example)

The locale-specific coverage level files contain only locale-specific overrides (but if we make und contain only the und.xml levels, locale files will contain a lot more coverage levels for XPaths they have that aren't in und.xml)

Then add a: "basedOn": "und" to all files

Have you checked difference between en-MO, en-CA etc?

we might want the possibility of "basedOn": "en" to further reduce the data (even if we don't do that at first)

Since this is infeasible for these coverage levels

Why? CLDR-JSON contains mainly redundant data already. ICU4X-style data-compression techniques are not required for CLDR-JSON.

different from the algorithm that would be used for und.json: for example, it includes XPaths that are not in root.xml.

It might include xpaths that aren't in root.xml, but the root file (I'll call it that) has the coverage levels that pertain to root aka und. So we should call it und. I'm not sure why you want to have a separate not-und file in a parent directory, and then an empty und file.

Then add a: "basedOn": "und" to all files
...

Since this is infeasible for these coverage levels

Why? CLDR-JSON contains mainly redundant data already. ICU4X-style data-compression techniques are not required for CLDR-JSON.

The root coverage file is 1.01 MB. I don't want to copy the 1.01 MB into all 780 locales when there is an easy alternative. At the same time, I do not want to add full-fledged inheritance. I just want to take care of the lowest hanging fruit to reduce the unzipped json size by nearly 800 MB.

I would agree with you both if CLDR JSON had inheritance, but it doesn't. und is a locale just like any other locale. You can remove it from your archive and nothing breaks. No locale depends on any other locale. You can pull a locale based on directory names. Putting the root coverage XPaths in a metadata-like JSON file maintains this property. I am really quite surprised that the CLDR JSON maintainers are suggesting to break this invariant.

If adding the 800 MB of data will unblock this PR, I would rather do that than add inheritance, including inheritance to the locale named und.

What I don't understand is how you're taking the single coverageLevels.xml file, which is 210kB, and produce 800MB of data out of that in the first place. Yes if you do that you need to come up with unprecedented ways of data modelling to reduce that size; but do you need to do that?

srl295 · 2026-07-21T13:07:20Z

+        // Emit und.json with empty overrides map for undetermined locale
+        JsonObject undObj = new JsonObject();
+        JsonObject undInnerObj = new JsonObject();
+        undObj.add("coverageByXPath", undInnerObj);
+        undInnerObj.add("und", new JsonObject());


srl295 · 2026-07-21T13:07:52Z

                    "languages:language:menu",
                    "monthWidth:month:yeartype",
                    "characters:parseLenients:scope",
+                    "characters:placeholderBoundarySpacing:type",


robertbastian

The format for cldr-misc-full is cldr-misc-full/<bcp47loc>/<dataType>.json, not cldr-misc-full/<dataType>/<bcp47loc>.json
Inside the files the top levels are

{
  "main": {
    "<bcp47loc>": {
      
    }
  }
}

The root data file should be in a directory called und – root is not used in CLDR-JSON

srl295 · 2026-07-21T13:43:12Z

The format for cldr-misc-full is cldr-misc-full/<bcp47loc>/<dataType>.json, not cldr-misc-full/<dataType>/<bcp47loc>.json

Almost right.

It's cldr-misc-full/main/<bcp47loc>/<datatype>.json

Inside the files the top levels are
{
  "main": {
    "<bcp47loc>": {
      
    }
  }
}

Also close but not exactly right:

{
  "main": {
    "<bcp47loc>": {
      "<datatype>": {
      }
    }
  }
}

Maybe we should keep the same.

So what about this:

cldr-misc-full/main/fr/coverageByXPath.json

{
  "main": {
    "fr": {
      "coverageByXpath": {
      }
    }
  }
}

And have it strictly limited to the 'main' tree.

The root data file should be in a directory called und – root is not used in CLDR-JSON

Yeah, I'd prefer this to having the root data file with a special name.

robertbastian · 2026-07-21T13:45:07Z

Ah yes I forgot the /main

sffc · 2026-07-21T21:04:31Z

I'm very happy to put these wherever CLDR INFRA wants me to put them. Last week when I asked @srl295 on Slack, we came up with the structure currently in this PR. We discussed that since this is not actual locale data, it doesn't really make sense to put in the locale main directories alongside the actual locale data. But, again, I will put this wherever you think it should go. I assume I will go with #5909 (comment) unless I hear otherwise.

The only structural comment where I have an opinion is putting the reference coverage levels in coverageByXPath.json rather than und.json for reasons explained in #5909 (comment)

srl295 · 2026-07-25T02:47:05Z

It should go in the locale specific subdirectories with other files. And, the und file should be in the und subdirectory.

…cale> directories Put locale coverage level override files into cldr-misc-full/main/<locale>/coverageByXPath.json following standard CLDR JSON package layout without identity section. Add TODO comments for NoSourceDirectoryException and und locale handling. Co-authored-by: Gemini <176961590+gemini-code-assist[bot]@users.noreply.github.com>

sffc-bot · 2026-07-30T08:56:20Z

🤖 This comment is written by an AI agent working with @sffc.

Updated JSON File Layout for `coverageByXPath`

In recent commits, the per-locale coverage override files in cldr-misc-full have been moved to standard main/<locale>/ package directories and follow the standard CLDR JSON main object format (without an identity section).

1. Root Baseline File

Path: cldr-misc-full/coverageByXPath.json (top-level file in package)

Snippet:

{
  "coverageByXPath": {
    "root": {
      "core": [
        "//ldml/characters/exemplarCharacters",
        "//ldml/characters/exemplarCharacters[@type=\"auxiliary\"]",
        "//ldml/characters/exemplarCharacters[@type=\"numbers\"]"
      ],
      "basic": [
        "//ldml/dates/calendars/calendar[@type=\"gregorian\"]/dateTimeFormats/availableFormats/dateFormatItem[@id=\"Hm\"]",
        "//ldml/dates/calendars/calendar[@type=\"gregorian\"]/dateTimeFormats/availableFormats/dateFormatItem[@id=\"Hms\"]",
        "//ldml/dates/calendars/calendar[@type=\"gregorian\"]/dateTimeFormats/availableFormats/dateFormatItem[@id=\"Hmsv\"]"
      ],
      "moderate": [
        "//ldml/characters/ellipsis[@type=\"final\"]",
        "//ldml/characters/ellipsis[@type=\"initial\"]",
        "//ldml/characters/ellipsis[@type=\"medial\"]"
      ]
    }
  }
}

2. Per-Locale Override Files

Path: cldr-misc-full/main/<bcp47loc>/coverageByXPath.json (e.g., cldr-misc-full/main/en/coverageByXPath.json, cldr-misc-full/main/es/coverageByXPath.json)

Snippet (sample from en/coverageByXPath.json):

{
  "main": {
    "en": {
      "coverageByXPath": {
        "basic": [
          "//ldml/localeDisplayNames/scripts/script[@type=\"Dsrt\"]",
          "//ldml/localeDisplayNames/scripts/script[@type=\"Latn\"]",
          "//ldml/localeDisplayNames/scripts/script[@type=\"Shaw\"]"
        ],
        "moderate": [
          "//ldml/dates/calendars/calendar[@type=\"chinese\"]/dateFormats/dateFormatLength[@type=\"full\"]/dateFormat[@type=\"standard\"]/pattern[@type=\"standard\"]",
          "//ldml/dates/calendars/calendar[@type=\"chinese\"]/dateFormats/dateFormatLength[@type=\"long\"]/dateFormat[@type=\"standard\"]/pattern[@type=\"standard\"]",
          "//ldml/dates/calendars/calendar[@type=\"chinese\"]/dateFormats/dateFormatLength[@type=\"medium\"]/dateFormat[@type=\"standard\"]/pattern[@type=\"standard\"]"
        ]
      }
    }
  }
}

robertbastian · 2026-07-30T08:59:33Z

XPaths are well-documented in the spec, and they are more stable. JSON paths are the opposite: not well-documented, and not as stable (they are subject to change when we find bugs). Maybe we want to have a JSON version of this in the future, though, which is why I named these files explicitly coverageByXPath.json.

Coming back to this: if we change a JSON path, consumers have to update their code anyway, so it's not a problem if the coverage path also changes.

sffc · 2026-07-30T09:04:26Z

@robertbastian Do you have a suggestion on how to generate the files with JSON paths? Or are you suggesting that as potential future work?

robertbastian · 2026-07-30T09:08:56Z

I believe the code for transforming an XML path to a JSON path is in https://github.com/robertbastian/cldr/blob/main/tools/cldr-code/src/main/java/org/unicode/cldr/json/LdmlConvertRules.java.

Given that you're planning on hardcoding this data in ICU4X anyway, why don't you just hardcode the XML file in the short term, and we can actually take time to design this?

sffc · 2026-07-30T09:27:03Z

Given that you're planning on hardcoding this data in ICU4X anyway, why don't you just hardcode the XML file in the short term, and we can actually take time to design this?

I'm +1 on "actually take time to design this"

I'm -1 on the XML file as an interim solution, because the XML file is not very portable or easy for ICU4X to consume. It has weird variables that are only resolvable by using the CoverageLevels2 Java class.

robertbastian · 2026-07-30T09:28:37Z

Given that you only need this for a handful of paths, you could literally just have a Rust file with the data you need.

sffc · 2026-07-30T09:30:39Z

Given that you only need this for a handful of paths, you could literally just have a Rust file with the data you need.

My branch in ICU4X currently contains a filtered-down version of the files generated by the previous version of this PR, and they are linked into Rust code using the Abstract Memory FS.

robertbastian · 2026-07-30T10:56:28Z

With my CLDR hat: I don't think this can be merged, there are a lot of design decisions that you took that I don't agree with and that we should discuss:

Fully expanding all the coverage regexes
Using XML paths instead of JSON paths
Having language-specific files
Deduplicating against und

With my ICU4X hat: Your branch contains a good amount of code that is very specific to all of these decisions, which is why I don't want to merge the harcoded data from this PR, but simpler data

github-actions Bot assigned sffc Jul 14, 2026

This comment was marked as resolved.

Sign in to view

srl295 requested changes Jul 14, 2026

View reviewed changes

sffc force-pushed the cldr-19641-coverage-levels-json branch from 9809d17 to bce94de Compare July 14, 2026 22:26