Repository navigation
feat(pull): warn about the aspects pull leaves out - #561
Merged
Merged
Conversation
feixufeixu
approved these changes
Oct 24, 2026
feixufeixu
left a comment
Collaborator
There was a problem hiding this comment.
Approving with two required fixes. In production a FULL read returns data only for the aspects an entry's type requires. So this pull stops storing the overview that #560 lists, and the new warning never prints. My sync.ts comment gives the fix, and my mocks.ts comment covers why the tests still pass. The other two comments are optional.
merr1988
force-pushed
the
feat/pull/warn-left-out-aspects
branch
from
October 26, 2026 17:59
a07e607 to
eea3915
Compare
feixufeixu
reviewed
Oct 26, 2026
Pull fetched each entry with only the aspect types catalog.yaml lists and its entry type requires, so it could not tell an author what it left out. A catalog.yaml written before init listed any aspects pulls a knowledge base document with an empty body and says nothing. Pull now fetches each entry with every aspect (the FULL view), keeps the aspects of the types it knows, and warns once for each aspect type it left out, naming the number of entries and how to pull it. What it stores is unchanged. Each scope names the aspects the service writes beyond the required ones (CatalogSource.systemAspects), which pull leaves out without a warning. For a BigQuery dataset these are bigquery-policy and usage, the aspects BigQuery writes on a table besides the ones its type requires. lookupEntry takes 'FULL' to return every aspect. Bug: b/576454554
A FULL read gives data only to the aspects an entry's type requires, and _fixEntry drops the rest, so pulling through it stored no listed aspect beyond the required ones and never warned. Pull now reads each entry with the aspect types it knows, as before, and stores that result. A new CatalogClient.lookupEntryAspectTypes reads the FULL view's keys alone and names each aspect type on the entry once, with project numbers resolved. Pull warns about a type the first read did not return, it does not know and the scope does not list as a system aspect. lookupEntry is back to its earlier form. The BigQuery system aspects also name data-profile and data-quality-scorecard, which Dataplex data scans write. Bug: b/576454554
…ects lookupEntryAspectTypes resolved the type of every aspect key on its own, and crm caches only the projects it can read. An entry with several field-path aspects of one type in an unreadable project made one failing lookup per key. It now resolves each distinct type once. Bug: b/576454554
merr1988
force-pushed
the
feat/pull/warn-left-out-aspects
branch
from
October 26, 2026 20:16
eea3915 to
55d6a0a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of b/576454554, after #560.
Problem
Pull fetched each entry with only the aspect types
catalog.yamllists and its entry type requires, so it could not say what it left out. With acatalog.yamlfrom before #560, a knowledge base document pulls with an empty body and kcmd is silent.Change
CatalogClient.lookupEntryAspectTypes, reads the FULL view's keys alone and names each aspect type on the entry once, with project numbers resolved. A FULL read gives data only to the aspects the entry's type requires, so pull cannot store from it.Warning: pull left out aspect type 'X' on N entries; list it under snapshot.aspects in catalog.yaml to pull it.CatalogSource.systemAspects: aspects the service writes beyond the required ones, left out without a warning. BigQuery dataset:bigquery-policy,data-profile,data-quality-scorecardandusage._fixEntryand the new method.Tests
pull_left_outscenario: two entries with an unlisted aspect and one withoverview, pulled with acatalog.yamlthat lists none. Nothing extra is stored, and pull warns once per type with the entry count.pull_bq: a table withusageandbigquery-policypulls without them and with no warning.console.warnand checksassert.warnings.lookupEntryreturns only the requested aspects, andlookupEntryAspectTypesevery aspect type on the entry.lookupEntryAspectTypesreads the FULL view and names each type once, with a project ID, looking up each type's project once.usagefailspull_bq; removing the warning or the second read failspull_left_out.tscclean; scenarios 24, layouts 2, gcp 13, semantic 2076, tool 421, all pass.