Skip to content

feat(pull): warn about the aspects pull leaves out - #561

Merged
merr1988 merged 3 commits into
mainfrom
feat/pull/warn-left-out-aspects
Oct 26, 2026
Merged

merr1988 merged 3 commits into
mainfrom
feat/pull/warn-left-out-aspects

Conversation

@merr1988

@merr1988 merr1988 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Part 2 of b/576454554, after #560.

Problem

Pull fetched each entry with only the aspect types catalog.yaml lists and its entry type requires, so it could not say what it left out. With a catalog.yaml from before #560, a knowledge base document pulls with an empty body and kcmd is silent.

Change

  • Pull reads each entry with the aspect types it knows, as before, and stores that result.
  • A second read per entry, CatalogClient.lookupEntryAspectTypes, reads the FULL view's keys alone and names each aspect type on the entry once, with project numbers resolved. A FULL read gives data only to the aspects the entry's type requires, so pull cannot store from it.
  • Pull warns once per aspect type the first read did not return, it does not know and the scope does not list as a system aspect:
    Warning: pull left out aspect type 'X' on N entries; list it under snapshot.aspects in catalog.yaml to pull it.
  • CatalogSource.systemAspects: aspects the service writes beyond the required ones, left out without a warning. BigQuery dataset: bigquery-policy, data-profile, data-quality-scorecard and usage.
  • One helper reads an aspect key's type for _fixEntry and the new method.

Tests

  • New pull_left_out scenario: two entries with an unlisted aspect and one with overview, pulled with a catalog.yaml that lists none. Nothing extra is stored, and pull warns once per type with the entry count.
  • pull_bq: a table with usage and bigquery-policy pulls without them and with no warning.
  • The scenario harness records console.warn and checks assert.warnings.
  • The catalog mock matches Dataplex: lookupEntry returns only the requested aspects, and lookupEntryAspectTypes every aspect type on the entry.
  • lookupEntryAspectTypes reads the FULL view and names each type once, with a project ID, looking up each type's project once.
  • Mutation checks: dropping usage fails pull_bq; removing the warning or the second read fails pull_left_out.
  • tsc clean; scenarios 24, layouts 2, gcp 13, semantic 2076, tool 421, all pass.

@merr1988
merr1988 requested a review from feixufeixu October 23, 2026 23:42

@feixufeixu feixufeixu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with two required fixes. In production a FULL read returns data only for the aspects an entry's type requires. So this pull stops storing the overview that #560 lists, and the new warning never prints. My sync.ts comment gives the fix, and my mocks.ts comment covers why the tests still pass. The other two comments are optional.

Comment thread toolbox/mdcode/src/libts/sync.ts Outdated
Comment thread toolbox/mdcode/tests/libts/mocks.ts Outdated
Comment thread toolbox/mdcode/src/libts/sources/bq-dataset.ts Outdated
Comment thread toolbox/mdcode/tests/libts/gcp/dataplex.test.ts Outdated
@merr1988
merr1988 force-pushed the feat/pull/warn-left-out-aspects branch from a07e607 to eea3915 Compare October 26, 2026 17:59
Comment thread toolbox/mdcode/src/libts/gcp/dataplex.ts Outdated
Pull fetched each entry with only the aspect types catalog.yaml lists and
its entry type requires, so it could not tell an author what it left out.
A catalog.yaml written before init listed any aspects pulls a knowledge
base document with an empty body and says nothing.

Pull now fetches each entry with every aspect (the FULL view), keeps the
aspects of the types it knows, and warns once for each aspect type it
left out, naming the number of entries and how to pull it. What it stores
is unchanged.

Each scope names the aspects the service writes beyond the required ones
(CatalogSource.systemAspects), which pull leaves out without a warning.
For a BigQuery dataset these are bigquery-policy and usage, the aspects
BigQuery writes on a table besides the ones its type requires.

lookupEntry takes 'FULL' to return every aspect.

Bug: b/576454554
A FULL read gives data only to the aspects an entry's type requires, and
_fixEntry drops the rest, so pulling through it stored no listed aspect
beyond the required ones and never warned.

Pull now reads each entry with the aspect types it knows, as before, and
stores that result. A new CatalogClient.lookupEntryAspectTypes reads the
FULL view's keys alone and names each aspect type on the entry once, with
project numbers resolved. Pull warns about a type the first read did not
return, it does not know and the scope does not list as a system aspect.
lookupEntry is back to its earlier form.

The BigQuery system aspects also name data-profile and
data-quality-scorecard, which Dataplex data scans write.

Bug: b/576454554
…ects

lookupEntryAspectTypes resolved the type of every aspect key on its own,
and crm caches only the projects it can read. An entry with several
field-path aspects of one type in an unreadable project made one failing
lookup per key. It now resolves each distinct type once.

Bug: b/576454554
@merr1988
merr1988 force-pushed the feat/pull/warn-left-out-aspects branch from eea3915 to 55d6a0a Compare October 26, 2026 20:16
@merr1988
merr1988 changed the base branch from feat/init/default-aspects to main October 26, 2026 20:16
@merr1988
merr1988 merged commit 0d485bc into main Oct 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants