Skip to content

Wait for grants to provision as part of depends on - #7142

Open
andrewnester wants to merge 1 commit into
mainfrom
fix/4573
Open

Wait for grants to provision as part of depends on#7142
andrewnester wants to merge 1 commit into
mainfrom
fix/4573

Conversation

@andrewnester

@andrewnester andrewnester commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Changes

Wait for grants to provision as part of depends on

Why

Fixes #4573

Tests

Added acceptance and unit tests

@andrewnester
andrewnester requested review from a team as code owners October 21, 2026 10:28
@github-actions github-actions Bot added the DABs DABs related issues label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Commit: 165f0c68d | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0

test branch main (92df5a7) latest (v1.21.0)
TestAccept/bundle/resources/grants/schemas/pipeline_run_as/DMS= ✅ ❌ ❌
TestAccept/bundle/resources/grants/schemas/pipeline_run_as/DMS=true ✅ ❌ ❌
TestAccept/bundle/resources/grants/schemas/pipeline_run_as/DMS=: branch ✅ | main (92df5a7) ❌ | latest (v1.21.0) ❌

main (92df5a7):

acceptance_test.go:1307: Diff:
        --- bundle/resources/grants/schemas/pipeline_run_as/output.txt
        +++ /tmp/TestAcceptbundleresourcesgrantsschemaspipeline_run_asDMS=2073389115/002/output.txt
        @@ -16,20 +16,12 @@
             {
               "node": "resources.schemas.src",
               "label": "${resources.schemas.src.name}"
        -    },
        -    {
        -      "node": "resources.schemas.src.grants",
        -      "label": "${resources.schemas.src.catalog_name}"
             }
           ],
           "resources.schemas.src": [
             {
               "node": "resources.catalogs.pipeline_catalog",
               "label": "${resources.catalogs.pipeline_catalog.name}"
        -    },
        -    {
        -      "node": "resources.catalogs.pipeline_catalog.grants",
        -      "label": "${resources.catalogs.pipeline_catalog.name}"
             }
           ],
           "resources.schemas.src.grants": [
        
FAIL

latest (v1.21.0):

acceptance_test.go:1307: Diff:
        --- bundle/resources/grants/schemas/pipeline_run_as/output.txt
        +++ /tmp/TestAcceptbundleresourcesgrantsschemaspipeline_run_asDMS=1219571006/002/output.txt
        @@ -16,20 +16,12 @@
             {
               "node": "resources.schemas.src",
               "label": "${resources.schemas.src.name}"
        -    },
        -    {
        -      "node": "resources.schemas.src.grants",
        -      "label": "${resources.schemas.src.catalog_name}"
             }
           ],
           "resources.schemas.src": [
             {
               "node": "resources.catalogs.pipeline_catalog",
               "label": "${resources.catalogs.pipeline_catalog.name}"
        -    },
        -    {
        -      "node": "resources.catalogs.pipeline_catalog.grants",
        -      "label": "${resources.catalogs.pipeline_catalog.name}"
             }
           ],
           "resources.schemas.src.grants": [
        @@ -52,8 +38,8 @@
         
         >>> print_requests.py --summary //unity-catalog/catalogs //unity-catalog/schemas //unity-catalog/permissions //pipelines
         POST /api/2.1/unity-catalog/catalogs
        -PATCH /api/2.1/unity-catalog/permissions/catalog/catalog_grants_pipeline_[UNIQUE_NAME]
         POST /api/2.1/unity-catalog/schemas
        +PATCH /api/2.1/unity-catalog/permissions/catalog/catalog_grants_pipeline_[UNIQUE_NAME]
         PATCH /api/2.1/unity-catalog/permissions/schema/catalog_grants_pipeline_[UNIQUE_NAME].src
         POST /api/2.0/pipelines
         
        
FAIL
TestAccept/bundle/resources/grants/schemas/pipeline_run_as/DMS=true: branch ✅ | main (92df5a7) ❌ | latest (v1.21.0) ❌

main (92df5a7):

acceptance_test.go:1307: Diff:
        --- bundle/resources/grants/schemas/pipeline_run_as/output.txt
        +++ /tmp/TestAcceptbundleresourcesgrantsschemaspipeline_run_asDMS=t1877262019/002/output.txt
        @@ -16,20 +16,12 @@
             {
               "node": "resources.schemas.src",
               "label": "${resources.schemas.src.name}"
        -    },
        -    {
        -      "node": "resources.schemas.src.grants",
        -      "label": "${resources.schemas.src.catalog_name}"
             }
           ],
           "resources.schemas.src": [
             {
               "node": "resources.catalogs.pipeline_catalog",
               "label": "${resources.catalogs.pipeline_catalog.name}"
        -    },
        -    {
        -      "node": "resources.catalogs.pipeline_catalog.grants",
        -      "label": "${resources.catalogs.pipeline_catalog.name}"
             }
           ],
           "resources.schemas.src.grants": [
        @@ -44,35 +30,27 @@
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/schema-grants-pipeline-[UNIQUE_NAME]/default/files...
         Created catalogs.pipeline_catalog
         Created catalogs.pipeline_catalog.grants
        -Created pipelines.load
         Created schemas.src
         Created schemas.src.grants
        -Files: 3 uploaded, 0 deleted
        -Resources: 5 created, 0 changed, 0 deleted, 0 unchanged
        +Error: cannot create resources.pipelines.load: User does not have CREATE TABLE on Schema 'catalog_grants_pipeline_[UNIQUE_NAME].src'. (403 PERMISSION_DENIED)
         
        ->>> print_requests.py --summary //unity-catalog/catalogs //unity-catalog/schemas //unity-catalog/permissions //pipelines
        -POST /api/2.1/unity-catalog/catalogs
        -PATCH /api/2.1/unity-catalog/permissions/catalog/catalog_grants_pipeline_[UNIQUE_NAME]
        -POST /api/2.1/unity-catalog/schemas
        -PATCH /api/2.1/unity-catalog/permissions/schema/catalog_grants_pipeline_[UNIQUE_NAME].src
        -POST /api/2.0/pipelines
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/pipelines
        +HTTP Status: 403 Forbidden
        +API error_code: PERMISSION_DENIED
        +API message: User does not have CREATE TABLE on Schema 'catalog_grants_pipeline_[UNIQUE_NAME].src'.
         
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 5 unchanged
        +Files: 3 uploaded, 0 deleted
         
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.catalogs.pipeline_catalog
        -  delete resources.pipelines.load
           delete resources.schemas.src
         
         This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
           delete resources.schemas.src
         
        -This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -  delete resources.pipelines.load
        -
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/schema-grants-pipeline-[UNIQUE_NAME]/default
         
        -Destroy: 3 deleted
        +Destroy: 2 deleted
        +
        +Exit code: 1

latest (v1.21.0):

acceptance_test.go:1307: Diff:
        --- bundle/resources/grants/schemas/pipeline_run_as/output.txt
        +++ /tmp/TestAcceptbundleresourcesgrantsschemaspipeline_run_asDMS=t2820220868/002/output.txt
        @@ -16,20 +16,12 @@
             {
               "node": "resources.schemas.src",
               "label": "${resources.schemas.src.name}"
        -    },
        -    {
        -      "node": "resources.schemas.src.grants",
        -      "label": "${resources.schemas.src.catalog_name}"
             }
           ],
           "resources.schemas.src": [
             {
               "node": "resources.catalogs.pipeline_catalog",
               "label": "${resources.catalogs.pipeline_catalog.name}"
        -    },
        -    {
        -      "node": "resources.catalogs.pipeline_catalog.grants",
        -      "label": "${resources.catalogs.pipeline_catalog.name}"
             }
           ],
           "resources.schemas.src.grants": [
        @@ -44,35 +30,27 @@
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/schema-grants-pipeline-[UNIQUE_NAME]/default/files...
         Created catalogs.pipeline_catalog
         Created catalogs.pipeline_catalog.grants
        -Created pipelines.load
         Created schemas.src
        +Error: cannot create resources.pipelines.load: User does not have CREATE TABLE on Schema 'catalog_grants_pipeline_[UNIQUE_NAME].src'. (403 PERMISSION_DENIED)
        +
        +Endpoint: POST [DATABRICKS_URL]/api/2.0/pipelines
        +HTTP Status: 403 Forbidden
        +API error_code: PERMISSION_DENIED
        +API message: User does not have CREATE TABLE on Schema 'catalog_grants_pipeline_[UNIQUE_NAME].src'.
        +
         Created schemas.src.grants
         Files: 3 uploaded, 0 deleted
        -Resources: 5 created, 0 changed, 0 deleted, 0 unchanged
         
        ->>> print_requests.py --summary //unity-catalog/catalogs //unity-catalog/schemas //unity-catalog/permissions //pipelines
        -POST /api/2.1/unity-catalog/catalogs
        -PATCH /api/2.1/unity-catalog/permissions/catalog/catalog_grants_pipeline_[UNIQUE_NAME]
        -POST /api/2.1/unity-catalog/schemas
        -PATCH /api/2.1/unity-catalog/permissions/schema/catalog_grants_pipeline_[UNIQUE_NAME].src
        -POST /api/2.0/pipelines
        -
        ->>> [CLI] bundle plan
        -Plan: 0 to add, 0 to change, 0 to delete, 5 unchanged
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.catalogs.pipeline_catalog
        -  delete resources.pipelines.load
           delete resources.schemas.src
         
         This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
           delete resources.schemas.src
         
        -This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
        -Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion:
        -  delete resources.pipelines.load
        -
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/schema-grants-pipeline-[UNIQUE_NAME]/default
         
        -Destroy: 3 deleted
        +Destroy: 2 deleted
        +
        +Exit code: 1

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 165f0c6

Run: 39584935572

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 281 16 4:09
✅​ aws-windows-latest-4core-16gb 283 14 7:49
✅​ azure linux-2core-8gb 280 16 4:26
✅​ azure-windows-latest-4core-16gb 282 14 8:01
✅​ gcp linux-2core-8gb 281 16 4:09
✅​ gcp-windows-latest-4core-16gb 283 14 7:08
Top 3 slowest tests (at least 2 minutes):
duration env testname
7:55 azure-windows-latest-4core-16gb TestAccept
7:42 aws-windows-latest-4core-16gb TestAccept
7:02 gcp-windows-latest-4core-16gb TestAccept

denik
denik reviewed Oct 21, 2026
View reviewed changes
Comment thread bundle/direct/bundle_plan.go
p.Plan[node] = &e
}

addGrantsDependencies(p)

@denik denik Oct 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I'm also adding lifecycle.depends_on #7120

Worth evaluating if it changes anything there.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@denik denik denik left review comments
@pietern pietern Awaiting requested review from pietern
@janniklasrose janniklasrose Awaiting requested review from janniklasrose

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Schema-Grants need to run before pipeline deployment

3 participants

@andrewnester @eng-dev-ecosystem-bot @denik