job_runs: add lifecycle.triggers.on_file_change by radakam · Pull Request #6309 · databricks/cli

radakam · 2026-08-18T12:30:32Z

Changes

Add lifecycle.triggers.on_file_change for resources.job_runs (direct engine).

resources:
  job_runs:
    migrate:
      job_id: ${resources.jobs.migrate.id}
      lifecycle:
        triggers:
          - on_file_change: migrations/*.sql

Resolve the path or glob relative to the defining YAML file, require the result stay under the sync root, and fingerprint each match by content hash, so an mtime-only touch does not re-fire.
Re-fire when a hash changes or when matches appear or disappear. Removing the trigger leaves the existing run alone: the plan reports it unchanged and no API call is made. A removed file still recreates (the planner sees on_file_change['path'], not the map itself).
Skip whatever sync skips (.gitignore and sync.exclude). Directories matched next to real files (typical for migrations/*) are skipped silently.
Store the fingerprint as a flat lifecycle.triggers.on_file_change path-to-hash map, nested by value so a plan diff lands on the file. Missing matches use the same sync-root-relative key as real files, so appear/disappear is a hash change rather than a key swap.
Reject absolute patterns (so filepath.Join cannot rebase /tmp/seed.txt under the bundle root), patterns outside the sync root, empty patterns, ** (filepath.Glob treats it as a single *), and combining a trigger with prevent_destroy. Warn when a pattern hashes nothing (no match, directory-only, or only sync-excluded files); the placeholder still re-arms once a matching file appears.

Why

Users need to re-run a job when local inputs change (SQL migrations, notebooks, configs) without re-firing on every deploy. Content hashes keep the trigger stable under sync noise while still catching real edits.

A pattern that hashes nothing used to look like an unarmed trigger. Recording a placeholder under a stable key means those cases re-arm later, so they are warnings rather than deploy failures.

Tests

Unit: content hashing, absolute-pattern rejection, and sync-root-relative missing-file keys in resolve_job_run_file_triggers; trigger validation for an empty pattern, both keys set, neither key set, on_bundle_deploy: false, and the prevent_destroy conflict (named by the trigger that is set); OverrideChangeDesc skip on a cleared trigger vs recreate on a cleared on_file_change['path'] child.
Acceptance:
- on_file_change — first deploy fires; unchanged redeploy and touch plan nothing; an edit recreates (including READPLAN); deleting the file warns and re-fires; restoring it re-fires again; removing the trigger is unchanged with no run-now.
- on_file_change_glob — migrations/* plus mysubdir/*.txt; matches appearing and disappearing re-fire; files below the glob and gitignored files do not; ** is rejected; READPLAN.
- on_file_change_errors — pattern outside the sync root (error); directory-only, directory plus sync-excluded, only sync-excluded, and no-match (warnings); unsupported ** and an absolute pattern (errors).
- on_file_change_include / on_file_change_sync_root / on_file_change_include_sync_root — YAML-relative globs, including when sync.paths reaches above the bundle root.

eng-dev-ecosystem-bot · 2026-08-18T13:54:48Z

Integration test report

Commit: b0437a3

Run: 32836050668

	Env	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
💚	aws linux	1	4	276	1186	5:30
💚	aws windows	1	4	278	1184	6:09
💚	azure linux	1	4	275	1186	5:52
💚	azure windows	1	4	277	1184	5:23
💚	gcp linux	1	4	276	1186	6:26
💚	gcp windows	1	4	278	1184	6:37

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S

Top 9 slowest tests (at least 2 minutes):

duration	env	testname
4:15	gcp windows	TestAccept
3:16	aws windows	TestAccept
3:16	azure windows	TestAccept
2:47	aws windows	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:32	azure linux	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:17	gcp windows	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:11	gcp linux	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:06	aws linux	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct
2:01	azure windows	TestAccept/bundle/resources/job_runs/failed_run/DATABRICKS_BUNDLE_ENGINE=direct

github-actions · 2026-08-19T08:05:38Z

Approval status: pending

`/acceptance/bundle/` - needs approval

55 files changed
Suggested: @janniklasrose
Also eligible: @denik, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

`/bundle/` - needs approval

16 files changed
Suggested: @janniklasrose
Also eligible: @denik, @pietern, @andrewnester, @shreyas-goenka, @lennartkats-db, @anton-107

General files (require maintainer)

Files: .nextchanges/bundles/job-runs-on-file-change.md
Based on git history:

@janniklasrose -- recent work in bundle/config/mutator/, bundle/schema/, bundle/internal/schema/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.

See OWNERS for ownership rules.}

denik · 2026-08-24T09:17:57Z

+      "seed.txt": ""
+    }
+  },
+  "lifecycle.triggers.on_file_change['seed.txt']": {


Non blocking but still a bit weird that have both "lifecycle.triggers.on_file_change['seed.txt']" and "lifecycle.triggers.on_file_change" here.

denik · 2026-08-24T14:05:42Z

-		// lifecycle.triggers, or the leaf.
-		if change.New == nil || change.New == "" {
+	pathString := path.String()
+	if isJobRunTriggerPath(pathString) {


can we use HasPrefix + Path objects instead of string matching? String matching will miss child and parent paths.

var TriggersPath = MustParsePath("lifecycle.triggers") ... if path.HasPrefix(TriggersPath) {

done, thanks

I reverted this -- it seems that prefix matching is too broad and would skip on_file_change['seed.txt'], while removed files need to be recreated.

Re-fire a run when matched file contents change, using stable hash fingerprints so mtime-only updates (touch) do not recreate.

Satisfy the linter on PrepareState literals after adding OnFileChange, and refresh out.fields.txt so validate-generated stays green.

Reject patterns outside the sync root and directory-only matches so the trigger cannot fingerprint escaped paths or silently disarm.

Drop size/mtime fingerprints and the resolve fast path so content identity alone drives recreate, and touch no longer needs a planner exception.

The modernize linter rejects the manual m[k]=v copy loop, failing CI lint.

Honor .gitignore and sync.exclude when hashing matches. Cover mysubdir globs, READPLAN, and keep LF on Windows so restored files hash cleanly.

The glob test edits migrations/ignored.txt, but the test .gitignore excluded it from the commit so CI clones would miss the file.

Plan serialization is already covered by on_file_change; the glob cases only assert the hash map.

Cover deploying glob hashes from a saved plan, not only from a live re-plan.

Keep the gitignore fixture as dot_gitignore so it does not affect the repo while developing, drop unit tests that duplicate acceptance coverage, and allocate trigger state once when a trigger is armed.

The previous simplification omitted OnBundleDeploy from the struct literal, which the linter rejects.

Always persist the same lifecycle.triggers shape so trigger fingerprints compare at the same paths. Wrap on_file_change hashes in files so dropping the trigger updates instead of recreating.

Nest lifecycle by value so structdiff descends to the leaf that changed instead of reporting the whole subtree, and let RemapState supply the empty shape. GetRun never returns the fingerprints, so the remote type no longer advertises a lifecycle it cannot fill.

Drop the files wrapper so plan diffs land on lifecycle.triggers.on_file_change, matching the config path without an extra state-only level.

Flattening on_file_change to a map made an unarmed trigger drop out of the serialized state, so the three plans that print an unarmed job_run now report an empty triggers object.

Directory-only and sync-excluded matches silently disarmed the trigger. Treat them as errors, warn that ** is not recursive, and cover both in acc.

The goldens were captured against a one-line-taller databricks.yml, so CI failed after the source settled on line 24.

Clearing a trigger is now a skip, so the plan reports the run as unchanged and the deploy makes no API call.

Patterns must still land under the sync root. Reject **, warn when a glob matches directories, and skip only when the trigger itself is cleared.

filepath.Join rebases an absolute glob under the bundle root, so reject those up front. Key missing-file placeholders to the sync root, and skip a cleared trigger from path.String() instead of prefix matching.

Every "nothing to hash" case re-arms once a matching file appears, now that the placeholder is always recorded, so none of them justify failing the deploy. Directories matched alongside real files are skipped silently: a glob like migrations/* routinely matches a subdirectory, and warning about it on every command was noise the user cannot act on. Also drop the unreachable "lifecycle.triggers" branch in OverrideChangeDesc (structdiff descends into the value struct and never reports a change at that path) and document the ** and sync-skipping rules in the schema.

Windows CI converted /tmp/seed.txt to %TEMP%/seed.txt, so the golden expecting the POSIX path failed. MSYS_NO_PATHCONV keeps the argument intact.

MSYS_NO_PATHCONV also stops Git Bash converting the path of a shebang helper, so update_file.py could not be found on Windows. Keep the leading-slash pattern out of argv entirely instead.

## Changes Drop `job_run` unit tests that duplicate existing acceptance coverage, and add `destroy_unfinished_run` so cancel-before-delete is asserted through the same path a user destroy takes. Kept the wait/GetRun stubs that acceptance cannot reach without faking the Jobs API. ## Why Split from review on databricks#6309: if a unit test is already covered by acc, keep the acc test. ## Tests - New acc: `acceptance/bundle/resources/job_runs/destroy_unfinished_run` (faulted wait, then destroy cancel + delete order)

radakam marked this pull request as ready for review August 19, 2026 08:04

Base automatically changed from deco-job-runs-lifecycle-on-bundle-deploy to main August 19, 2026 09:41

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from b8c20f6 to 8744e5e Compare August 19, 2026 09:47

radakam requested a review from denik August 19, 2026 11:06

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from 8744e5e to 47994d8 Compare August 19, 2026 11:39

denik reviewed Aug 19, 2026

View reviewed changes

Comment thread acceptance/bundle/resources/job_runs/on_file_change/output.txt

radakam requested a review from denik August 20, 2026 08:12

denik reviewed Aug 20, 2026

View reviewed changes

Comment thread acceptance/bundle/resources/job_runs/on_file_change_glob/test.toml

radakam requested a review from denik August 20, 2026 09:27

denik reviewed Aug 20, 2026

View reviewed changes

radakam requested a review from denik August 20, 2026 14:10

denik reviewed Aug 21, 2026

View reviewed changes

Comment thread bundle/direct/dresources/job_run.go Outdated

denik reviewed Aug 21, 2026

View reviewed changes

Comment thread bundle/direct/dresources/job_run.go

radakam requested a review from denik August 24, 2026 08:50

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from 0756969 to 4e7b960 Compare August 24, 2026 13:24

denik reviewed Aug 24, 2026

View reviewed changes

radakam mentioned this pull request Aug 24, 2026

Job runs drop unit tests covered by acc #6370

Merged

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from 17857f9 to 45c4ccd Compare August 25, 2026 08:12

radakam changed the base branch from main to job-runs-drop-unit-tests-covered-by-acc August 25, 2026 08:13

radakam requested a review from denik August 25, 2026 08:32

Base automatically changed from job-runs-drop-unit-tests-covered-by-acc to main August 25, 2026 10:05

radakam added 6 commits August 25, 2026 10:09

job_runs: add lifecycle.triggers.on_file_change

459d58e

Re-fire a run when matched file contents change, using stable hash fingerprints so mtime-only updates (touch) do not recreate.

job_runs: fix exhaustruct and regenerate refschema for on_file_change

9306589

Satisfy the linter on PrepareState literals after adding OnFileChange, and refresh out.fields.txt so validate-generated stays green.

job_runs: tighten on_file_change path and directory checks

c7cf616

Reject patterns outside the sync root and directory-only matches so the trigger cannot fingerprint escaped paths or silently disarm.

job_runs: store on_file_change as path-to-hash only

6e63c23

Drop size/mtime fingerprints and the resolve fast path so content identity alone drives recreate, and touch no longer needs a planner exception.

job_runs: use maps.Copy in on_file_change hash merge

88f08fd

The modernize linter rejects the manual m[k]=v copy loop, failing CI lint.

job_runs: drop filepath.Glob jargon from on_file_change schema docs

8977dcd

radakam added 20 commits August 25, 2026 10:09

acc: refresh on_file_change goldens for resource-key progress lines

2db8ec5

acc: cover on_file_change globs, errors, and appear/disappear

6fd75c8

acc: expand on_file_change glob coverage and skip ignored files

2806e12

Honor .gitignore and sync.exclude when hashing matches. Cover mysubdir globs, READPLAN, and keep LF on Windows so restored files hash cleanly.

acc: force-add gitignored on_file_change fixture

3e19c16

The glob test edits migrations/ignored.txt, but the test .gitignore excluded it from the commit so CI clones would miss the file.

acc: drop READPLAN from the on_file_change glob test

7e9e1a0

Plan serialization is already covered by on_file_change; the glob cases only assert the hash map.

acc: restore READPLAN on the on_file_change glob test

6bdcffc

Cover deploying glob hashes from a saved plan, not only from a live re-plan.

job_runs: address on_file_change review comments

5ac0e4b

Keep the gitignore fixture as dot_gitignore so it does not affect the repo while developing, drop unit tests that duplicate acceptance coverage, and allocate trigger state once when a trigger is armed.

job_runs: list every trigger field to satisfy exhaustruct

0a62f66

The previous simplification omitted OnBundleDeploy from the struct literal, which the linter rejects.

job_runs: keep empty nested lifecycle on state and remote

5c3bed3

Always persist the same lifecycle.triggers shape so trigger fingerprints compare at the same paths. Wrap on_file_change hashes in files so dropping the trigger updates instead of recreating.

job_runs: store on_file_change hashes on the trigger itself

07e6c79

Drop the files wrapper so plan diffs land on lifecycle.triggers.on_file_change, matching the config path without an extra state-only level.

acc: refresh job_runs goldens for the omitted empty trigger

580aaea

Flattening on_file_change to a map made an unarmed trigger drop out of the serialized state, so the three plans that print an unarmed job_run now report an empty triggers object.

job_runs: error when on_file_change hashes nothing

3718f1c

Directory-only and sync-excluded matches silently disarmed the trigger. Treat them as errors, warn that ** is not recursive, and cover both in acc.

acc: fix on_file_change_errors diagnostic line numbers

db83a46

The goldens were captured against a one-line-taller databricks.yml, so CI failed after the source settled on line 24.

acc: removing on_file_change is unchanged, not an update

a9f1c7d

Clearing a trigger is now a skip, so the plan reports the run as unchanged and the deploy makes no API call.

job_runs: resolve on_file_change relative to the defining YAML file

f020446

Patterns must still land under the sync root. Reject **, warn when a glob matches directories, and skip only when the trigger itself is cleared.

job_runs: reject absolute on_file_change patterns

8501706

filepath.Join rebases an absolute glob under the bundle root, so reject those up front. Key missing-file placeholders to the sync root, and skip a cleared trigger from path.String() instead of prefix matching.

acc: stop Git Bash rewriting the on_file_change absolute pattern

c433182

Windows CI converted /tmp/seed.txt to %TEMP%/seed.txt, so the golden expecting the POSIX path failed. MSYS_NO_PATHCONV keeps the argument intact.

acc: read the absolute on_file_change pattern from a committed file

b0437a3

MSYS_NO_PATHCONV also stops Git Bash converting the path of a shebang helper, so update_file.py could not be found on Windows. Keep the leading-slash pattern out of argv entirely instead.

radakam force-pushed the deco-job-runs-lifecycle-on-file-change branch from d65e640 to b0437a3 Compare August 25, 2026 10:12

Uh oh!

Conversation

radakam commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

github-actions Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/acceptance/bundle/ - needs approval

/bundle/ - needs approval

General files (require maintainer)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

denik Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

denik Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

radakam Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

radakam Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

radakam commented Aug 18, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Aug 18, 2026 •

edited

Loading

github-actions Bot commented Aug 19, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/bundle/` - needs approval