fix: unblock job1_prod (DQX quarantine schema) + rollback playbook and CI secret hardening

[andre-salvati]

What?

Three related pieces of CI/CD and production hardening:

1. Fixes a live prod outage — job1_prod has failed since 2026-07-24 09:05 UTC. raw.order_quarantine is no longer schema-pinned.
2. Adds a rollback playbook to specs/workflow.md.
3. Narrows the CI secret's blast radius and pins every GitHub action by commit SHA.

Why?

The outage: DQX 0.15.0 (#51) added rule_fingerprint, rule_set_fingerprint and skipped to the _errors/_warnings structs it appends to quarantined rows. The write used overwriteSchema=false, so the first prod run after that merge hard-failed with DELTA_METADATA_MISMATCH and skipped every downstream task.

The rollback gap: merging to main deploys to prod with no approval gate, and nothing documented how to get back. Flagged independently by a CI/CD article review and by the official Databricks best-practices doc.

The secret: DATABRICKS_CLIENT_SECRET sat in the workflow-level env block, visible to every step — including dependency installation, which is exactly where third-party code runs.

How?

The quarantine struct shape is owned by the DQX library version, not by the source data contract, so pinning it turns every dependency bump into a prod outage — a false positive, not the drift signal the guard exists to raise. The guard isn't lost: the business columns reaching quarantine come from the same DataFrame written to raw.order a few lines later, which keeps overwriteSchema=false.

The rollback section is organised around one rule — a redeploy reverts code, never data. Code rollback works because nothing deployed is stored apart from the checkout (jobs.yml, the SDP pipeline and the dashboard are all regenerated), so git history is the artifact store.

Actions are pinned by SHA because tags are mutable. databricks/setup-cli resolves the CLI version from a VERSION file at the pinned ref — verified it still installs 1.9.0, not assumed.

The architecture spec records why OIDC would be the real fix and why this repo can't use it: the trust anchor is an account-level API, and Free Edition has none. Neither mitigation shortens the credential's lifetime; only OIDC does.

Validation?

Unit tests pass (16); ruff lint + format clean. The staging integration test in this PR's CI exercises extract_source2 end-to-end, which is the actual regression path. The onpush.yml changes are validated by this very run — an auth or SHA mistake shows up as a red CI here rather than on main.

Impact in prod

• Schema change — raw.order_quarantine gains three nested fields inside _errors/_warnings (DQX framework metadata; no business column changes).

Change	Risk	Remediation
Add field (nested, framework metadata)	🟢 The write now carries overwriteSchema=true, so it self-heals instead of failing	None required

No consumer reads raw.order_quarantine — not the dashboard, not the SDP pipeline. Business columns are untouched.

Chosen strategy: leave as-is — the next job1_prod run rewrites the table schema and recovers on its own. This PR ends an outage rather than risking one.