From 97a4a5cbbbab38e499ff278c8dfe5c6eacdab9e1 Mon Sep 17 00:00:00 2001 From: Tomos Corbin Date: Wed, 22 Jul 2026 22:51:26 +0100 Subject: [PATCH 1/3] docs: consolidate all documentation into a single README The README is now the only piece of documentation: - Fold docs/limitations.md into the README as a Limitations section, dropping the items already covered by Health, Widgets, and How it works. - Delete the vimdoc (doc/databricks-notebook.txt), the ADRs, and the maintainer smoke-test/probe documents; the full texts remain in git history. - Reword the smoke-kit README and molten_probe.lua so nothing links to the deleted files. --- README.md | 65 ++++- doc/databricks-notebook.txt | 226 ------------------ .../0001-notebook-scoped-sessions.md | 97 -------- .../0002-molten-backed-execution-service.md | 95 -------- docs/dev/molten-integration-probe.md | 120 ---------- docs/dev/smoke-test-runbook.md | 205 ---------------- docs/dev/smoke-test.md | 192 --------------- docs/limitations.md | 139 ----------- examples/smoke/README.md | 15 +- tests/manual/molten_probe.lua | 2 +- 10 files changed, 67 insertions(+), 1089 deletions(-) delete mode 100644 doc/databricks-notebook.txt delete mode 100644 docs/decisions/0001-notebook-scoped-sessions.md delete mode 100644 docs/decisions/0002-molten-backed-execution-service.md delete mode 100644 docs/dev/molten-integration-probe.md delete mode 100644 docs/dev/smoke-test-runbook.md delete mode 100644 docs/dev/smoke-test.md delete mode 100644 docs/limitations.md diff --git a/README.md b/README.md index a35375f..1036e77 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,7 @@ is rendered by [molten-nvim](https://github.com/benlubas/molten-nvim). - Local widget emulation from project-owned YAML. - Structured `:checkhealth databricks-notebook` diagnostics with remediation. -See [docs/limitations.md](docs/limitations.md) for what this deliberately does -not do in v1. +See [Limitations](#limitations) for what this deliberately does not do in v1. ## Requirements @@ -223,6 +222,64 @@ the installed Connect version), kernelspec location, and each open notebook's saved selection. It runs offline and never contacts your workspace; workspace reachability is exercised by `:DatabricksInit`. +## Limitations + +Deliberate v1 boundaries; several are tracked as post-v1 roadmap items. + +- **Execution is local-first.** Notebook code runs in a local IPython kernel; + Spark operations run remotely on your selected compute via Databricks + Connect. The filesystem, environment variables, and subprocesses are your + machine's, not a cluster driver's. `%pip` installs into the local managed + environment, not onto the cluster, and Databricks Runtime preinstalled + libraries are not present locally — install what you need into the managed + environment. Databricks Connect itself constrains which Spark features are + available. +- **`dbutils.fs` depends on your workspace tier.** `/Volumes/...` paths use + the Files API; everything else uses the legacy DBFS API. Workspaces without + a public DBFS root (Free Edition and other serverless-only tiers) reject + non-Volumes paths with `PermissionDenied` — exactly as they do in a + workspace notebook on the same tier. Use `/Volumes/...` paths there. +- **No secret redaction.** `dbutils.secrets.get` returns real secret values, + and unlike a workspace notebook there is no output-redaction layer locally: + printing a secret prints the secret. +- **Unsupported surface fails by name, not obscurely.** `%sql`, `%run`, + `dbutils.notebook.run`, `dbutils.jobs`, and `dbutils.library` raise a named + error. `.ipynb` notebooks are out of scope entirely — v1 supports the + Databricks `.py` source format only. +- **No run-all command.** The pinned Molten revision exposes no supported + per-evaluation completion event, so serial stop-on-failure semantics cannot + be implemented truthfully. Run cells individually; run-all returns when the + upstream capability exists. +- **One kernel per notebook.** Isolation is what makes interrupts targeted + and state predictable, but the first `:DatabricksInit` for a project pays + for environment creation, package installation, kernel startup, and Connect + session establishment — and several open notebooks means several kernel + processes. +- **One managed environment for all projects.** `venv_path` and + `databricks_connect_version` are global: every project shares one + virtualenv and one Connect constraint, and conflicting requirements fight + over it (the most recent preparation wins). The bundled kernel dialect + itself requires `databricks-connect>=15.1`, resolved jointly with your + configured constraint. +- **Molten is a runtime dependency with a verified surface.** The integration + is verified against Molten revision `bedea638`; health checks that Molten's + public commands and functions exist, not that they still behave as probed. + Pin Molten if you need stability. +- **The UI stalls while a kernel starts.** Molten handles cursor and scroll + events plus its readiness tick as synchronous RPCs into a single-threaded + Python host, so input turns sluggish while a kernel boots and builds its + Connect session; it recovers the moment the kernel attaches. Keeping the + cursor still during the attach stage avoids most of it, and raising + `vim.g.molten_tick_rate` reduces the stall frequency. +- **Reload closes the session.** Unloading, wiping, or reloading a notebook + buffer (including `:e!`) shuts its execution session down. Run + `:DatabricksInit` again after a reload; your saved profile/compute + selection is reused. +- **Namespace bindings are conventional, not enforced.** `spark`, `dbutils`, + and `_databricks_runtime` are ordinary IPython namespace bindings injected + at kernel startup. Overwriting them in notebook code breaks the dialect for + that session. + ## Development ```sh @@ -231,9 +288,7 @@ make test-lua # plenary specs via headless nvim make test-py # pytest ``` -Architecture decisions live in [docs/decisions/](docs/decisions/). Maintainer -material — the manual smoke-test procedure and its latest results — lives in -[docs/dev/](docs/dev/). +CI runs the same checks on every pull request. ## License diff --git a/doc/databricks-notebook.txt b/doc/databricks-notebook.txt deleted file mode 100644 index 2395115..0000000 --- a/doc/databricks-notebook.txt +++ /dev/null @@ -1,226 +0,0 @@ -*databricks-notebook.txt* Edit and run Databricks source notebooks in Neovim - - *databricks-notebook* *databricks-notebook.nvim* - -Lua owns the notebook as an editable document; a bundled Python kernel dialect -owns it as an executable Databricks program. Inline output is rendered by -molten-nvim. - -============================================================================== -CONTENTS *databricks-notebook-contents* - - 1. Requirements ................................... |databricks-notebook-requirements| - 2. Setup .......................................... |databricks-notebook-setup| - 3. Notebook detection ............................. |databricks-notebook-detection| - 4. Commands ....................................... |databricks-notebook-commands| - 5. Mappings and textobjects ....................... |databricks-notebook-mappings| - 6. Projects and state ............................. |databricks-notebook-projects| - 7. Environment and sessions ....................... |databricks-notebook-sessions| - 8. Widgets ........................................ |databricks-notebook-widgets| - 9. Directives and dbutils ......................... |databricks-notebook-directives| - 10. Health ......................................... |databricks-notebook-health| - 11. Limitations .................................... |databricks-notebook-limitations| - -============================================================================== -1. REQUIREMENTS *databricks-notebook-requirements* - -- Neovim 0.11 or later. -- `uv` on $PATH (manages the kernel environment). -- Python 3.11+ discoverable by uv. -- molten-nvim, installed and initialised per its own documentation. The - integration is verified against Molten revision bedea638. -- A Databricks config file (default `~/.databrickscfg`) with at least one - profile; create one with `databricks auth login`. - -============================================================================== -2. SETUP *databricks-notebook-setup* - -Call setup once; every key is optional and validated. Defaults shown: >lua - - require("databricks-notebook").setup({ - venv_path = vim.fn.expand("~/.local/share/databricks-notebook.nvim/venv"), - databricks_connect_version = ">=15.1", - databricks_config_path = vim.fn.expand("~/.databrickscfg"), - keymaps = { - next_cell = "]n", - prev_cell = "[n", - run_cell = "dr", - }, - conceal_magic = true, - }) -< - *databricks-notebook-options* -venv_path Absolute path of the uv-managed kernel - environment. Shared across projects. -databricks_connect_version pip-style constraint installed into the managed - environment. -databricks_config_path Databricks config file profiles are read from. -keymaps Buffer-local notebook mappings; see - |databricks-notebook-mappings|. -conceal_magic Conceal `# MAGIC ` prefixes in notebook windows. - -setup() reconciles already-loaded buffers, so lazy loading on the python -filetype is safe. Repeated setup() replaces mappings and options in place and -preserves live notebook sessions. - -============================================================================== -3. NOTEBOOK DETECTION *databricks-notebook-detection* - -A buffer is an active notebook when its 'filetype' is python and its first -line is exactly: > - - # Databricks notebook source -< -The filetype stays python, so LSP and treesitter keep working. Empty or -header-prefix Python buffers are watched until they gain the header or diverge -from it. Removing the header deactivates the notebook and closes its -execution session. - -Cells are separated by `# COMMAND ----------` lines; `# MAGIC ` prefixes -carry directive and markdown content, matching the Databricks export format. - -============================================================================== -4. COMMANDS *databricks-notebook-commands* - -All commands are buffer-local and exist only in active notebooks. - - *:DatabricksInit* -:DatabricksInit[!] Resolve the project, prepare the environment - (venv, profile/compute selection, kernelspec), and - start this notebook's execution session. Progress is - reported in stages; ready is reported only after the - kernel is attached. With !, force profile/compute - reselection. Duplicate initialisation while one is in - flight is rejected. - - *:DatabricksRunCell* -:DatabricksRunCell Run the cell under the cursor in this notebook's - kernel. The buffer snapshot is captured once and - validated before submission; if the buffer changed - after capture the run is rejected instead of - executing stale text. Empty cells and separator lines - are named no-ops. - - *:DatabricksInterrupt* -:DatabricksInterrupt Interrupt this notebook's kernel only. Without a - session it is a harmless no-op. - - *:DatabricksCellAdd* -:DatabricksCellAdd {above|below} - Insert an empty cell relative to the current cell. - - *:DatabricksCellDelete* -:DatabricksCellDelete Delete the current cell. - - *:DatabricksCellMove* -:DatabricksCellMove {up|down} - Swap the current cell with its neighbour. Boundary - moves are rejected without changing the buffer. - -Cell edits are single undo steps with deterministic cursor placement. - - *:DatabricksWidgets* -:DatabricksWidgets Open the project's `.databricks/widgets.yaml`. For a - new file, an unsaved template keyed by the current - notebook is prepared; existing content is never - modified. See |databricks-notebook-widgets|. - - *:DatabricksConcealToggle* -:DatabricksConcealToggle - Toggle `# MAGIC` concealment in the current window. - -============================================================================== -5. MAPPINGS AND TEXTOBJECTS *databricks-notebook-mappings* *databricks-notebook-textobjects* - -Buffer-local, active notebooks only. Configurable via |databricks-notebook-options|. - - ]n Normal Jump to the next cell. - [n Normal Jump to the previous cell. - dr Normal Run the current cell. - ic Visual/Operator Inner cell (content only). - ac Visual/Operator Around cell (including separator). - -============================================================================== -6. PROJECTS AND STATE *databricks-notebook-projects* - -The project root is the nearest ancestor directory containing `.databricks/` -or `.git` (falling back to the notebook's own directory). The notebook key is -the project-relative path without `.py`, e.g. `etl/daily_report`. Identity is -derived from the file path, never from the current working directory. - -Profile/compute selections are saved per project under: > - - stdpath("state")/databricks-notebook.nvim/projects/ -< -The file is user-local, versioned, and written atomically; it never contains -credentials and never lives inside the repository. A malformed or -incompatible file is reported rather than silently replaced — recover with -|:DatabricksInit|! or by deleting the file. - -============================================================================== -7. ENVIRONMENT AND SESSIONS *databricks-notebook-sessions* - -Initialisation provisions a uv-managed virtualenv at `venv_path` containing -the bundled kernel dialect, ipykernel, and databricks-connect, then writes a -notebook-scoped Jupyter kernelspec (named `databricks--`) -carrying the project root, notebook key, profile, and optional cluster ID. - -Each notebook document owns exactly one execution session with its own -kernel, namespace, and widget context. Two notebooks never share state, and -|:DatabricksInterrupt| targets only its own notebook. - -Closing, unloading, or wiping the notebook buffer — including an |:edit!| -reload — shuts its session down. Run |:DatabricksInit| again after a reload. - -============================================================================== -8. WIDGETS *databricks-notebook-widgets* - -dbutils.widgets works locally. Defaults come from widget registrations in -notebook code; per-notebook overrides come from project-owned YAML: > - - # .databricks/widgets.yaml - "etl/daily_report": - run_date: "2026-07-17" - environment: "dev" -< -Top-level keys are notebook keys (see |databricks-notebook-projects|); widget values -must be quoted strings. Supported API: text, dropdown, combobox, multiselect, -get, getArgument, getAll, remove, removeAll. - -============================================================================== -9. DIRECTIVES AND DBUTILS *databricks-notebook-directives* - - %md Markdown source shown inline as text output. - %pip Runs in the local managed environment, not on the - cluster. - %sql, %run Named error; unsupported in v1. - dbutils.fs, .secrets Real workspace calls via the Databricks SDK. - dbutils.widgets Local YAML-backed emulation. - dbutils.notebook.exit Stops the cell cleanly and renders the exit value. - dbutils.notebook.run Named error; unsupported in v1. - dbutils.jobs, .library Named error; unsupported. - -Cell transformation preserves line counts, so kernel tracebacks point at the -correct buffer lines. Directive semantics belong to the Python kernel -dialect; the editor never reinterprets them. - -============================================================================== -10. HEALTH *databricks-notebook-health* - -Run `:checkhealth databricks-notebook`. It aggregates structured findings from the -execution service (backend availability, per-notebook session state) and the -environment service (uv, profiles, saved selection validity, managed -environment and installed Connect version, kernelspec write location), with a -section per open notebook project. Health runs offline and never contacts -your workspace; reachability is exercised by |:DatabricksInit|. - -============================================================================== -11. LIMITATIONS *databricks-notebook-limitations* - -Notebook code executes locally through Databricks Connect: Spark runs -remotely, but the local filesystem, subprocesses, and installed libraries are -not those of a cluster driver. There is no run-all command in v1, `.ipynb` -files are out of scope, and `%sql`/`%run` are unsupported. See -`docs/limitations.md` in the repository for the complete list. - -vim:tw=78:ts=8:noet:ft=help:norl: diff --git a/docs/decisions/0001-notebook-scoped-sessions.md b/docs/decisions/0001-notebook-scoped-sessions.md deleted file mode 100644 index 22a04db..0000000 --- a/docs/decisions/0001-notebook-scoped-sessions.md +++ /dev/null @@ -1,97 +0,0 @@ -# ADR 0001: Notebook-scoped execution sessions - -- **Status:** Accepted -- **Date:** 17 July 2026 -- **Scope:** v1 - -## Context - -The plugin has two related but different kinds of state: - -- reusable environment choices and artifacts for a Databricks project; and -- mutable Python/Jupyter runtime state for an open notebook. - -Treating both as project-scoped would let notebooks share a Python namespace and -kernel, but notebook identity would then have to change before every execution. -Widget configuration, `%run`-style relative paths, output attachment, interruption, -and cleanup would all depend on ambient buffer state. Treating all state as -notebook-scoped would isolate execution but duplicate project selection and -environment provisioning. - -The profile and compute selection are also user- and machine-specific. Persisting -them in the repository would create noise and risk sharing values that are not valid -for another developer. - -## Decision - -Project environment and notebook execution have separate scopes. - -The project owns: - -- its canonical root; -- the selected Databricks profile and optional compute ID; -- the managed Python environment; -- widget configuration at `.databricks/widgets.yaml`; and -- the inputs from which notebook kernelspecs are generated. - -Profile and compute selection are stored in versioned, atomic JSON under -`stdpath("state")/databricks-notebook.nvim/projects/.json`, keyed by canonical -project root. This state is never written to the project repository. - -Each canonical notebook document owns one live session in v1. The session contains: - -- its Python namespace and widget registry; -- its Molten kernel ID and buffer attachment; -- its inline outputs; -- its interrupt and cleanup lifecycle; and -- stable startup context: project root, project-relative notebook key, profile, - optional compute ID, and widget configuration path. - -The execution registry is keyed by canonical document URI and binds the session to -the active Neovim buffer. A profile, compute, project, or document change invalidates -the old session. Buffer unload or notebook deactivation closes that notebook's -session without affecting another notebook. - -Cell position is not durable identity. A captured execution uses the document URI, -cell index, changedtick, source text, and editor range as a snapshot-local locator. -Any incompatible edit causes the request to be rejected and rebuilt from the current -buffer. - -## Consequences - -### Positive - -- Notebook namespaces and widgets cannot leak into one another. -- Notebook identity is stable for the lifetime of the kernel extension. -- Range evaluation, inline output, interrupt, and cleanup have one clear owner. -- Project selection and environment provisioning remain reusable. -- User-specific profile and compute values do not pollute repositories. - -### Negative - -- Opening several executable notebooks consumes one local kernel process per - notebook. -- A newly opened notebook pays its own kernel startup cost. -- Cross-notebook interactive Python state is not shared in v1. - -These costs are accepted for v1 because semantic isolation is more valuable than -kernel reuse. Shared sessions can be considered later from measured demand. - -## Rejected alternatives - -### One session per project - -Rejected for v1 because it makes notebook context dynamic and interruption/cleanup -ambiguous. It can only be reconsidered with an explicit document-context protocol -and tested output ownership. - -### One global session - -Rejected because project selection, namespaces, dependencies, and output would leak -across unrelated workspaces. - -### Persist selection in `.databricks/nvim.json` - -Rejected because profile and compute IDs are user-local operational choices, not -shared project configuration. The project-owned widget YAML remains in the -repository because notebook parameter defaults are deliberately shareable. diff --git a/docs/decisions/0002-molten-backed-execution-service.md b/docs/decisions/0002-molten-backed-execution-service.md deleted file mode 100644 index ac19713..0000000 --- a/docs/decisions/0002-molten-backed-execution-service.md +++ /dev/null @@ -1,95 +0,0 @@ -# ADR 0002: Use a deep Molten-backed execution service - -- **Status:** Accepted -- **Date:** 17 July 2026 -- **Scope:** v1 - -## Context - -The original design proposed a generic backend adapter with `start`, `execute`, and -`interrupt` operations so Molten and a future Databricks driver/REST mechanism could -be exchanged. That interface hides the word “Molten” but not the important -differences: - -- Molten executes a range from the current Neovim buffer and owns inline output; -- a driver API would execute supplied source and require a separate output model; -- Molten's supported public API does not report a terminal outcome for an individual - evaluation; and -- session readiness, attachment, interrupt, and cleanup are notebook-specific - lifecycle concerns. - -Designing a generic interface before a second implementation exists would either -expose backend details to callers or claim completion semantics the v1 mechanism -cannot provide. - -The pinned capability probe is recorded in -[the Molten integration probe](../dev/molten-integration-probe.md). - -## Decision - -v1 uses a deep notebook execution service backed privately by Molten. - -`lua/databricks/execution/init.lua` owns: - -- opaque notebook session handles; -- starting, ready, failed, and closed states; -- the document-to-session registry; -- captured-buffer validation; -- run-current, targeted interrupt, replacement, and cleanup semantics; and -- stable findings for unavailable or invalid operations. - -`lua/databricks/execution/molten.lua` is the only production module that invokes -Molten. It may depend only on the public contract verified by the probe: - -- `:MoltenInit ` starts a deterministic kernelspec for the current - buffer; -- `User MoltenKernelReady` reports readiness and `data.kernel_id`; -- `MoltenRunningKernels(...)` reports buffer-local or global Molten kernel IDs; -- `MoltenEvaluateRange(kernel_id, start_line, end_line)` submits a range from the - current buffer to an attached explicit kernel; -- `:MoltenInterrupt ` interrupts an attached explicit kernel; -- `:MoltenDeinit` closes the current buffer's kernels; and -- Molten deinitializes attached kernels on `BufUnload`. - -`MoltenInitPost` is not readiness; it fires before the kernel is ready. Callers must -wait for the kernel-ready event. - -No mutable `adapter.use(...)` or backend registration API is introduced. A second -execution mechanism will be integrated by first comparing two real implementations -and then extracting only a proven common contract. - -## `run_all` gate - -`run_all` is application policy, not Molten policy. Correct behavior requires the -application to know when each submitted cell reaches a terminal state and whether it -succeeded, failed, was interrupted, intentionally exited, or lost its kernel. - -The tested Molten revision exposes those states internally for rendering but provides -no supported event, callback, or function that correlates them to an evaluation. -Busy/idle polling is also not a supported per-cell outcome contract. - -Therefore `run_all` is deferred in v1. The command remains unregistered. The plugin -will not enqueue the entire notebook, parse rendered output, or import Molten's -private Python state. The gate may be revisited when Molten exposes a documented -per-evaluation completion event with correlation and at least success/failure status. - -## Consequences - -### Positive - -- Application callers use Databricks notebook concepts, not Molten mechanics. -- Session identity and lifecycle are explicit. -- The implementation depends only on a small, tested upstream surface. -- Run-current and targeted interrupt can ship without inventing false completion - semantics. -- A future backend boundary will be based on evidence rather than speculation. - -### Negative - -- Molten is the only v1 execution mechanism. -- `run_all` is absent from v1 despite remaining a desired application feature. -- A future driver mechanism may require reshaping internals behind the execution - service. - -The execution service deliberately absorbs that future change so notebook parsing, -commands, and project environment policy do not have to. diff --git a/docs/dev/molten-integration-probe.md b/docs/dev/molten-integration-probe.md deleted file mode 100644 index 6fd4e52..0000000 --- a/docs/dev/molten-integration-probe.md +++ /dev/null @@ -1,120 +0,0 @@ -# Molten integration capability probe - -- **Date:** 17 July 2026 -- **Result:** Run-current and targeted interrupt are supportable; `run_all` is - deferred -- **Molten revision:** - [`bedea63819c618e007e7c40059fc6e72d598c8df`](https://github.com/benlubas/molten-nvim/commit/bedea63819c618e007e7c40059fc6e72d598c8df) - (3 July 2026) - -## Purpose - -This probe fixes the smallest public molten-nvim contract that databricks-notebook.nvim v1 may -use. It specifically avoids designing execution APIs from Molten's private Python -objects or rendered output. - -Molten was not installed in the active Neovim data directory on the probe machine. -The result therefore comes from an isolated checkout pinned to the revision above, -not from an implicit claim about a personal Neovim configuration. - -## Environment - -- Neovim 0.11.6 -- Python 3.12.3 -- pynvim 0.6.0 -- jupyter-client 8.9.1 -- ipykernel 7.3.0 -- two distinct local IPython kernelspecs: - `databricks-nvim-probe-a` and `databricks-nvim-probe-b` -- no image provider -- Molten tick rate reduced to 50 ms for the probe - -The manual script is -[`tests/manual/molten_probe.lua`](../tests/manual/molten_probe.lua). It creates two -named Neovim buffers, starts a distinct kernelspec in each, waits for readiness, -submits ranges to explicit kernel IDs, targets one interrupt, unloads one buffer, and -asserts that only that buffer's kernel is removed. - -Observed terminal result: - -```text -[Molten] Kernel 'databricks-nvim-probe-a' (id: databricks-nvim-probe-a) is ready. -[Molten] Kernel 'databricks-nvim-probe-b' (id: databricks-nvim-probe-b) is ready. -MOLTEN_PROBE_OK kernel_a=databricks-nvim-probe-a kernel_b=databricks-nvim-probe-b -``` - -## Capability results - -| Question | Result | Supported contract or limit | -|---|---|---| -| Can `MoltenInit` use a deterministic kernelspec without prompting? | Yes | `:MoltenInit ` starts that kernelspec for the current buffer. The command returning does not mean the kernel is ready. | -| How is the kernel ID observed? | Yes | `User MoltenKernelReady` supplies `data.kernel_id`. This is Molten's public, buffer-attached kernel ID, not the Jupyter manager's internal UUID. `MoltenRunningKernels(true/false)` lists buffer-local or all Molten IDs. | -| Can two buffers run independent notebook kernelspecs? | Yes | The probe started distinct kernelspecs in two buffers and observed distinct IDs. Unloading A removed A while B remained registered. | -| Can range evaluation target an explicit kernel? | Yes | `MoltenEvaluateRange(kernel_id, start_line, end_line[, start_col, end_col])` accepts an explicit attached Molten ID. Lines are 1-based and inclusive. | -| Must the target buffer be current? | Yes | Molten always reads the range and creates output extmarks in the current buffer. The explicit kernel must also be attached to that buffer. databricks-notebook.nvim must temporarily select the target buffer/window or reject execution when it cannot do so safely. | -| Is there a public per-cell started/completed event or callback? | No | The documented user events cover init, deinit, and first kernel readiness only. Range submission returns no execution handle or callback. | -| Can completion distinguish success, failure, interrupt, and kernel loss? | No | Molten tracks status and success internally for rendering, but exposes no supported per-evaluation status/outcome function or event. databricks-notebook.nvim must not read those private objects or infer status from output text/highlights. | -| Can interrupt target one kernel? | Yes, buffer-locally | `:MoltenInterrupt ` targets the explicit ID among kernels attached to the current buffer. It does not provide a terminal outcome notification. | -| What happens on buffer unload? | The attached kernel is closed | Molten registers `BufUnload`, deinitializes the buffer's attached kernels, and removes them from its registry. In the probe, the other buffer's independent kernel survived. | -| Which Molten surfaces may v1 use? | Narrow public surface only | `MoltenInit`, `MoltenDeinit`, `MoltenInterrupt`, `MoltenEvaluateRange`, `MoltenRunningKernels`, and the documented `MoltenKernelReady`/init/deinit user events. Private Python modules, output tables, timers, and internal Vim functions are excluded. | - -`MoltenInitPost` is intentionally not used as readiness evidence: upstream fires it -after constructing the Molten kernel wrapper, while actual Jupyter readiness arrives -later through `MoltenKernelReady`. - -## `run_all` decision - -`run_all` is deferred for v1. - -A correct serial queue must correlate a submission with its terminal state, wait -before submitting the next cell, stop on failure, distinguish intentional -`NotebookExit`, and respond deterministically to interrupt or kernel loss. The public -surface above cannot support that contract. Kernel busy/idle polling would still not -provide per-cell outcome and is not a documented Molten integration API. - -The plugin must therefore: - -- leave `:DatabricksRunAll` unregistered; -- support run-current and targeted interrupt only; -- avoid enqueueing all ranges into Molten; -- avoid parsing rendered output or importing Molten's private Python state; and -- revisit the feature only after Molten exposes a documented, correlated - per-evaluation completion/outcome event. - -## Reproduction outline - -The probe used isolated `/private/tmp` directories so it did not mutate the user's -Neovim installation. In outline: - -1. Download and extract the pinned Molten revision. -2. Create an isolated venv and install `pynvim`, `jupyter-client`, and `ipykernel`. -3. Install two local kernelspecs under an isolated prefix. -4. Set isolated XDG data/config/state/cache paths and `JUPYTER_PATH`. -5. Add the Molten checkout to `runtimepath`, point `g:python3_host_prog` at the venv, - and run `:UpdateRemotePlugins` once. -6. Run the script headlessly with: - -```text -MOLTEN_NVIM_PATH= \ -MOLTEN_PYTHON=/bin/python \ -MOLTEN_KERNEL_A=databricks-nvim-probe-a \ -MOLTEN_KERNEL_B=databricks-nvim-probe-b \ -nvim --headless --noplugin -u NONE -l tests/manual/molten_probe.lua -``` - -Starting local Jupyter kernels requires permission to create subprocesses and local -sockets. A restricted sandbox may need to allow those operations. - -## Implementation contract fixed by this probe - -The future private Molten integration may implement: - -- non-interactive start for the current target buffer; -- readiness reporting with the explicit Molten kernel ID; -- current-buffer range submission to that ID; -- buffer-local targeted interrupt; -- explicit deinit; and -- cleanup reconciliation after buffer unload. - -It may not implement or imply a per-cell execution handle or terminal outcome until -the upstream public surface grows to support one. diff --git a/docs/dev/smoke-test-runbook.md b/docs/dev/smoke-test-runbook.md deleted file mode 100644 index d911f42..0000000 --- a/docs/dev/smoke-test-runbook.md +++ /dev/null @@ -1,205 +0,0 @@ -# Manual smoke-test runbook - -The step-by-step run-through for the manual smoke suites. Work through the -phases in order and record the outcome in -[`smoke-test.md`](smoke-test.md), which holds the status table and the -environment record. - -Automated coverage (`make check`) fakes the Molten, process, SDK, and UI -boundaries; these phases are what exercise the real seams. - -## Prerequisites - -- Neovim 0.11+, `uv` on `$PATH`, and a `~/.databrickscfg` with at least one - profile (`databricks auth login`). See the [README](../README.md) for - install and configuration. -- **Phases 2–4 need your regular Neovim config with BOTH databricks-notebook.nvim and - molten-nvim installed** (README install snippet). The standalone launcher - (`nvim -u examples/smoke/init.lua …`) loads this checkout only — no molten — - so it can run **Phase 1 only**; under it `:DatabricksInit` fails immediately - with `molten_unavailable` by design. -- databricks-notebook.nvim installed and `setup()` run. The plugin loads on the - `python` filetype, so open a notebook first, then confirm - `:checkhealth databricks-notebook` renders. -- molten-nvim ready for the execution phases. molten is a Python remote - plugin: run `:UpdateRemotePlugins` once after install/update (lazy.nvim: - `build = ":UpdateRemotePlugins"`) and restart Neovim, or its commands do - not exist at all. The manifest generation itself needs a working Python - provider (`pynvim` installed for `g:python3_host_prog`), so configure the - provider first — a manifest generated without one is silently empty. Then, - **with a notebook open** (lazy loading means molten may not load until a - Python file does), confirm `:checkhealth molten` passes its required checks - (Neovim, Python, `pynvim`, `jupyter_client`) and `:checkhealth databricks-notebook` - reports no molten warning. Do not use `:echo exists('*MoltenRunningKernels')` - as the signal: remote-plugin functions are defined lazily, so it prints `0` - on a healthy install until the function is first called - (`:echo exists(':MoltenInit')` printing `2` is the eager check). The - optional image modules are not needed — these phases use text output only. -- **On a machine that has never run a full Jupyter server/console**, molten's - connection-file write fails with - `Could not initialize kernel ... No such file or directory: - .../jupyter/runtime/kernel-.json`: neither `jupyter_client` nor - `jupyter_core` creates `$(jupyter --data-dir)/runtime` on its own — that - normally happens as a side effect of `jupyter notebook`/`jupyter server` - startup, which this workflow never runs. Create it once: - `mkdir -p -m 700 "$( -c 'from jupyter_core.paths import - jupyter_data_dir; print(jupyter_data_dir())')/runtime"` (using the managed - venv's Python), or just run any classic Jupyter command once. - -## Fixtures - -- `examples/smoke/notebook_a.py` and `examples/smoke/notebook_b.py` — a paired - project (`examples/smoke/.databricks/` marks the root) for the local suite: - session isolation, per-notebook widget resolution, and a targeted interrupt. - Their runnable cells are plain Python, so they produce output in the local - kernel without any Spark work. -- `examples/demo_notebook.py` — a single notebook for directive/dbutils breadth - and the workspace suite (a Spark action, `dbutils.fs`, `%pip`). - -## Launch - -With your regular config (required from Phase 2 onward): - -``` -nvim examples/smoke/notebook_a.py examples/smoke/notebook_b.py -``` - -Phase 1 alone may instead use the dependency-free launcher: - -``` -nvim -u examples/smoke/init.lua examples/smoke/notebook_a.py examples/smoke/notebook_b.py -``` - -Note on keys: `dr` below is databricks-notebook.nvim's default buffer-local -run-cell mapping. `` is whatever your config sets; in a bare `-u` -launcher session no leader is configured, so the literal key is `\dr`. - -## Phase 1 — Editor layer (no execution) - -- [ ] **Detection** — `:echo &filetype` → `python`; `# MAGIC ` prefixes are - concealed; each `# COMMAND ----------` shows an overlaid rule. -- [ ] **Commands present** — `:Databricks` lists Init, RunCell, Interrupt, - CellAdd, CellDelete, CellMove, Widgets, ConcealToggle. -- [ ] **Navigation** — `]n` / `[n` move cell-to-cell; `]n` stops at the last - cell. -- [ ] **Textobjects** — `vic` selects cell content only; `vac` includes the - separator. -- [ ] **Conceal toggle** — `:DatabricksConcealToggle` hides/shows `# MAGIC `. -- [ ] **Cell add + undo** — cursor in a cell, `:DatabricksCellAdd below` → new - empty cell + separator; `u` reverts in one step. -- [ ] **Cell move** — `:DatabricksCellMove up`/`down` swaps with the neighbour; - on the first cell, `:DatabricksCellMove up` is rejected without changing - the buffer. -- [ ] **Cell delete** — `:DatabricksCellDelete` removes the current cell in one - undo step. -- [ ] **No-ops** — cursor on a `# COMMAND ----------` line, `:DatabricksRunCell` - → named separator no-op (not an error). Same on an empty cell. -- [ ] **Widgets file** — `:DatabricksWidgets` opens - `examples/smoke/.databricks/widgets.yaml`; existing content is untouched. -- [ ] **Health** — `:checkhealth databricks-notebook` renders; profiles ✅; kernelspec - ✅ (molten ✅ once installed). - -## Phase 2 — Sessions and execution (needs molten + a profile/compute) - -**Switch to your regular config now if Phase 1 used the launcher.** If -`:DatabricksInit` reports `molten_unavailable`, you are still in the -launcher (or the remote-plugin manifest is missing — see Prerequisites); it -fails fast, before any environment work. - -`:DatabricksInit` still contacts the workspace to list compute, but the -runnable cells below execute plain Python in the **local** kernel — no Spark -work. - -While the kernel starts, stay in the notebook buffer: Molten only observes -readiness for the current buffer's kernels. If readiness never arrives, -initialisation fails after `kernel_ready_timeout_ms` (default 3 minutes) -with `molten_ready_timeout` instead of hanging, and `:DatabricksInit!` -cancels a stuck initialisation at any point and starts over. - -- [ ] **Init A** — in `notebook_a`, `:DatabricksInit`. Staged progress - notifications appear: preparing environment → choose a profile (picker) → - validating the managed kernel environment (a first run installs packages, - minutes) → discovering compute → choose a compute target (picker) → - writing the kernelspec → execution ready. Readiness is reported only once - the kernel attaches. -- [ ] **Run-before-ready** — while it is still starting, `dr` → a - message that initialisation is still in progress (not "no session"). -- [ ] **Stuck-init escape** — while it is still starting, `:DatabricksInit` - (no bang) → "already in progress" naming `:DatabricksInit!` as the way - out; `:DatabricksInit!` → cancels and restarts (both pickers reappear). -- [ ] **Run local cell** — `dr` on the `value_a` cell → inline - `A initialised: alpha`. -- [ ] **`%md` renders** — run the markdown cell → the output shows the - markdown text itself, not ``. -- [ ] **Widget override** — run the widget cell → prints `override-from-yaml` - (the YAML override beats the `a-default` code default and proves the - `notebook_a` key resolves). -- [ ] **`%sql` error** — run the `%sql` cell → named unsupported-directive - error, not a traceback. -- [ ] **Init B, no pickers** — `:b notebook_b.py`, `:DatabricksInit` → no - profile/compute prompts (same project, reused selection) and no venv - rebuild. -- [ ] **Widget isolation** — run B's widget cell → prints `b-default`; A's - override does not leak. -- [ ] **Namespace isolation** — run B's last cell `print(value_a)` → - `NameError`; run A's last cell `print(value_b)` → `NameError`. -- [ ] **Targeted interrupt** — in A, run the `range(60)` loop (prints - 0, 1, 2…); switch to B and run `print("B is responsive")` → B answers - while A runs; back in A, `:DatabricksInterrupt` → `KeyboardInterrupt` in A - only; B untouched. `:DatabricksInterrupt` in idle B → harmless no-op. - -## Phase 3 — Lifecycle - -- [ ] **Header removal closes the session** — in `notebook_b`, delete line 1 → - commands/keymaps/conceal disappear (`:DatabricksRunCell` no longer exists - on that buffer). -- [ ] **Undo reactivates** — `u` restores the header → the notebook - reactivates; `:DatabricksInit` → session again, no pickers. -- [ ] **Reload closes the session** — in A, `:e!` → the session closes by - design; `:DatabricksInit` → reuses the selection, quick ready. -- [ ] **Unload cleanup** — `:bd! notebook_b.py` → no errors. -- [ ] **Repeated setup preserves the session** — - `:lua require("databricks-notebook").setup()` → A's live session survives; run a - cell without re-init. -- [ ] **Health with a live session** — `:checkhealth databricks-notebook` → the - execution section shows a ready session; the environment section shows - selection ✅, venv ✅ with versions, and the "compute reachability not - checked" line. -- [ ] **No stray kernels** — quit Neovim, then in the shell - `pgrep -fl ipykernel` → nothing. - -## Phase 4 — Breadth and workspace (`demo_notebook.py`, needs live compute) - -``` -nvim examples/demo_notebook.py -``` - -`:DatabricksInit`, then run cells. Read-only operations only; **never print a -secret value**. - -- [ ] **Local cell** — the `message` cell → `hello from the local kernel` (no - compute needed). -- [ ] **Spark action** — the `spark.range(100)…df.show(5)` cell → a result that - matches what you verify independently in the workspace. -- [ ] **Widgets default** — the `run_date` cell → `run_date = 2026-07-17`. -- [ ] **`%sql` / `%run` errors** — both raise their named errors. -- [ ] **`%pip`** — installs into the **local** managed venv, not the cluster. -- [ ] **`dbutils.fs.ls("/")`** — returns DBFS root entries on workspaces with - public DBFS root. On Free Edition / serverless-only workspaces the - platform disables public DBFS root, so the expected result there is a - clean `PermissionDenied: Public DBFS root is disabled` error surfaced at - the cell; listing a `/Volumes/...` path is the tier-independent - alternative. -- [ ] **Interrupt** — run the `range(60)` loop, `:DatabricksInterrupt` → stops. -- [ ] **`dbutils.notebook.exit`** — the exit cell stops cleanly (no error - state) and its output reads `Notebook exited: finished demo early`; - the next cell still runs. -- [ ] **`dbutils.notebook.run`** — raises the named unsupported error. -- [ ] **`dbutils.secrets.list`** (optional) — a safe list only, with no secret - value appearing in output or the record. - -## Recording - -Tick the boxes above as you go, then transfer the outcome — and the environment -versions — into [`smoke-test.md`](smoke-test.md). Report any step whose result -differs from the expectation with the command and what you saw. diff --git a/docs/dev/smoke-test.md b/docs/dev/smoke-test.md deleted file mode 100644 index c49f7a1..0000000 --- a/docs/dev/smoke-test.md +++ /dev/null @@ -1,192 +0,0 @@ -# Manual smoke-test record - -Two manual suites gate v1 completion: a local two-notebook suite that needs -no Databricks workspace, and a real-workspace suite that needs explicit, -non-destructive credentials. Automated coverage (`make check`) fakes the -Molten, process, SDK, and UI boundaries — these suites are what verify the -real seams. - -This file is the **record**. For the step-by-step run-through with exact -commands and expected results, follow -[`smoke-test-runbook.md`](smoke-test-runbook.md), then transfer the outcome -here. - -## Status - -| Suite | Status | Date | Verified by | -| ------------------------------- | -------------------------- | ---------- | ---------------- | -| Local two-notebook smoke | **Passed** | 2026-07-22 | Tomos Corbin | -| Real Databricks workspace smoke | **Passed** | 2026-07-22 | Tomos Corbin | - -Record results by checking the boxes below and filling in the environment -table. Never paste secret values or tokens into this file. - -## Environment record - -Fill in when a suite is run: - -| Item | Value | -| -------------------------------------------- | -------------------------- | -| Date | 2026-07-22 | -| OS | Linux 7.1.3-arch1-2 x86_64 | -| Neovim version | v0.12.4 | -| molten-nvim commit | `bedea63` (probe-pinned) | -| Python version | 3.12.3 | -| uv version | 0.9.5 | -| databricks-connect version | 18.3.2 | -| databricks-sdk version | 0.121.0 | -| Compute (serverless / cluster + DBR version) | Serverless (Free Edition) | -| Profile auth type | OAuth (browser login) | - -## Local two-notebook smoke - -Setup: the committed fixtures `examples/smoke/notebook_a.py` and -`examples/smoke/notebook_b.py` (one project, marked by -`examples/smoke/.databricks/`), opened in your regular config with molten-nvim -installed — see the runbook's Prerequisites and Launch sections. Local IPython -kernels only; no workspace is required. - -- [x] `:DatabricksInit` in each notebook creates two independent sessions; - variables defined in one are absent in the other. -- [x] Widget registrations and YAML overrides resolve per notebook key and do - not cross notebooks. -- [x] `:DatabricksRunCell` submits the exact cell range; inline output - appears at the correct cell. -- [x] `%md` renders as Markdown output. -- [x] `%sql` raises the named unsupported-directive error. -- [x] `:DatabricksInterrupt` interrupts a long-running cell in one notebook - while the other notebook keeps working. -- [x] `:DatabricksInit` reports staged progress (environment validation, - compute discovery, picker prompts), and `:DatabricksRunCell` during - initialisation reports that initialisation is still in progress. - (Stale-capture rejection is not manually reachable — capture and - submission happen in one synchronous command — and is covered by the - automated tests.) -- [x] Removing the header deactivates the notebook and closes only its - session. -- [x] `:e!` reload closes the session; `:DatabricksInit` afterwards reuses - the saved selection and works. -- [x] Unloading and wiping a notebook buffer cleans up without errors. -- [x] Repeated `require("databricks-notebook").setup()` preserves the live session of - an untouched notebook. -- [x] `:checkhealth databricks-notebook` renders execution and per-project - environment findings without contacting a workspace. -- [x] Quitting Neovim leaves no stray kernel processes. - -Result notes: - -``` -This run surfaced and fixed five real bugs, none of which were about the -checklist items above per se — they blocked getting far enough to run the -checklist at all: - -- kernelspec.lua wrote "metadata": [] (JSON array) instead of {} (object); - jupyter_client rejects a kernelspec shaped like that outright, so no - kernel could ever start. Fixed with vim.empty_dict() (9c33af8). -- extension.py never called builder.serverless() when no cluster_id was - selected, leaving the compute target ambiguous; on this Free Edition - workspace that crashed the kernel process during startup. Fixed - (ddc585d); bumped databricks-connect floor to >=15.1 for .serverless(). -- molten-nvim's own client-side connection-file write assumes - ~/.local/share/jupyter/runtime already exists; on a machine that's never - run a full Jupyter server it doesn't, and neither jupyter_client nor - jupyter_core creates it. Documented as a one-time prerequisite (a9960d6). -- No cleanup ran on a normal Neovim quit (:qa) — only on BufUnload/ - BufWipeout — so a live kernel outlived the editor as an orphaned - ipykernel process. Confirmed live with `pgrep -fl ipykernel` after - quitting. Fixed with a VimLeavePre hook that closes every tracked - session (cf104a1); re-verified clean afterwards. - -Once past those, every item above passed as expected. - -2026-07-22 re-run (on main after the review-remediation PRs #17-#22), from -a cleared selection state so both pickers were re-exercised: full pass. -The %md check initially showed IPython's Markdown repr — root-caused to -molten rendering only text/plain (it has no text/markdown support) while -the dialect's plain fallback was the object repr; fixed by publishing a -readable plain-text fallback (PR #23), after which markdown rendered as -the reworded runbook expects. Molten's synchronous event handling makes -the editor sluggish during the kernel-attach window; recorded as a v1 -limitation (PR #24). Markdown-cell authoring friction went to the post-v1 -roadmap (PR #25). -``` - -## Real Databricks workspace smoke - -Requires explicit authorization, a non-destructive profile, and safe -resources. Read-only operations only; never print secret values. - -- [x] `:DatabricksInit` selects the intended profile and compute; the - Connect session targets them (verify via `spark` configuration). -- [x] A simple Spark action (e.g. `spark.range(5).collect()`) returns a - result independently verified in the workspace UI or CLI. -- [x] `dbutils.fs.ls` on a known-safe path returns expected entries. - **Root-caused 2026-07-21** (originally recorded as an open bug): replayed - outside the editor against the same kernel and workspace, the call fails - with the platform error `PermissionDenied: Public DBFS root is disabled. - Access is denied on path: /` — Free Edition / serverless-only workspaces - have no public DBFS root, and a workspace notebook on the same tier - behaves identically. The plugin surfaces the SDK error at the cell - correctly; the runbook expectation was corrected and the tier behaviour - is documented in [limitations](limitations.md). -- [x] `dbutils.secrets.list` (or an equivalent safe read) works without any - secret value appearing in output or this record. -- [x] Widget defaults and `.databricks/widgets.yaml` overrides resolve - correctly; two notebooks keep isolated widget contexts. -- [x] `%md` renders; `%pip` installs into the local environment; `%sql` and - `%run` raise their named errors. -- [x] `:DatabricksInterrupt` cancels a deliberately long but safe action. -- [x] Restarting Neovim reuses the saved profile/compute selection without - prompting. -- [ ] Two notebooks against the workspace maintain namespace and output - isolation. Not separately exercised this run (`demo_notebook.py` is a - single notebook); already verified against local kernels in the suite - above, which is architecturally the same isolation mechanism. - -Result notes: - -``` -Run against a Free Edition workspace (serverless only). `%pip install -requests` initially failed with "No module named pip": `uv venv` creates an -interpreter with no pip installed, and IPython's built-in %pip magic shells -out to `python -m pip`. Fixed by seeding pip at venv creation (fcb5b89); -confirmed working after deleting and rebuilding the managed venv. - -Two known bugs found this run were triaged on 2026-07-21 (see the plan's -"Open bugs from smoke testing" for the full write-up): - -- dbutils.fs.ls("/"): workspace-tier behaviour, not a plugin defect. The - platform rejects it with "PermissionDenied: Public DBFS root is disabled" - on Free Edition / serverless-only workspaces. Reproduced over the raw - Jupyter protocol with the same kernelspec, venv, and workspace; runbook - and limitations updated. -- dbutils.notebook.exit(...) "does not print its value": not reproducible at - the kernel level. Replaying the exact cell through the managed kernelspec - over the raw Jupyter protocol shows the kernel publishing display_data - ("**Notebook exited:** finished demo early") with a successful reply, and - the behaviour is pinned by existing automated tests against real IPython - and ipykernel shells. The original note's code analysis was wrong — the - render call has existed since 8b1f2aa, three days before the smoke run. - Most plausible cause of the observation: the pre-rebuild venv held a stale - dialect package that environment validation could not detect (validation - only checked importability, not that the import resolved to the plugin - checkout). That gap is now closed — verification rejects a - databricks_kernel_dialect that resolves outside the plugin's bundled - package and preparation reinstalls it. Visual confirmation of the Molten - rendering is queued for the next manual smoke run. - -Correction (2026-07-22): the stale-venv hypothesis above was wrong. The -next run reproduced the missing render with a fresh, verified venv — %md -cells showed ''. Real cause: molten -renders only text/plain (it has no text/markdown support), and the -dialect's text/plain for its Markdown values was IPython's object repr. -Fixed by publishing a readable text/plain alongside the markdown; the -import-resolution validation added during the original triage stays, on -its own merits. - -2026-07-22 re-run: full pass, closing the queued visual item — -dbutils.notebook.exit renders "Notebook exited: finished demo early" at -the cell (via the PR #23 plain-text fallback) and the next cell still -runs. dbutils.fs.ls("/") raised the tier-expected PermissionDenied per -the corrected runbook. No secrets appear in output or this record. -``` diff --git a/docs/limitations.md b/docs/limitations.md deleted file mode 100644 index 80aede9..0000000 --- a/docs/limitations.md +++ /dev/null @@ -1,139 +0,0 @@ -# Known limitations (v1) - -Honest boundaries of databricks-notebook.nvim v1. Each is deliberate; several -are tracked as post-v1 roadmap items. - -## Execution runs locally through Databricks Connect - -Notebook code executes in a local IPython kernel. Spark operations run -remotely on your selected compute via Databricks Connect, but everything else -is local: - -- The filesystem, environment variables, and subprocesses are your machine's, - not a cluster driver's. -- `%pip` installs into the local managed environment, **not** onto the - cluster. -- Databricks Runtime preinstalled libraries are not present locally; install - what you need into the managed environment. -- Databricks Connect itself constrains which Spark features are available; - see Databricks' own Connect limitations for your runtime version. - -## dbutils.fs depends on your workspace tier - -`dbutils.fs` calls the workspace's storage REST APIs through the Databricks -SDK: `/Volumes/...` paths use the Files API, everything else the legacy DBFS -API. Workspaces without public DBFS root (Free Edition and other -serverless-only tiers) reject non-Volumes paths — `dbutils.fs.ls("/")` raises -`PermissionDenied: Public DBFS root is disabled` there, exactly as it does in -a workspace notebook on the same tier. Use `/Volumes/...` paths on those -workspaces. - -## No secret redaction - -`dbutils.secrets.get` returns real secret values via the Databricks SDK. -Unlike a workspace notebook, there is no output-redaction layer locally: -printing a secret prints the secret. Treat notebook output accordingly. - -## Unsupported directives and dbutils namespaces - -These raise a named error rather than failing obscurely: - -- `%sql` and `%run` -- `dbutils.notebook.run` -- `dbutils.jobs` and `dbutils.library` - -`.ipynb` notebooks are out of scope entirely; v1 supports the Databricks -`.py` source format only. - -## No run-all command - -`:DatabricksRunAll` does not exist in v1. The pinned Molten revision exposes -no supported per-evaluation completion/outcome event, so serial -stop-on-failure semantics cannot be implemented truthfully (see -[ADR 0002](decisions/0002-molten-backed-execution-service.md) and the -[Molten capability probe](dev/molten-integration-probe.md)). Run cells -individually; run-all returns when the upstream capability exists. - -## One kernel per notebook - -Every notebook owns an isolated session: its own kernel process, Python -namespace, widget context, and Databricks Connect session. That is what makes -interrupts targeted and state predictable, but it has a cost: - -- The first `:DatabricksInit` for a project pays for environment creation, - package installation, kernel startup, and Connect session establishment. -- Several open notebooks means several kernel processes. - -Sharing a kernel between notebooks is a possible post-v1 opt-in, not a v1 -behaviour. - -## One managed environment for all projects - -`venv_path` and `databricks_connect_version` are global settings: every -project shares one virtualenv and one Connect constraint. Two projects that -need conflicting `databricks-connect` versions will fight over it — the most -recent preparation wins. If this affects you, change the configured -constraint (and expect reinstalls when switching projects); per-project -environments are a post-v1 consideration. - -The bundled kernel-dialect package itself requires -`databricks-connect>=15.1` (serverless support), and uv resolves that -requirement jointly with your configured constraint — a constraint that -excludes 15.1+ (e.g. `">=14.3,<15"`) fails installation with a resolver -conflict. - -## Molten is a runtime dependency with a verified surface - -Execution and inline output depend on molten-nvim. The integration is -verified against Molten revision `bedea638`; `:checkhealth databricks-notebook` checks -that Molten's public commands and functions **exist**, not that they still -behave as probed. A Molten upgrade that changes behaviour can break execution -at runtime — pin Molten if you need stability. - -## The UI stalls while a kernel starts - -`:DatabricksInit` itself is asynchronous — environment preparation, compute -discovery, and the kernelspec write all happen in the background — but -Molten handles `CursorMoved`, `CursorMovedI`, and `WinScrolled` events plus -its readiness tick (every `molten_tick_rate` ms, default 500) as -synchronous RPCs into a single-threaded Python host. While a kernel boots -and builds its Connect session, that host is busy polling readiness, so -editor events queue behind it and input turns sluggish; it recovers the -moment the kernel attaches. Leaving the cursor still during the attach -stage avoids most of it, and raising `vim.g.molten_tick_rate` reduces the -stall frequency. The real fix is upstream in molten-nvim's event handling. - -## Reload closes the session - -Unloading, wiping, or reloading a notebook buffer (including `:e!`) shuts its -execution session down, because Molten deinitialises the buffer's kernels on -unload and a session record without a kernel would be falsely ready. Run -`:DatabricksInit` again after a reload; your saved profile/compute selection -is reused. - -## Edits invalidate captured runs - -`:DatabricksRunCell` captures the cell once and validates the snapshot -(document, buffer, change tick, range, exact source) immediately before -submission. Any edit in between rejects the run with a message instead of -executing text that no longer matches the buffer. This is deliberate: stale -work is refused, never remapped onto changed text. - -## Widgets are emulated, string-valued, and file-configured - -Widget values resolve from registrations in code plus quoted-string overrides -in `.databricks/widgets.yaml`. There is no widget UI, and non-string values -are rejected by design. - -## Namespace bindings are conventional, not enforced - -`spark`, `dbutils`, and `_databricks_runtime` are ordinary IPython namespace -bindings injected at kernel startup. Overwriting them in notebook code breaks -the dialect for that session; nothing can technically prevent it. - -## Health is offline - -`:checkhealth databricks-notebook` never authenticates or contacts a workspace, so it -cannot tell you whether your saved compute target is reachable — it labels -that explicitly. `:DatabricksInit` is the operation that verifies the -connection. diff --git a/examples/smoke/README.md b/examples/smoke/README.md index a4ab13f..f52ccd2 100644 --- a/examples/smoke/README.md +++ b/examples/smoke/README.md @@ -1,8 +1,8 @@ # Smoke-test kit -Fixtures for the manual local smoke test (see [`../../docs/dev/smoke-test.md`](../../docs/dev/smoke-test.md)): -two paired notebooks in one project, plus a launcher that loads this checkout of -databricks-notebook.nvim on its own. +Fixtures for manual local smoke testing: two paired notebooks in one project, +plus a launcher that loads this checkout of databricks-notebook.nvim on its +own. The command that makes `:Databricks*` appear is `setup()`, not the file's location. `init.lua` here calls it for you. @@ -25,12 +25,9 @@ molten-nvim is not loaded here, so it cannot run cells. `:DatabricksInit` and ## Execution smoke The launcher above cannot run cells — do not use it past the editor layer. -Install databricks-notebook.nvim **and** molten-nvim in your normal Neovim config (see -the [README](../../README.md) install snippet, including -`:UpdateRemotePlugins`), open both notebooks with that config, and follow the -step-by-step runbook in -[`../../docs/dev/smoke-test-runbook.md`](../../docs/dev/smoke-test-runbook.md) -(record the outcome in [`../../docs/dev/smoke-test.md`](../../docs/dev/smoke-test.md)). +Install databricks-notebook.nvim **and** molten-nvim in your normal Neovim +config (see the [README](../../README.md) install snippet, including +`:UpdateRemotePlugins`) and open both notebooks with that config. `notebook_a.py` and `notebook_b.py` are built to exercise session isolation, per-notebook widget resolution, the `%sql` named error, and a targeted interrupt. diff --git a/tests/manual/molten_probe.lua b/tests/manual/molten_probe.lua index bd1d7b1..680c11f 100644 --- a/tests/manual/molten_probe.lua +++ b/tests/manual/molten_probe.lua @@ -7,7 +7,7 @@ -- MOLTEN_KERNEL_B second installed local kernelspec name -- -- Generate the remote-plugin manifest in the isolated XDG data directory before --- running this script. See docs/dev/molten-integration-probe.md for the full recipe. +-- running this script (run :UpdateRemotePlugins with XDG_DATA_HOME pointed there). local molten_path = assert(vim.env.MOLTEN_NVIM_PATH, "MOLTEN_NVIM_PATH is required") local python_host = assert(vim.env.MOLTEN_PYTHON, "MOLTEN_PYTHON is required") From f33e7d3a1cd81b0f53b2b3656eb063099a29b7ea Mon Sep 17 00:00:00 2001 From: Tomos Corbin Date: Wed, 22 Jul 2026 22:53:20 +0100 Subject: [PATCH 2/3] docs: drop the Development section from the README --- README.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/README.md b/README.md index 1036e77..6672913 100644 --- a/README.md +++ b/README.md @@ -280,16 +280,6 @@ Deliberate v1 boundaries; several are tracked as post-v1 roadmap items. at kernel startup. Overwriting them in notebook code breaks the dialect for that session. -## Development - -```sh -make check # stylua/ruff format checks, ruff lint, strict mypy, Lua + Python tests -make test-lua # plenary specs via headless nvim -make test-py # pytest -``` - -CI runs the same checks on every pull request. - ## License [MIT](LICENSE) From 8960e2e8cf1b0269a98a2612e66d1c8210d39fe0 Mon Sep 17 00:00:00 2001 From: Tomos Corbin Date: Wed, 22 Jul 2026 22:54:46 +0100 Subject: [PATCH 3/3] docs: remove the examples directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The smoke kit lost its purpose when the smoke-test runbook was deleted, and the demo notebook was only a Quickstart convenience — the header line in the README is enough to create one. --- README.md | 3 +- examples/demo_notebook.py | 98 ------------------------- examples/smoke/.databricks/widgets.yaml | 6 -- examples/smoke/README.md | 33 --------- examples/smoke/init.lua | 20 ----- examples/smoke/notebook_a.py | 42 ----------- examples/smoke/notebook_b.py | 35 --------- 7 files changed, 1 insertion(+), 236 deletions(-) delete mode 100644 examples/demo_notebook.py delete mode 100644 examples/smoke/.databricks/widgets.yaml delete mode 100644 examples/smoke/README.md delete mode 100644 examples/smoke/init.lua delete mode 100644 examples/smoke/notebook_a.py delete mode 100644 examples/smoke/notebook_b.py diff --git a/README.md b/README.md index 6672913..fc715c1 100644 --- a/README.md +++ b/README.md @@ -67,8 +67,7 @@ it on install/update) and Neovim has restarted. If `:DatabricksInit` reports ## Quickstart 1. Open a Databricks source notebook — any Python file whose first line is - exactly `# Databricks notebook source` (try - [examples/demo_notebook.py](examples/demo_notebook.py)). + exactly `# Databricks notebook source`. 2. Run `:DatabricksInit`. On first use this creates the managed kernel environment, asks you to pick a Databricks profile and compute target (serverless or a running cluster), and starts the notebook kernel. Watch the diff --git a/examples/demo_notebook.py b/examples/demo_notebook.py deleted file mode 100644 index 1af48e0..0000000 --- a/examples/demo_notebook.py +++ /dev/null @@ -1,98 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC # Demo notebook -# MAGIC -# MAGIC A single source-format notebook for hand-testing databricks-notebook.nvim's -# MAGIC breadth: detection, cell editing, directives, dbutils, and one Spark -# MAGIC action. Open it with your normal setup: -# MAGIC -# MAGIC ``` -# MAGIC nvim examples/demo_notebook.py -# MAGIC ``` -# MAGIC -# MAGIC The `# MAGIC ` prefixes should be concealed, and each -# MAGIC `# COMMAND ----------` separator should show an overlaid rule. For -# MAGIC two-notebook session isolation, widget overrides, and a targeted -# MAGIC interrupt, use the paired notebooks in `examples/smoke/`. - -# COMMAND ---------- - -# A plain Python cell: it runs in the local kernel, so it produces output with -# no compute attached. LSP, ruff, and completion keep working because the -# buffer stays filetype=python. -message = "hello from the local kernel" -print(message) - -# COMMAND ---------- - -# A Spark action: this runs remotely on your selected compute via Databricks -# Connect, so the cell needs a running cluster or serverless. -from pyspark.sql import functions as F - -df = spark.range(100).withColumn("squared", F.col("id") * F.col("id")) -df.show(5) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## Editing cheatsheet -# MAGIC -# MAGIC - Navigate cells with `]n` / `[n`. -# MAGIC - Select a cell with `vic` (inner) or `vac` (around). -# MAGIC - Add, move, or delete cells with `:DatabricksCellAdd`, -# MAGIC `:DatabricksCellMove`, and `:DatabricksCellDelete`. -# MAGIC - Toggle conceal with `:DatabricksConcealToggle`. - -# COMMAND ---------- - -# Widgets are emulated locally. With no .databricks/widgets.yaml override for -# this notebook, get() returns the registered default. -dbutils.widgets.text("run_date", "2026-07-17") -print("run_date =", dbutils.widgets.get("run_date")) - -# COMMAND ---------- - -# MAGIC %sql -# MAGIC -- Unsupported in v1: running this cell raises a named error. -# MAGIC SELECT id, id * id AS squared FROM range(10) WHERE id % 2 = 0 - -# COMMAND ---------- - -# MAGIC %run ./shared/setup - -# COMMAND ---------- - -# MAGIC %pip install requests - -# COMMAND ---------- - -# dbutils.fs is a real workspace call via the Databricks SDK, so it needs a -# profile. Listing DBFS root is read-only. On workspaces without public DBFS -# root (Free Edition / serverless-only tiers) this raises PermissionDenied: -# "Public DBFS root is disabled" — use a /Volumes/... path there instead. -dbutils.fs.ls("/") - -# COMMAND ---------- - -# Long-running cell for the interrupt test. Run it, then :DatabricksInterrupt. -import time - -for i in range(60): - print(i) - time.sleep(1) - -# COMMAND ---------- - -# dbutils.notebook.exit stops this cell cleanly and renders the exit value. -dbutils.notebook.exit("finished demo early") - -# COMMAND ---------- - -# dbutils.notebook.run is unsupported in v1: running this cell raises a named -# error rather than starting another notebook. -dbutils.notebook.run("./shared/child", 60, {"run_date": "2026-07-17"}) - -# COMMAND ---------- - -# The final cell. Cell navigation should stop here at the bottom. -print("done") diff --git a/examples/smoke/.databricks/widgets.yaml b/examples/smoke/.databricks/widgets.yaml deleted file mode 100644 index 921b704..0000000 --- a/examples/smoke/.databricks/widgets.yaml +++ /dev/null @@ -1,6 +0,0 @@ -# Widget overrides for the smoke project. Top-level keys are notebook keys -# (the project-relative path without .py); values must be quoted strings. -# This directory also marks examples/smoke as the project root, so both -# notebooks share one project and keep their selection/state self-contained. -"notebook_a": - greeting: "override-from-yaml" diff --git a/examples/smoke/README.md b/examples/smoke/README.md deleted file mode 100644 index f52ccd2..0000000 --- a/examples/smoke/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# Smoke-test kit - -Fixtures for manual local smoke testing: two paired notebooks in one project, -plus a launcher that loads this checkout of databricks-notebook.nvim on its -own. - -The command that makes `:Databricks*` appear is `setup()`, not the file's -location. `init.lua` here calls it for you. - -## Editor-layer smoke (no dependencies) - -``` -nvim -u examples/smoke/init.lua examples/smoke/notebook_a.py -``` - -`init.lua` prepends this repo to the runtimepath and calls `setup()`, so -notebook detection and every buffer-local command work with no plugin manager -and no molten-nvim. Use this to verify detection, the `:Databricks*` commands, -cell add/delete/move, the `ic`/`ac` textobjects, `# MAGIC` conceal, -`:DatabricksWidgets`, and `:checkhealth databricks-notebook`. - -molten-nvim is not loaded here, so it cannot run cells. `:DatabricksInit` and -`:DatabricksRunCell` exist but need molten-nvim and a Databricks profile. - -## Execution smoke - -The launcher above cannot run cells — do not use it past the editor layer. -Install databricks-notebook.nvim **and** molten-nvim in your normal Neovim -config (see the [README](../../README.md) install snippet, including -`:UpdateRemotePlugins`) and open both notebooks with that config. -`notebook_a.py` and `notebook_b.py` are built to exercise session isolation, -per-notebook widget resolution, the `%sql` named error, and a targeted -interrupt. diff --git a/examples/smoke/init.lua b/examples/smoke/init.lua deleted file mode 100644 index 8b0789a..0000000 --- a/examples/smoke/init.lua +++ /dev/null @@ -1,20 +0,0 @@ --- Editor-layer smoke launcher for databricks-notebook.nvim. --- --- Loads THIS checkout of the plugin and nothing else, so notebook detection --- and every `:Databricks*` command can be verified with no plugin manager and --- no other plugins installed: --- --- nvim -u examples/smoke/init.lua examples/smoke/notebook_a.py --- --- molten-nvim is NOT loaded here, so cells cannot actually run: `:DatabricksInit` --- and `:DatabricksRunCell` exist but will fail without molten and a Databricks --- profile. This launcher verifies detection, buffer-local commands, cell --- add/delete/move, the `ic`/`ac` textobjects, `# MAGIC` conceal, --- `:DatabricksWidgets`, and `:checkhealth databricks-notebook`. Use your full config --- (with molten-nvim) for the execution steps. - -local script = debug.getinfo(1, "S").source:sub(2) -local repo_root = vim.fn.fnamemodify(script, ":p:h:h:h") -vim.opt.runtimepath:prepend(repo_root) - -require("databricks-notebook").setup() diff --git a/examples/smoke/notebook_a.py b/examples/smoke/notebook_a.py deleted file mode 100644 index c4d64b8..0000000 --- a/examples/smoke/notebook_a.py +++ /dev/null @@ -1,42 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC # Smoke notebook A -# MAGIC -# MAGIC Paired with `notebook_b.py` to exercise session isolation. The `# MAGIC ` -# MAGIC prefixes should be concealed and this cell should render as Markdown. - -# COMMAND ---------- - -# Python cell: defines a variable that lives only in this notebook's kernel. -value_a = "alpha" -print("A initialised:", value_a) - -# COMMAND ---------- - -# Widget cell: the code default is "a-default", but .databricks/widgets.yaml -# overrides the "notebook_a" key, so get() should print "override-from-yaml". -dbutils.widgets.text("greeting", "a-default") -print(dbutils.widgets.get("greeting")) - -# COMMAND ---------- - -# MAGIC %sql -# MAGIC -- Unsupported in v1: running this cell should raise a named error -# MAGIC -- rather than crashing the kernel. -# MAGIC SELECT 1 - -# COMMAND ---------- - -# Long-running cell for the interrupt test. Run it, switch to notebook B, then -# come back and :DatabricksInterrupt -- only A should stop. -import time - -for i in range(60): - print(i) - time.sleep(1) - -# COMMAND ---------- - -# Isolation check: value_b lives only in notebook B's kernel, so running this -# cell in A should raise NameError. -print(value_b) diff --git a/examples/smoke/notebook_b.py b/examples/smoke/notebook_b.py deleted file mode 100644 index 6fb6fcf..0000000 --- a/examples/smoke/notebook_b.py +++ /dev/null @@ -1,35 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC # Smoke notebook B -# MAGIC -# MAGIC Paired with `notebook_a.py`. Its kernel, namespace, and widget context -# MAGIC are independent of notebook A. - -# COMMAND ---------- - -# Python cell: a variable unique to this notebook's kernel. -value_b = "beta" -print("B initialised:", value_b) - -# COMMAND ---------- - -# No YAML override exists for "notebook_b", so get() should print the code -# default "b-default" -- proving widget keys do not cross notebooks. -dbutils.widgets.text("greeting", "b-default") -print(dbutils.widgets.get("greeting")) - -# COMMAND ---------- - -# MAGIC %sql -# MAGIC SELECT 2 - -# COMMAND ---------- - -# Quick cell to run while notebook A's loop is busy -- B should stay responsive. -print("B is responsive") - -# COMMAND ---------- - -# Isolation check: value_a lives only in notebook A's kernel, so running this -# cell in B should raise NameError. -print(value_a)