databricks-notebook.nvim

Edit and run Databricks .py source notebooks in Neovim.

The plugin treats a Databricks source notebook as two things at once: Lua owns it as an editable document (cells, navigation, structural edits, appearance), and a bundled Python kernel dialect owns it as an executable Databricks program (directives, widgets, dbutils, Spark via Databricks Connect). Inline output is rendered by molten-nvim.

• One isolated execution session per notebook — no shared namespaces, targeted interrupts.
• Real spark and dbutils.fs/dbutils.secrets backed by Databricks Connect and the Databricks SDK.
• Local widget emulation from project-owned YAML.
• Structured :checkhealth databricks-notebook diagnostics with remediation.

See Limitations for what this deliberately does not do in v1.

Requirements

• Neovim 0.11+
• uv on PATH (manages the kernel environment)
• Python 3.11+ discoverable by uv
• molten-nvim, installed and initialised per its own documentation (remote plugin + providers). The integration is verified against Molten revision bedea638.
• A Databricks config file (default ~/.databrickscfg) with at least one profile — create one with databricks auth login.

Installation

With lazy.nvim:

{
  "Tomoscorbin/databricks-notebook.nvim",
  dependencies = {
    {
      "benlubas/molten-nvim",
      -- The revision this plugin is verified against (see Requirements).
      commit = "bedea63819c618e007e7c40059fc6e72d598c8df",
      build = ":UpdateRemotePlugins",
    },
  },
  ft = "python",
  config = function()
    require("databricks-notebook").setup()
  end,
}

setup() reconciles buffers that were loaded before the plugin, so lazy loading on ft = "python" is safe.

Any plugin manager works — the plugin only needs to be on the runtimepath before require("databricks-notebook").setup() runs. With other managers, install both plugins as usual and call setup() from your config.

molten-nvim is a Python remote plugin: its commands exist only after :UpdateRemotePlugins has generated the manifest (the build key above runs it on install/update) and Neovim has restarted. If :DatabricksInit reports molten_unavailable even though molten is installed, run :UpdateRemotePlugins and restart.

Quickstart

1. Open a Databricks source notebook — any Python file whose first line is exactly # Databricks notebook source.
2. Run :DatabricksInit. On first use this creates the managed kernel environment, asks you to pick a Databricks profile and compute target (serverless or a running cluster), and starts the notebook kernel. Watch the staged progress notifications; readiness is reported only when the kernel is actually attached.
3. Run the cell under the cursor with <leader>dr (or :DatabricksRunCell). Output appears inline via Molten.
4. Interrupt the notebook's own kernel with :DatabricksInterrupt.

Subsequent :DatabricksInit calls reuse the saved profile/compute selection. Use :DatabricksInit! to force reselection — it also cancels an in-progress initialisation and starts over, which is the way out if a stage ever stalls.

Notebook detection

A buffer is treated as a notebook when its filetype is python and its first line is exactly the Databricks source header. The filetype stays python, so your LSP, treesitter, and formatting keep working. New or empty Python buffers are watched until they either gain the header (activating notebook features) or diverge from it. Deleting the header deactivates the notebook and closes its execution session.

Cells are separated by # COMMAND ---------- lines, and # MAGIC prefixes carry directive/markdown content, matching the Databricks export format byte-for-byte. # MAGIC prefixes can be visually concealed (conceal_magic = true by default; toggle per window with :DatabricksConcealToggle).

Commands

All commands are buffer-local and exist only in active notebooks.

Command	Action
:DatabricksInit[!]	Prepare the project environment and start the notebook session. ! cancels an in-progress initialisation and forces profile/compute reselection.
:DatabricksRunCell	Run the cell under the cursor in this notebook's kernel.
:DatabricksInterrupt	Interrupt this notebook's kernel only.
:DatabricksCellAdd above|below	Insert an empty cell relative to the current cell.
:DatabricksCellDelete	Delete the current cell.
:DatabricksCellMove up|down	Swap the current cell with its neighbour.
:DatabricksWidgets	Open the project's .databricks/widgets.yaml (prepares a template for a new file).
:DatabricksConcealToggle	Toggle # MAGIC concealment in the current window.

Cell edits are single undo steps with deterministic cursor placement, and they never rewrite cell content — only structure.

Keymaps and textobjects

Buffer-local, active notebooks only:

Mapping	Mode	Action
]n	n	Next cell
[n	n	Previous cell
<leader>dr	n	Run current cell
ic / ac	x, o	Inner / around cell textobject

Configuration

Defaults shown; every key is optional and validated.

require("databricks-notebook").setup({
  -- Managed kernel environment (shared across projects).
  venv_path = vim.fn.expand("~/.local/share/databricks-notebook.nvim/venv"),
  -- pip-style constraint for the installed databricks-connect.
  databricks_connect_version = ">=15.1",
  -- Where Databricks profiles are read from.
  databricks_config_path = vim.fn.expand("~/.databrickscfg"),
  -- How long DatabricksInit waits for the notebook kernel to report
  -- readiness before failing the session (kernel startup includes building
  -- the Databricks Connect session, which is network-bound).
  kernel_ready_timeout_ms = 180000,
  keymaps = {
    next_cell = "]n",
    prev_cell = "[n",
    run_cell = "<leader>dr",
  },
  -- Conceal `# MAGIC ` prefixes in notebook windows.
  conceal_magic = true,
})

Repeated setup() calls reconcile mappings and options in place and preserve live notebook sessions.

How it works

Project identity. The project root is the nearest ancestor containing a .databricks/ directory or a .git marker (falling back to the notebook's own directory). The notebook key is the project-relative path without .py — e.g. etl/daily_report. Identity is derived from the file path, never from Neovim's working directory.

Environment. :DatabricksInit provisions a uv-managed virtualenv at venv_path containing the bundled kernel dialect, ipykernel, and databricks-connect, then writes a notebook-scoped Jupyter kernelspec (named databricks-<profile>-<hash>, under your Jupyter data directory) that carries the project root, notebook key, profile, and optional cluster ID as environment variables. No credentials are ever written.

Selection persistence. Your profile/compute choice is saved per project under stdpath("state")/databricks-notebook.nvim/projects/ (user-local, versioned, atomic — never inside the repository). A malformed or incompatible state file is reported, not silently replaced; fix it with :DatabricksInit! or delete the file.

Sessions. Each notebook document owns exactly one execution session. Running a cell captures the buffer snapshot once and validates it before submission — if the buffer changed after capture, the run is rejected with a message instead of executing stale text. Closing, unloading, or wiping the buffer (including :e! reload) shuts the session down; run :DatabricksInit again after a reload.

Widgets

dbutils.widgets works locally: defaults come from the widget registration in your code, and per-notebook overrides come from project-owned YAML:

# .databricks/widgets.yaml — open with :DatabricksWidgets
"etl/daily_report":
  run_date: "2026-07-17"
  environment: "dev"

Top-level keys are notebook keys; values must be quoted strings. The text/dropdown/combobox/multiselect/get/getArgument/getAll/ remove/removeAll API is supported.

Directives and dbutils

Feature	Behaviour
%md	Markdown source shown inline as text output.
%pip	Runs in the local managed environment (not on the cluster).
%sql, %run	Named error (unsupported in v1).
dbutils.fs, dbutils.secrets	Real workspace calls via the Databricks SDK.
dbutils.widgets	Local YAML-backed emulation.
dbutils.notebook.exit(value)	Stops the cell cleanly and renders the exit value.
dbutils.notebook.run, dbutils.jobs, dbutils.library	Named error (unsupported).

Cell transformation preserves line numbers, so kernel tracebacks point at the correct lines of your buffer.

Health

:checkhealth databricks-notebook reports structured findings for the execution backend, notebook sessions, uv, profiles, the managed environment (including the installed Connect version), kernelspec location, and each open notebook's saved selection. It runs offline and never contacts your workspace; workspace reachability is exercised by :DatabricksInit.

Limitations

Deliberate v1 boundaries; several are tracked as post-v1 roadmap items.

• Execution is local-first. Notebook code runs in a local IPython kernel; Spark operations run remotely on your selected compute via Databricks Connect. The filesystem, environment variables, and subprocesses are your machine's, not a cluster driver's. %pip installs into the local managed environment, not onto the cluster, and Databricks Runtime preinstalled libraries are not present locally — install what you need into the managed environment. Databricks Connect itself constrains which Spark features are available.
• dbutils.fs depends on your workspace tier. /Volumes/... paths use the Files API; everything else uses the legacy DBFS API. Workspaces without a public DBFS root (Free Edition and other serverless-only tiers) reject non-Volumes paths with PermissionDenied — exactly as they do in a workspace notebook on the same tier. Use /Volumes/... paths there.
• No secret redaction. dbutils.secrets.get returns real secret values, and unlike a workspace notebook there is no output-redaction layer locally: printing a secret prints the secret.
• Unsupported surface fails by name, not obscurely. %sql, %run, dbutils.notebook.run, dbutils.jobs, and dbutils.library raise a named error. .ipynb notebooks are out of scope entirely — v1 supports the Databricks .py source format only.
• No run-all command. The pinned Molten revision exposes no supported per-evaluation completion event, so serial stop-on-failure semantics cannot be implemented truthfully. Run cells individually; run-all returns when the upstream capability exists.
• One kernel per notebook. Isolation is what makes interrupts targeted and state predictable, but the first :DatabricksInit for a project pays for environment creation, package installation, kernel startup, and Connect session establishment — and several open notebooks means several kernel processes.
• One managed environment for all projects. venv_path and databricks_connect_version are global: every project shares one virtualenv and one Connect constraint, and conflicting requirements fight over it (the most recent preparation wins). The bundled kernel dialect itself requires databricks-connect>=15.1, resolved jointly with your configured constraint.
• Molten is a runtime dependency with a verified surface. The integration is verified against Molten revision bedea638; health checks that Molten's public commands and functions exist, not that they still behave as probed. Pin Molten if you need stability.
• The UI stalls while a kernel starts. Molten handles cursor and scroll events plus its readiness tick as synchronous RPCs into a single-threaded Python host, so input turns sluggish while a kernel boots and builds its Connect session; it recovers the moment the kernel attaches. Keeping the cursor still during the attach stage avoids most of it, and raising vim.g.molten_tick_rate reduces the stall frequency.
• Reload closes the session. Unloading, wiping, or reloading a notebook buffer (including :e!) shuts its execution session down. Run :DatabricksInit again after a reload; your saved profile/compute selection is reused.
• Namespace bindings are conventional, not enforced. spark, dbutils, and _databricks_runtime are ordinary IPython namespace bindings injected at kernel startup. Overwriting them in notebook code breaks the dialect for that session.