Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 59 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ is rendered by [molten-nvim](https://github.com/benlubas/molten-nvim).
- Local widget emulation from project-owned YAML.
- Structured `:checkhealth databricks-notebook` diagnostics with remediation.

See [docs/limitations.md](docs/limitations.md) for what this deliberately does
not do in v1.
See [Limitations](#limitations) for what this deliberately does not do in v1.

## Requirements

Expand Down Expand Up @@ -68,8 +67,7 @@ it on install/update) and Neovim has restarted. If `:DatabricksInit` reports
## Quickstart

1. Open a Databricks source notebook — any Python file whose first line is
exactly `# Databricks notebook source` (try
[examples/demo_notebook.py](examples/demo_notebook.py)).
exactly `# Databricks notebook source`.
2. Run `:DatabricksInit`. On first use this creates the managed kernel
environment, asks you to pick a Databricks profile and compute target
(serverless or a running cluster), and starts the notebook kernel. Watch the
Expand Down Expand Up @@ -223,17 +221,63 @@ the installed Connect version), kernelspec location, and each open notebook's
saved selection. It runs offline and never contacts your workspace; workspace
reachability is exercised by `:DatabricksInit`.

## Development

```sh
make check # stylua/ruff format checks, ruff lint, strict mypy, Lua + Python tests
make test-lua # plenary specs via headless nvim
make test-py # pytest
```

Architecture decisions live in [docs/decisions/](docs/decisions/). Maintainer
material — the manual smoke-test procedure and its latest results — lives in
[docs/dev/](docs/dev/).
## Limitations

Deliberate v1 boundaries; several are tracked as post-v1 roadmap items.

- **Execution is local-first.** Notebook code runs in a local IPython kernel;
Spark operations run remotely on your selected compute via Databricks
Connect. The filesystem, environment variables, and subprocesses are your
machine's, not a cluster driver's. `%pip` installs into the local managed
environment, not onto the cluster, and Databricks Runtime preinstalled
libraries are not present locally — install what you need into the managed
environment. Databricks Connect itself constrains which Spark features are
available.
- **`dbutils.fs` depends on your workspace tier.** `/Volumes/...` paths use
the Files API; everything else uses the legacy DBFS API. Workspaces without
a public DBFS root (Free Edition and other serverless-only tiers) reject
non-Volumes paths with `PermissionDenied` — exactly as they do in a
workspace notebook on the same tier. Use `/Volumes/...` paths there.
- **No secret redaction.** `dbutils.secrets.get` returns real secret values,
and unlike a workspace notebook there is no output-redaction layer locally:
printing a secret prints the secret.
- **Unsupported surface fails by name, not obscurely.** `%sql`, `%run`,
`dbutils.notebook.run`, `dbutils.jobs`, and `dbutils.library` raise a named
error. `.ipynb` notebooks are out of scope entirely — v1 supports the
Databricks `.py` source format only.
- **No run-all command.** The pinned Molten revision exposes no supported
per-evaluation completion event, so serial stop-on-failure semantics cannot
be implemented truthfully. Run cells individually; run-all returns when the
upstream capability exists.
- **One kernel per notebook.** Isolation is what makes interrupts targeted
and state predictable, but the first `:DatabricksInit` for a project pays
for environment creation, package installation, kernel startup, and Connect
session establishment — and several open notebooks means several kernel
processes.
- **One managed environment for all projects.** `venv_path` and
`databricks_connect_version` are global: every project shares one
virtualenv and one Connect constraint, and conflicting requirements fight
over it (the most recent preparation wins). The bundled kernel dialect
itself requires `databricks-connect>=15.1`, resolved jointly with your
configured constraint.
- **Molten is a runtime dependency with a verified surface.** The integration
is verified against Molten revision `bedea638`; health checks that Molten's
public commands and functions exist, not that they still behave as probed.
Pin Molten if you need stability.
- **The UI stalls while a kernel starts.** Molten handles cursor and scroll
events plus its readiness tick as synchronous RPCs into a single-threaded
Python host, so input turns sluggish while a kernel boots and builds its
Connect session; it recovers the moment the kernel attaches. Keeping the
cursor still during the attach stage avoids most of it, and raising
`vim.g.molten_tick_rate` reduces the stall frequency.
- **Reload closes the session.** Unloading, wiping, or reloading a notebook
buffer (including `:e!`) shuts its execution session down. Run
`:DatabricksInit` again after a reload; your saved profile/compute
selection is reused.
- **Namespace bindings are conventional, not enforced.** `spark`, `dbutils`,
and `_databricks_runtime` are ordinary IPython namespace bindings injected
at kernel startup. Overwriting them in notebook code breaks the dialect for
that session.

## License

Expand Down
226 changes: 0 additions & 226 deletions doc/databricks-notebook.txt

This file was deleted.

Loading
Loading