docs: add versioned documentation site built with Zensical

[nielspardon]

What

Adds a user-facing documentation site under docs/, built with
Zensical (from the Material for MkDocs team) and published
as multi-version docs via the Zensical-compatible fork of
mike. There was previously no user-facing
guide — users learned the API from the READMEs and tests. Tooling and layout
mirror substrait-io/substrait-python so the two repositories stay consistent.

Contents

• docs/ — comprehensive guides segmented into the four modules:
  • Core — building plans (the SubstraitBuilder DSL), types, expressions,
    relations, serialization, extended expressions, function & type extensions.
  • Isthmus — SQL ⇄ Substrait, SQL expressions, Substrait → SQL / Calcite,
    the type system, customization, and supported SQL.
  • Isthmus CLI — install/build, usage (all flags), and worked examples.
  • Spark — compatibility & per-variant dependencies, producing and consuming
    plans, supported features, and the end-to-end example.
  • Plus a landing page and getting-started. Every code sample is grounded in the
    real APIs and tests rather than invented.
• zensical.toml — Material theme, site_url, explicit nav, and the mike
  version provider (renders the version selector).
• pixi.toml — a docs environment (zensical + the mike fork) with
  docs-build / docs-serve tasks. Python and the docs dependencies are managed
  with pixi.
• CI — .github/workflows/docs.yml build-checks the docs on every PR and push
  to main. .github/workflows/docs-deploy.yml publishes versioned docs to the
  gh-pages branch via mike on release tags (the bare X.Y.Z tag publishes the
  minor series X.Y and updates the latest alias), and can be run manually via
  workflow_dispatch.
• readme.md / CONTRIBUTING.md — a Documentation section (hosted URL, local
  preview/build, authoring guidelines).

Building locally

pixi run docs-serve   # live-reloading preview at http://localhost:8000
pixi run docs-build   # build the static site into ./site

Enabling GitHub Pages (one-time, after merge)

Versioned publishing needs the gh-pages branch to exist before Pages can point
at it:

1. Actions → Deploy documentation → Run workflow (creates gh-pages), or
   let the next release tag create it.
2. Settings → Pages → Source → Deploy from a branch → gh-pages / (root).
3. From then on, each release tag publishes that minor series and updates the
   latest alias. Adjust site_url in zensical.toml if the site is served from
   a custom domain.

Verification

• pixi run docs-build completes with no issues (Zensical includes link validation).
• Dry-ran mike deploy + mike set-default locally (no push): correct gh-pages
  layout (<version>/, latest/, .nojekyll, versions.json, root redirect).
• Draft PR: the deploy workflow is intentionally not run against the canonical repo
  until GitHub Pages is enabled.

🤖 Generated with AI

[mbwhite]

will we be able to include the documentation from the JavaDoc as well?

[nielspardon]

will we be able to include the documentation from the JavaDoc as well?

probably something we can investigate in a follow-up. technically, it is included via links but need to check what Zensical currently supports in terms of embedding it

[bestbeforetoday]

Is there output we can look at where you have run this on your fork so we don't have to manually stand up and run the doc build process to review it?

[nielspardon]

Is there output we can look at where you have run this on your fork so we don't have to manually stand up and run the doc build process to review it?

I don't have this deployed. Running it locally is straightforward if you have pixi installed:

pixi run docs-serve

[bestbeforetoday]

It would give some confidence that the GitHub Actions workflows are good if they were run in a fork before merging here, although I guess it should not break any other part of the build.

I suspect some repository settings might need to change to allow the workflow to run successfully and published documentation to be viewable. Any changes can be identified and documented by testing in a fork.

[nielspardon]

It would give some confidence that the GitHub Actions workflows are good if they were run in a fork before merging here, although I guess it should not break any other part of the build.

I suspect some repository settings might need to change to allow the workflow to run successfully and published documentation to be viewable. Any changes can be identified and documented by testing in a fork.

I have everything prepared. I can not finalize the repo settings without this being merged and run.

[benbellick]

I haven't read super closely, but the docs themselves are nice looking! My biggest concern is the risk of docs getting out of date. For example, a lot of the docs have code embedded in them. Is there some way we can ensure that the code there does run and is up-to-date with the current codebase?

[nielspardon]

Great point, and thanks for pushing on it — I've reworked the embedded code so it's verified rather than hand-maintained.

Runnable samples are now pulled into the Markdown from compiled, CI-executed tests using pymdownx.snippets, so the code shown in the docs is the code CI compiles and runs:

• core and isthmus — backing JUnit tests in io.substrait…docs packages (50 and 11 tests).
• spark — a DocExamplesSuite that runs a full produce → serialize → consume → execute round trip on every Spark variant. (The Spark pages are now Scala, to match the executed source.)

A test marks a region with // --8<-- [start:name], and the page references it, e.g. --8<-- "core/src/test/java/io/substrait/docs/TypesDocTest.java:aliases". Two safety nets keep it honest:

• pixi run docs-build fails if a referenced file or region is missing (check_paths), and
• the backing tests fail the Gradle build if the API changes.

Writing the tests already earned their keep — they surfaced three examples that never actually ran: the building-plans and relations end-to-end plans needed a remap (project appends its expressions, so the named root(...) had the wrong field count), and the core landing-page snippet needed an explicit ExecutionBehavior.

A few blocks stay inline on purpose — pure API-signature reference (e.g. the isthmus convert(...) signatures) and a couple of illustrative/resource-dependent snippets. The convention ("edit the test, not the Markdown") is written up in CONTRIBUTING.md and AGENTS.md.