feat(charts): draw any Vega-Lite chart over the table

AGENTS.md

@@ -13,7 +13,7 @@ A pnpm workspace. Every package in `packages/` is published to npm under its own
   - Only the HTML renderer exists: notes are Markdown, and Vue, React, and Svelte versions would each copy the whole figure template.
   - `src/marks.ts` holds the hand-drawn marks, taken from AnnotationKit, and `src/script.ts` is the browser script the HTML renderer inlines.
 - `packages/carousel` is `comark-carousel`. Its parse plugin turns `::carousel` content into explicit `slide` nodes with their position, so no renderer splits children. Its script is built like zoom's, and a carousel is a zoom gallery.
-- `packages/charts` is `comark-charts`. `src/read.ts` checks a `::chart` and its table, and the parse plugin draws it with Flint and Vega and replaces its attributes with `type`, `title`, `label`, `svg`, and `marks`, keeping only the table as content, so no renderer loads Vega. Marks need comark-annotate's figure, so only the HTML renderer draws them.
+- `packages/charts` is `comark-charts`. `src/read.ts` checks a `::chart` and its table, or the `vega-lite` spec that draws any other chart over the table, and the parse plugin draws it with Flint and Vega and replaces its attributes with `type`, `title`, `label`, `svg`, and `marks`, keeping only the table as content, so no renderer loads Vega. Marks need comark-annotate's figure, so only the HTML renderer draws them.
 - `packages/chips` is `comark-chips`. Its parse plugin turns every web link, `@user`, and `:chip` into a `chip` node with `href`, `kind` (`favicon` or `avatar`), `src`, and `title`, so no renderer reads URLs. `src/images.ts` downloads each image while parsing and inlines it as a data URI, cached per process; an HTTP error maps to its status, and any other failure keeps the URL for the reader's browser. No script: the components add the styles on mount.
 - `packages/code` is `comark-code`. Its components replace `pre`: each wraps the block, rendered as it would be without them, in `.ck` with a bar of buttons, so Shiki's output and code groups work in any plugin order. The bar's markup is one string from `buttons()` in `src/client.ts` that every renderer inlines. Its parse plugin only writes the configured `actions` onto matching `pre` nodes; the script sends a `comark-code-action` event for them and does nothing else.
 - `packages/diff` is `comark-diff`. `src/read.ts` reads a `::diff` node, two fenced blocks before and after, into rows with a small LCS over lines and then over the words of each changed pair, and reports its issues; the plugin and the renderer both go through it. The HTML renderer draws a table with no script that doesn't zoom: its code scrolls in place. It bundles only `comark-zoom/theme`, like github-callouts.

docs/pages/compare/charts.md

@@ -78,7 +78,7 @@ The other two plugins draw the same chart from a JSON spec, and the page holds o
 | Frontmatter data *A spec bound with `:spec`* | — | ✓ | ✓ |
 | **Drawing** | | | |
 | Layout *Who picks the axes, scales, and labels* | Flint | You | Flint |
-| Chart types | Eight common ones | Any Vega chart | Flint's templates |
+| Chart types | Eight common ones, or any Vega-Lite chart over the table | Any Vega chart | Flint's templates |
 | Engine | Vega | Vega | Vega or ECharts |
 | **Output** | | | |
 | Format | SVG | SVG, image, or Vue component | SVG, image, or Vue component |
@@ -90,8 +90,8 @@ The other two plugins draw the same chart from a JSON spec, and the page holds o
 
 ## Which to use
 
-Use comark-charts for documents that agents write or check, such as pull request evidence and reports: the table is the source, and it stays in the page. It draws the common chart types and nothing else.
+Use comark-charts for documents that agents write or check, such as pull request evidence and reports: the table is the source, and it stays in the page. It lays out the common chart types itself, and draws any other from a Vega-Lite spec over the same table.
 
-Use [comark-vega](https://github.com/miguelrk/comark-packages/tree/main/packages/comark-vega) for charts those types don't cover, such as box plots, maps, and error bands, or for interactive charts in a Vue app.
+Use [comark-vega](https://github.com/miguelrk/comark-packages/tree/main/packages/comark-vega) for full Vega specs, data that isn't a table, such as maps, or interactive charts in a Vue app.
 
 Use [comark-flint-chart](https://github.com/miguelrk/comark-packages/tree/main/packages/comark-flint-chart) for Flint's layouts in a Vue app, or for charts drawn with ECharts.

packages/charts/README.md

@@ -46,7 +46,7 @@ The espresso machine arrives.
 :::
 ```
 
-Bars turn sideways when their labels are long, so file paths and test names stay whole. The [docs](https://comark.onmax.me/charts/) show the table each type expects.
+Bars turn sideways when their labels are long, so file paths and test names stay whole. For any other chart, put a [Vega-Lite](https://vega.github.io/vega-lite/) spec after the table in a `vega-lite` block; the table is its data. The [docs](https://comark.onmax.me/charts/syntax/) show the table each type expects.
 
 The plugin adds every problem to `meta.issues`, so a host can reject a document before publishing it. Each issue names its chart and says how to fix it.
 

packages/charts/docs/syntax.md

@@ -1,6 +1,6 @@
 ---
 title: Syntax
-description: The table each chart type reads, and the marks you can draw on a chart.
+description: The table each chart type reads, Vega-Lite specs for any other chart, and the marks you can draw on a chart.
 ---
 
 ```md
@@ -117,6 +117,33 @@ Bars turn sideways when their labels are long or there are many of them, so file
 :::
 ::
 
+## Any chart
+
+For a chart the types don't cover, such as a box plot, a heatmap, or error bands, put a [Vega-Lite](https://vega.github.io/vega-lite/) spec after the table in a `vega-lite` block. The table is still its data, so it stays in the page: each row becomes one data point, with a field per column named by its header.
+
+````md
+::chart{title="Latency per request (ms)"}
+| endpoint | ms |
+| --- | --: |
+| /search | 120 |
+| /search | 410 |
+| /login | 40 |
+| /login | 61 |
+
+```vega-lite
+{
+  "mark": "boxplot",
+  "encoding": {
+    "x": { "field": "endpoint", "type": "nominal" },
+    "y": { "field": "ms", "type": "quantitative" }
+  }
+}
+```
+::
+````
+
+Leave `data` out of the spec; the table replaces it. Cells that read as numbers become numbers, empty ones null, and the rest stay text. The chart takes the same colors and type as the others wherever the spec sets none, and its own `config` wins. `type`, `x`, `y`, and marks don't apply: name the columns in the spec's `encoding`, and draw notes with a `text` or `rule` layer. The plugin reports a spec that isn't JSON, a `data` it replaced, and a field that's neither a column nor made by a transform. The spec can't load files or URLs, and links in it are dropped.
+
 ## Marks
 
 Point at data with [comark-annotate](/annotate/)'s marks: put an `::annotation` inside the chart, after the table, and name the row it marks with `at`, the row's x value as the table writes it. Add `series` to mark one column's value rather than the whole row.

packages/charts/skills/comark-charts/SKILL.md

@@ -39,6 +39,8 @@ Props, all optional: `type` (below), `x` (the x column, default the first), `y`
 | How changes add up to a total | `waterfall` | the start total, each change (negative for a drop), then the end total, in one column |
 | Shares of one whole, five or fewer | `pie` | names, then one column of non-negative numbers |
 
+When none of these fit, such as a box plot or a heatmap, put a Vega-Lite spec after the table in a fenced `vega-lite` block. The table is its data, so leave `data` out of the spec and name the table's headers as fields; marks don't apply.
+
 For a before and after of one or two metrics, grouped `bar` reads best; past three, use `dumbbell`. Keep the rows in the order the reader should read them: charts draw them in table order.
 
 ## Mark what matters

packages/charts/src/draw.ts

@@ -7,7 +7,7 @@ import type { Field } from "vega-lite/types_unstable/channeldef.js";
 import type { LayerSpec, UnitSpec } from "vega-lite/types_unstable/spec/index.js";
 import { compile } from "vega-lite";
 import { INK, SERIES } from "./colors.ts";
-import type { Chart, XKind } from "./read.ts";
+import type { Chart, SpecChart, XKind } from "./read.ts";
 
 const { logger, None, parse, View } = vega;
 
@@ -346,6 +346,30 @@ function boxes(chart: Chart, view: vega.View, width: number, height: number): Ma
   return found;
 }
 
+/** Draws a compiled Vega spec to an SVG string, and returns the view for finding where its marks sit. */
+async function render(compiled: vega.Spec, loader?: vega.Loader) {
+  // Vega compiles its expressions with `new Function` unless told to interpret them, and Cloudflare Workers, where
+  // documents like these often render, forbid that.
+  const options: vega.ViewOptions = {
+    renderer: "none",
+    logLevel: None,
+    expr: expressionInterpreter,
+  };
+  if (loader) options.loader = loader;
+  const view = new View(parse(compiled, undefined, { ast: true }), options);
+  // Vega gives every group a role for screen readers, which the one `role="img"` hides anyway, an empty foreground
+  // path, and float noise such as `10.000000000000021`. Only tags change, since text holds the table's own numbers,
+  // and Vega escapes `>` in attributes.
+  const svg = (await view.toSVG())
+    .replace(/<path class="foreground"[^>]* d=""[^>]*\/>/g, "")
+    .replace(/<[^>]+>/g, (tag) =>
+      tag
+        .replace(/ (?:role|aria-roledescription|aria-hidden)="[^"]*"/g, "")
+        .replace(/(\.\d\d)\d+/g, "$1"),
+    );
+  return { view, svg };
+}
+
 /**
  * Draws a chart to an SVG string at the size Flint laid it out for, so its text reads at the size it was set, and
  * finds where each of its marks sits.
@@ -586,25 +610,85 @@ export async function draw(chart: Chart): Promise<{ svg: string; boxes: Map<numb
       spec.layer.push({ mark: { type: "point", opacity: 0 }, encoding });
     }
   }
-  // Vega compiles its expressions with `new Function` unless told to interpret them, and Cloudflare Workers, where
-  // documents like these often render, forbid that.
-  const view = new View(parse(compile(spec, { logger: quiet }).spec, undefined, { ast: true }), {
-    renderer: "none",
-    logLevel: None,
-    expr: expressionInterpreter,
-  });
-  // Vega gives every group a role for screen readers, which the one `role="img"` hides anyway, an empty foreground
-  // path, and float noise such as `10.000000000000021`. Only tags change, since text holds the table's own numbers,
-  // and Vega escapes `>` in attributes.
-  const svg = (await view.toSVG())
-    .replace(/<path class="foreground"[^>]* d=""[^>]*\/>/g, "")
-    .replace(/<[^>]+>/g, (tag) =>
-      tag
-        .replace(/ (?:role|aria-roledescription|aria-hidden)="[^"]*"/g, "")
-        .replace(/(\.\d\d)\d+/g, "$1"),
-    );
+  const { view, svg } = await render(compile(spec, { logger: quiet }).spec);
   const [, width = "0", height = "0"] = /viewBox="0 0 ([\d.]+) ([\d.]+)"/.exec(svg) ?? [];
   const found = boxes(chart, view, Number(width), Number(height));
   view.finalize();
   return { svg, boxes: found };
 }
+
+// The house look, set where a spec leaves it: shadcn/ui's charts, in the inks the stylesheet swaps for the page's.
+const SPEC_CONFIG = {
+  aria: false,
+  font: "sans-serif",
+  view: { stroke: null },
+  range: { category: [...SERIES] },
+  mark: { color: SERIES[0] },
+  bar: { cornerRadiusEnd: 4 },
+  line: { strokeWidth: 2 },
+  axis: {
+    domain: false,
+    ticks: false,
+    grid: true,
+    gridColor: INK.border,
+    labelColor: INK.muted,
+    titleColor: INK.muted,
+    labelFontSize: 12,
+    titleFontSize: 12,
+    titleFontWeight: "normal",
+    labelPadding: 8,
+    titlePadding: 10,
+  },
+  axisX: { grid: false },
+  legend: { labelColor: INK.muted, titleColor: INK.muted, labelFontSize: 12, titleFontSize: 12 },
+  text: { color: INK.foreground, fontSize: 12 },
+  title: { color: INK.foreground },
+} as const;
+
+// A document's spec may not reach the network or the disk where it's parsed: the table is its only data.
+const sealed = vega.loader();
+sealed.load = () =>
+  Promise.reject(new Error("a chart's spec can't load files or URLs; its data is the table"));
+// An empty href loads no image and makes no link.
+sealed.sanitize = () => Promise.resolve({ href: "" });
+
+// Composed Vega-Lite views can carry their own data declarations. The table is the sole data source,
+// so remove those declarations at every depth before injecting the table values at the root.
+function withoutData(value: unknown): unknown {
+  if (Array.isArray(value)) return value.map(withoutData);
+  if (!(value instanceof Object)) return value;
+  // SAFETY: JSON objects have string keys, and this value is known to be a non-array object.
+  const record = value as Record<string, unknown>;
+  const from = record.from;
+  if ("lookup" in record && from instanceof Object && "data" in from) {
+    throw new Error(
+      "lookup transforms with from.data are unsupported because the table is the only data source",
+    );
+  }
+  return Object.fromEntries(
+    Object.entries(value)
+      .filter(([key]) => key !== "data")
+      .map(([key, child]) => [key, withoutData(child)]),
+  );
+}
+
+/** Draws a Vega-Lite spec over the chart's table, in the house look wherever the spec sets none. */
+export async function drawSpec(chart: SpecChart): Promise<{ svg: string }> {
+  const { config, ...rest } = chart.spec;
+  // SAFETY: chart.spec is parsed JSON, so rest is always a plain object and the sanitizer preserves that shape.
+  const sanitized = withoutData(rest) as Record<string, unknown>;
+  // The spec came from JSON, and compile() checks it at runtime and throws on what it can't draw.
+  const spec: TopLevelSpec = JSON.parse(
+    JSON.stringify({
+      background: "transparent",
+      padding: PADDING,
+      ...sanitized,
+      data: { values: chart.values },
+      config: Object.assign({}, SPEC_CONFIG, config),
+    }),
+  );
+  const { view, svg } = await render(compile(spec, { logger: quiet }).spec, sealed);
+  view.finalize();
+  // A link in the drawing would run whatever URL the spec gives it, `javascript:` included.
+  return { svg: svg.replace(/<\/?a\b[^>]*>/g, "") };
+}

packages/charts/src/html.ts

@@ -5,7 +5,15 @@ import type { DrawnChart } from "./index.ts";
 import { css } from "./styles.ts";
 
 export { default } from "./index.ts";
-export type { Chart, ChartsMeta, ChartsOptions, ChartType, DrawnChart, XKind } from "./index.ts";
+export type {
+  Chart,
+  ChartsMeta,
+  ChartsOptions,
+  ChartType,
+  DrawnChart,
+  SpecChart,
+  XKind,
+} from "./index.ts";
 export { css };
 
 // Each render gets a fresh `state.context`, so a document carries the styles once.

packages/charts/src/index.ts

@@ -5,7 +5,7 @@ import { escapeHtml } from "comark/utils";
 import type { draw } from "./draw.ts";
 import { readChart } from "./read.ts";
 
-export type { Chart, ChartType, XKind } from "./read.ts";
+export type { Chart, ChartType, SpecChart, XKind } from "./read.ts";
 
 export type ChartsMeta = {
   /** One readable sentence per problem with a `::chart`. Other plugins add theirs too. */
@@ -68,24 +68,36 @@ export default defineComarkPlugin<ChartsOptions | void, ChartsMeta>((options) =>
         node[1] = {};
         continue;
       }
+      const spec = chart.type === "vega-lite";
       const label =
-        chart.title || `${chart.type} chart of ${chart.series.join(", ")} by ${chart.x}`;
+        chart.title ||
+        (spec
+          ? `chart of ${chart.columns.join(", ")}`
+          : `${chart.type} chart of ${chart.series.join(", ")} by ${chart.x}`);
       let drawn: Awaited<ReturnType<typeof draw>>;
       try {
         // Flint and Vega take hundreds of milliseconds to load, so a document with no chart never loads them.
-        drawn = await (await import("./draw.ts")).draw(chart);
+        const module = await import("./draw.ts");
+        drawn = spec
+          ? { ...(await module.drawSpec(chart)), boxes: new Map() }
+          : await module.draw(chart);
       } catch (error) {
+        const fix = spec
+          ? "Check its Vega-Lite spec against https://vega.github.io/vega-lite/docs/"
+          : "Check the table, or report it at https://github.com/onmax/comark-plugins/issues";
         issues.push(
-          `\`::chart\` ${number} couldn't be drawn (${error instanceof Error ? error.message : String(error)}), so it shows its table alone. Check the table, or report it at https://github.com/onmax/comark-plugins/issues.`,
+          `\`::chart\` ${number} couldn't be drawn (${error instanceof Error ? error.message : String(error)}), so it shows its table alone. ${fix}.`,
         );
         node.splice(1, node.length - 1, {}, table);
         continue;
       }
       // Marks the chart couldn't place, such as one on an empty cell, are left out rather than drawn in a corner.
-      const marks = chart.marks.flatMap((mark) => {
-        const box = drawn.boxes.get(mark.n);
-        return box ? [{ ...mark, ...box }] : [];
-      });
+      const marks = spec
+        ? []
+        : chart.marks.flatMap((mark) => {
+            const box = drawn.boxes.get(mark.n);
+            return box ? [{ ...mark, ...box }] : [];
+          });
       const attrs: DrawnChart = {
         type: chart.type,
         title: chart.title,