document private agent tools

pkg-r/R/chat.R

@@ -11,13 +11,11 @@
 #' @return A [shiny::shinyApp()] object.
 #'
 #' @section Citations and provenance:
-#' The server verifies each `<commons-citation>` the model writes against its
-#' own context, measure definitions, and data documentation as the answer
-#' streams, and rewrites verified citations inline as server-authored
-#' `<shiny-aside>` elements. Citation details name the
-#' trusted source. A provenance marker `<shiny-aside>` follows
-#' the answer when it was produced by a trusted calculation, or when a
-#' fallback answer cites nothing verified.
+#' The server verifies citations against trusted calculations, context, and data
+#' documentation as the answer streams. Verified citations appear inline, with
+#' details that name the trusted source. A provenance marker follows the answer
+#' when it was produced by a trusted calculation, or when a fallback answer
+#' cites nothing verified.
 #'
 #' @examples
 #' \dontrun{

pkg-r/R/commons.R

@@ -1,7 +1,10 @@
 #' Create a commons agent
 #'
-#' `commons()` creates an [ellmer::Chat] subclass with tools for a semantic
-#' layer, context search, table inspection, and SQL queries.
+#' `commons()` creates an [ellmer::Chat] subclass with tools and prompting that
+#' allow the agent to navigate its data sources, semantic layer, and context
+#' layer. Depending on the agent's choice of tools, responses can be
+#' deterministically classified as based on a trusted calculation, cited, or
+#' untrusted.
 #'
 #' The provider and model come from `client`; commons sets its own system prompt
 #' and tools. Use `agent$chat()` to ask questions, [commons_theme()] and
@@ -13,8 +16,7 @@
 #'   ignored, with a warning; use `instructions` to add to commons' prompt.
 #' @param data_sources A [data_source()], or a named list of them. Measures
 #'   can take a source's connection as an argument named after the source; see
-#'   [semantic_layer()]. When there are several sources, the `run_sql` and
-#'   `describe_table` tools take a source's name as a `source` argument.
+#'   [semantic_layer()].
 #' @param semantic_layer An optional [semantic_layer()].
 #' @param context_layer An optional [context_layer()].
 #' @param ... These dots are for future extensions and must be empty.
@@ -28,7 +30,8 @@
 #'     instructions = "Use the organization's fiscal-year conventions."
 #'   )
 #'   ```
-#' @param network Whether the `run_r` session has network access. One of
+#' @param network Whether the agent's R session has network access. One
+#'   of
 #'   `"none"` (the default) or `"full"`. The session uses OS sandboxing on
 #'   Linux and macOS. On unsupported hosts, local development can opt in to
 #'   best-effort R guardrails with
@@ -51,6 +54,27 @@
 #'   role is viewer cannot read traces even when named here; trace readers
 #'   need at least a publisher account.
 #'
+#' @section Agent tools:
+#' Depending on its semantic layer, context layer, and data sources, a commons
+#' agent receives some combination of these tools:
+#'
+#' * `search_pool` searches trusted calculations and semantic models.
+#' * `search_catalog` searches a warehouse catalog.
+#' * `call_measure` invokes an R measure.
+#' * `call_metrics` invokes governed or warehouse-native metrics.
+#' * `call_calculation` invokes an exact trusted query.
+#' * `search_context` retrieves relevant business context.
+#' * `describe_table` inspects a table or semantic model.
+#' * `run_sql` executes a read-only SQL query.
+#' * `run_r` executes R code to analyze results and render plots in the agent's
+#'   R session.
+#'
+#' These model-facing tools should be considered private. Their constructors
+#' are intentionally not exported, and their names, arguments, availability,
+#' and behavior may change without notice. Application code should configure
+#' an agent through `commons()` and its layer constructors rather than depend
+#' on individual tools.
+#'
 #' @return An [ellmer::Chat] subclass.
 #'
 #' @examples

pkg-r/R/context-layer.R

@@ -3,11 +3,10 @@
 #' A context layer contains text that helps a [commons()] agent interpret its
 #' data source.
 #'
-#' Files are chunked and indexed with \pkg{ragnar} when the agent first
-#' searches its context. Facts that should be in every prompt belong in the
-#' `instructions` passed to [commons()], not here.
+#' Context is retrieved when relevant. Facts needed in every conversation
+#' belong in the `instructions` passed to [commons()], not here.
 #'
-#' @param files Character vector of paths to text/markdown files to index.
+#' @param files Character vector of paths to text or Markdown files.
 #'
 #' @return A `commons_context_layer` object.
 #'

pkg-r/R/data-source.R

@@ -12,17 +12,13 @@
 #' * A `pins` board, e.g. [pins::board_connect()], is read into the same
 #'   in-process database: each pin in `tables` becomes a table. Pin names are
 #'   validated against the board at construction (a single listing call), but
-#'   each pin is downloaded only when its table is first used---by the
-#'   `describe_table` tool, a SQL query that references it, or a measure that
-#'   takes the source's connection. [commons_server()] starts a background
-#'   process right after startup that downloads the remaining pins into the
-#'   local pins cache, so a first use typically only reads an
-#'   already-downloaded file. A table reflects the pin's
+#'   each pin is downloaded only when its table is first used.
+#'   [commons_server()] starts a background process right after startup that
+#'   downloads the remaining pins into the local pins cache, so a first use
+#'   typically only reads an already-downloaded file. A table reflects the pin's
 #'   value at first use and is not refreshed for the lifetime of the data
-#'   source; if a pin can't be read (e.g. a network failure), the error
-#'   surfaces at that first use and the read is retried on the next one.
-#'
-#' The resulting object gives the agent a DBI connection plus a table registry.
+#'   source; if a pin can't be read (e.g. a network failure), the error surfaces
+#'   at that first use and the read is retried on the next one.
 #'
 #' @param ... A single DBI connection, a single `pins` board, or named data
 #'   frames to register as tables. When passing data frames, each name becomes
@@ -40,22 +36,13 @@
 #'   current schema. A Databricks `hive_metastore` selection must include a
 #'   schema. Snowflake selections import semantic views, and Databricks
 #'   selections import metric views, as native trusted metrics and dimensions.
-#'   Namespace selections retain lightweight model metadata, then read a
-#'   model's definition when the agent describes or uses it. Explicitly
-#'   selected models are read and validated when the data source is created.
-#'   Snowflake semantic variables and Databricks metric-view parameters are
-#'   passed as typed JSON arguments to `call_metrics`. Databricks wildcard
-#'   members require concrete column metadata from the warehouse.
-#'   Native semantic models are available through `search_pool`,
-#'   `describe_table`, and `call_metrics`, but are not registered as physical
-#'   tables.
-#'   Snowflake verified queries are exposed separately as exact trusted
-#'   calculations through `search_pool` and `call_calculation`; their SQL is
-#'   executed as stored rather than parsed to infer dependencies.
+#'   Namespace selections read model definitions lazily. Explicitly selected
+#'   models are read and validated when the data source is created.
+#'   Databricks wildcard members require concrete column metadata from the
+#'   warehouse.
 #'   An exact physical-table selection also imports associated models when
-#'   every physical dependency is selected. Public relationships, facts,
-#'   filters, and instructions become table-scoped first-touch and retrieval
-#'   context; private members remain hidden.
+#'   every physical dependency is selected. Only public relationships, facts,
+#'   filters, and instructions are exposed to the agent.
 #'
 #'   For a board, a named character vector of pins to read: the names become
 #'   table names, and the values are pin names passed to [pins::pin_read()].
@@ -69,43 +56,32 @@
 #' @section Data dictionaries:
 #' A data dictionary describes a data source's tables and columns: what each
 #' table's rows represent, what its columns mean, allowed values and units,
-#' how tables join, and definitions of domain terms. Its content reaches the
-#' agent three ways:
+#' how tables join, and definitions of domain terms. commons uses it to provide
+#' business context and governed definitions to the agent. See
+#' `vignette("commons", package = "commons")` for guidance on writing one.
 #'
-#' * The dataset-level `description` and `details`, along with the glossary,
-#'   are included in the system prompt. These fields are the place for rules
-#'   that span tables and for guidance on which tables answer which kinds of
-#'   questions.
-#' * The first time a conversation touches a table---via the `describe_table`
-#'   tool or a SQL query---the table's full dictionary entry rides along with
-#'   the tool result: its prose, documented columns, relationships, and
-#'   definitions of glossary terms it references. `describe_table` merges
-#'   documented columns with the table's live schema.
-#' * For Snowflake and Databricks sources, a fully qualified dictionary table
-#'   name matches the same selected relation. A relative name is accepted when
-#'   it matches only one selected relation. Authored prose takes precedence,
-#'   while warehouse column types remain authoritative.
-#' * When the agent also has a [context_layer()], the dictionary's prose is
-#'   indexed for the `search_context` tool.
+#' For Snowflake and Databricks sources, a fully qualified dictionary table
+#' name matches the same selected relation. A relative name is accepted when
+#' it matches only one selected relation. Authored prose takes precedence,
+#' while warehouse column types remain authoritative.
 #'
 #' A table's entry can also declare `definitions`: named expressions in the
 #' [data-dict expression language](https://data-dict.tidyverse.org/expressions.html).
 #' commons validates their inferred types and references, compiles them for
-#' the source's SQL backend, and lets the model apply them as `{{name}}`
-#' tokens in `run_sql` or through `call_metrics`. Definitions are delivered
-#' through all three channels above.
+#' the source's SQL backend, and makes them available to trusted metric
+#' calculations and custom SQL.
 #'
 #' @section Trust:
-#' The `run_sql` tool runs only read-only `SELECT` queries; statements that
+#' The agent runs only read-only `SELECT` queries; statements that
 #' would modify data or schema (`INSERT`, `UPDATE`, `DROP`, and similar) are
 #' rejected before reaching the database. For the in-process DuckDB built from
 #' data frames, commons additionally disables extension loading and filesystem
 #' access. These are safeguards, not a sandbox: when you supply your own
 #' connection, still open it in read-only mode where the backend supports it.
 #' Snowflake and Databricks sources snapshot the principal, active role, and
-#' namespace at creation, then reject catalog and governed execution after
-#' those values change. Authored and native semantic material is exposed only
-#' after a zero-row query succeeds for the current principal.
+#' namespace at creation, then reject catalog access and trusted calculations
+#' after those values change. Authored and native semantic material is exposed
+#' only after a zero-row query succeeds for the current principal.
 #'
 #' @return A `commons_data_source` object.
 #'

pkg-r/R/measures.R

@@ -19,9 +19,9 @@
 #' such as `string[]`. Without a declaration, commons infers the type from the
 #' default, falling back to `string`.
 #'
-#' Measure and helper source is visible in `run_r`; evaluating a measure's name
-#' there prints its definition. Function environments, connections, and
-#' credentials are not shared with that session.
+#' Measure and helper source is visible in the agent's R session; evaluating a
+#' measure's name there prints its definition. Function environments,
+#' connections, and credentials are not shared with that session.
 #'
 #' @section Measure arguments:
 #' A measure function can take two kinds of arguments:
@@ -130,16 +130,13 @@ expand_measures <- function(args, env = rlang::caller_env()) {
 #' supply.
 #'
 #' Two return types receive special display handling: ggplots and [gt::gt()]
-#' tables are shown directly to the user in the opened measure result. The model
-#' is told that the plot or table has already been shown, so it can interpret the
-#' result without repeating it.
+#' tables are shown directly to the user in the opened measure result.
 #'
 #' For full control over a result, `fn` can return an
 #' [ellmer::ContentToolResult]. Its `value` is sent to the model and its
-#' `extra$display` controls the shinychat display. When the display includes
-#' HTML, Markdown, or text, the model is told that the result is already visible
-#' to the user. An optional `extra$data` value is made available to `run_r` and
-#' removed from the result before it is returned to ellmer.
+#' `extra$display` controls the shinychat display. An optional `extra$data`
+#' value is made available in the agent's R session and removed from the result
+#' before it is returned to ellmer.
 #'
 #' @param name Measure name.
 #' @param description What the measure computes.

pkg-r/R/trajectory-review.R

@@ -34,12 +34,13 @@
 #' @section Transcript contents:
 #' The transcript uses the same commons and shinychat renderer as live
 #' conversations, preserving recorded messages and tool activity. Provenance
-#' markers are reconstructed from recorded provenance tags, but inline citations
-#' are not recreated. Generated review documents list the recorded citation
-#' decisions separately.
+#' markers are reconstructed from recorded provenance tags, but inline
+#' citations are not recreated. Generated review documents list the recorded
+#' citation decisions separately.
 #'
-#' Search-pool results are omitted because later tool calls record any selected
-#' measure; other tool results are limited to 50 lines or 20,000 characters.
+#' Trusted-calculation discovery results are omitted because later activity
+#' records any selected calculation; other tool results are limited to 50 lines
+#' or 20,000 characters.
 #'
 #' Trust filters use each answer's provenance tag exactly as [trajectory_read()]
 #' recorded it. Missing or conflicting records are omitted rather than inferred.

pkg-r/inst/skills/commons/references/data-dictionaries.md

@@ -35,7 +35,7 @@ Prioritize information the agent needs to interpret and query the source:
 - Column meanings, types, units, values, ranges, examples, and constraints.
 - Relationships demonstrated by schemas, trusted code, or verified exploration.
 
-Treat the dictionary as curated context, not a copy of the source schema. Be mindful of its total size and do not document every column by default. If a column entry would only repeat a name or type available from the live schema, omit it unless the agent needs that column as a dimension or `where` operand in `call_metrics`. Prioritize columns whose meaning, units, valid values, caveats, relationships, or analytical role are not evident from the schema.
+Treat the dictionary as curated context, not a copy of the source schema. Be mindful of its total size and do not document every column by default. If a column entry would only repeat a name or type available from the live schema, omit it unless the agent needs that column for grouping or filtering a trusted metric calculation. Prioritize columns whose meaning, units, valid values, caveats, relationships, or analytical role are not evident from the schema.
 
 Prefer the dictionary over `context_layer()` for information that belongs to a dataset, table, column, or relationship. Keep one authoritative copy of each fact and do not add claims that the available evidence does not support.
 
@@ -44,7 +44,7 @@ Prefer the dictionary over `context_layer()` for information that belongs to a d
 commons delivers dictionary content progressively:
 
 - Dataset-level descriptions and details are available in the system prompt, along with as many glossary terms as fit within its size cap. Remaining glossary terms are searchable and arrive on first touch when a table entry references them.
-- The first time a conversation uses a table, its description, details, documented columns, relationships, and governed definitions are delivered with the tool result. `describe_table` combines the dictionary entry with the live schema.
+- The first time a conversation uses a table, its description, details, documented columns, relationships, and governed definitions are delivered with the tool result. Table inspection combines the dictionary entry with the live schema.
 - Dictionary prose and governed definitions are indexed for context search. Column details remain first-touch content rather than a second searchable copy.
 
 Put broadly applicable source guidance at the dataset level and table-specific guidance on the table. This keeps ambient context small while making detailed information available when it becomes relevant.
@@ -73,7 +73,7 @@ tables:
 
 Each definition requires `name` and `expr`. `label`, `description`, and `details` are optional. commons validates the expression's inferred type and references, determines whether it is a filter, metric, or derived row value, and compiles it for the source's SQL backend. Document every referenced column and its type so the expression can be checked. Definition names must not shadow columns. Definitions on the same table may refer to one another by name, but references must not form a cycle.
 
-The agent can apply definitions as `{{name}}` tokens in SQL. When the same name exists on multiple tables in a query, qualify it as `{{table::name}}`. Metrics can also be called through `call_metrics`. Add a definition only when its business meaning and computation come from trusted existing material and the user confirms any consequential choice. Do not invent a calculation merely to complete the dictionary.
+The agent can apply definitions as `{{name}}` tokens in SQL. When the same name exists on multiple tables in a query, qualify it as `{{table::name}}`. Metrics can also be invoked as trusted calculations. Add a definition only when its business meaning and computation come from trusted existing material and the user confirms any consequential choice. Do not invent a calculation merely to complete the dictionary.
 
 ## Validate
 
@@ -83,4 +83,4 @@ Use data-dict validation where useful, then verify the commons integration:
 - Compare documented columns and types with the live schema.
 - Confirm relationships and definition expressions use real tables and columns.
 - Construct the data sources and agent to catch expression parsing, type, reference, source-mapping, and backend-compilation errors.
-- Exercise each definition against the live source with `run_sql` or `call_metrics`, as appropriate. Construction compiles but does not execute definitions, so representative queries are still needed to catch runtime and data-dependent errors.
+- Exercise each definition against the live source with custom SQL or a trusted metric calculation, as appropriate. Construction compiles but does not execute definitions, so representative queries are still needed to catch runtime and data-dependent errors.