diff --git a/README.md b/README.md index f5ff871..6648b7a 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ vignette. -A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and produces a verified answer. The lower-trust path searches context and writes custom SQL or R, producing either a cited or untrusted answer. +Flow diagram. A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and ends with a green check-shield marker for the Verified answer outcome. The lower-trust path searches context and writes custom SQL or R, ending with either a blue quote-mark citation marker for the Cited outcome or a yellow exclamation marker for the Untrusted outcome. ## Evaluation diff --git a/pkg-r/README.Rmd b/pkg-r/README.Rmd index 76ab4da..d8b3060 100644 --- a/pkg-r/README.Rmd +++ b/pkg-r/README.Rmd @@ -82,7 +82,8 @@ For more information, see the [Introduction to commons](https://posit-dev.github ```{r trust-flow} #| echo: false -#| fig-alt: "A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and produces a verified answer. The lower-trust path searches context and writes custom SQL or R, producing either a cited or untrusted answer." +#| fig-alt: "Flow diagram. A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and ends with a green check-shield marker for the Verified answer outcome. The lower-trust path searches context and writes custom SQL or R, ending with either a blue quote-mark citation marker for the Cited outcome or a yellow exclamation marker for the Untrusted outcome." +#| out-width: 684 knitr::include_graphics("man/figures/README-trust-flow.png") ``` diff --git a/pkg-r/README.md b/pkg-r/README.md index fa3b0ac..c1667dc 100644 --- a/pkg-r/README.md +++ b/pkg-r/README.md @@ -82,7 +82,7 @@ vignette. -A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and produces a verified answer. The lower-trust path searches context and writes custom SQL or R, producing either a cited or untrusted answer. +Flow diagram. A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and ends with a green check-shield marker for the Verified answer outcome. The lower-trust path searches context and writes custom SQL or R, ending with either a blue quote-mark citation marker for the Cited outcome or a yellow exclamation marker for the Untrusted outcome. ## Evaluation diff --git a/pkg-r/man/figures/README-trust-flow.png b/pkg-r/man/figures/README-trust-flow.png index 6c1e89c..83c9c8a 100644 Binary files a/pkg-r/man/figures/README-trust-flow.png and b/pkg-r/man/figures/README-trust-flow.png differ diff --git a/pkg-r/vignettes/commons.Rmd b/pkg-r/vignettes/commons.Rmd index 676ffc7..56bf6e2 100644 --- a/pkg-r/vignettes/commons.Rmd +++ b/pkg-r/vignettes/commons.Rmd @@ -18,7 +18,25 @@ knitr::opts_chunk$set( comment = "#>", eval = FALSE ) -knitr::knit_meta_add(list(commons:::commons_chat_dependency())) + +documentation_marker <- function( + tag, + alt = commons:::provenance_display[[tag]]$label +) { + icon <- c( + A = "trusted-icon.svg", + B = "citation-mark.svg", + C = "warning-icon.svg" + )[[tag]] + + htmltools::tags$img( + class = "commons-documentation-marker", + src = knitr::image_uri( + system.file("www", "commons-chat", "figs", icon, package = "commons") + ), + alt = alt + ) +} ``` This document explains the structure of a commons project and how to start building a commons agent. @@ -37,15 +55,13 @@ The high-trust "happy path" occurs when the user asks a question that correspond > How many total animals were observed at Oak Bluff? -The agent then searches for a trusted calculation that can answer the question. If it finds one, it runs that code and then reports the result with a `Verified answer` provenance marker. +The agent then searches for a trusted calculation that can answer the question. If it finds one, it runs that code and then reports the result with the green check-shield provenance marker for the `Verified answer` outcome. -> At Oak Bluff, 59 individual animals were observed across 5 species, based on 28 hours of survey effort. Note this reflects observed individuals during surveys, not necessarily a full census of every animal present at the site. -> -> `r commons:::commons_answer_pill("A")` +> At Oak Bluff, 59 individual animals were observed across 5 species, based on 28 hours of survey effort. Note this reflects observed individuals during surveys, not necessarily a full census of every animal present at the site. `r documentation_marker("A")` Although the agent had to decide which trusted calculation to run, it did not have to decide _what code to write_, reducing degrees of freedom and allowing it to take advantage of pre-vetted code. -However, we also expect users to ask questions that stray from the "happy path." For those, the agent searches for additional context and writes custom code (either SQL or R). Answers from this path either include a verified citation or display an `Untrusted` provenance marker. +However, we also expect users to ask questions that stray from the "happy path." For those, the agent searches for additional context and writes custom code (either SQL or R). Answers from this path either include a blue quote-mark citation marker or display the yellow exclamation provenance marker for an `Untrusted` outcome. ## Working with the agent skill @@ -69,14 +85,14 @@ file.copy(skill, ".claude/skills", recursive = TRUE) ## Trust flow -commons agents will use trusted calculations whenever possible. When the user asks a question, the agent first searches the semantic layer for a trusted calculation. If it finds one, it then calls that calculation and the resulting answer displays a green provenance marker labeled `Verified answer`. +commons agents will use trusted calculations whenever possible. When the user asks a question, the agent first searches the semantic layer for a trusted calculation. If it finds one, it then calls that calculation and the resulting answer displays the green check-shield provenance marker for the `Verified answer` outcome. -If a relevant trusted calculation is not found, the agent proceeds down the lower-trust path. It searches through the context for additional information, then uses that information to write custom SQL or R code to answer the user's question. These answers either include a verified citation in a footnote or display an `Untrusted` provenance marker. +If a relevant trusted calculation is not found, the agent proceeds down the lower-trust path. It searches through the context for additional information, then uses that information to write custom SQL or R code to answer the user's question. These answers either include blue quote-mark citation markers that open details about verified sources or display the yellow exclamation provenance marker for an `Untrusted` outcome.
@@ -105,11 +121,9 @@ If a relevant trusted calculation is not found, the agent proceeds down the lowe Write
SQL/R → - -Cited answer1 - +`r documentation_marker("B")` or -`r commons:::commons_answer_pill("C")` +`r documentation_marker("C")`
@@ -117,17 +131,17 @@ Cited answer1
-The lower-trust path has two possible provenance outcomes. When the agent writes custom SQL or R, it can also include supporting text quoted from a trusted source. If commons verifies that the quoted text appears in that source, the provenance outcome is `Cited` and the verified citations are displayed as footnotes. If no citation verifies, the provenance outcome is `Untrusted` and the answer displays a provenance marker. +The lower-trust path has two possible provenance outcomes. When the agent writes custom SQL or R, it can also include supporting text quoted from a trusted source. If commons verifies that the quoted text appears in that source, the provenance outcome is `Cited` and the answer displays blue quote-mark citation markers that open the source details. If no citation verifies, the provenance outcome is `Untrusted` and the answer displays the yellow exclamation provenance marker. The following table details the various ways each provenance outcome can occur: | How the answer is produced | Provenance outcome | |---|---| -| A trusted R [measure](#semantic-layer) | `r commons:::commons_answer_pill("A")` | -| A [data dictionary metric](#definitions), possibly grouped or filtered with [definitions](#data-dictionaries) | `r commons:::commons_answer_pill("A")` | -| A [Snowflake semantic-view or Databricks metric-view metric](#warehouse-semantic-layers) | `r commons:::commons_answer_pill("A")` | -| Custom SQL, including SQL that uses [data dictionary definitions](#definitions) | Cited or `r commons:::commons_answer_pill("C")` | -| Custom R | Cited or `r commons:::commons_answer_pill("C")` | +| A trusted R [measure](#semantic-layer) | Verified answer `r documentation_marker("A", alt = "")` | +| A [data dictionary metric](#definitions), possibly grouped or filtered with [definitions](#data-dictionaries) | Verified answer `r documentation_marker("A", alt = "")` | +| A [Snowflake semantic-view or Databricks metric-view metric](#warehouse-semantic-layers) | Verified answer `r documentation_marker("A", alt = "")` | +| Custom SQL, including SQL that uses [data dictionary definitions](#definitions) | Cited `r documentation_marker("B", alt = "")` or Untrusted `r documentation_marker("C", alt = "")` | +| Custom R | Cited `r documentation_marker("B", alt = "")` or Untrusted `r documentation_marker("C", alt = "")` | | No data tool used (e.g., because the agent already had sufficient information or the question could not be answered from accessible information) | No provenance outcome | The agent itself does not determine the provenance outcome. commons derives it deterministically from the agent's behavior. @@ -198,7 +212,7 @@ Observed individuals reflect organisms recorded during surveys. They should not ### Warehouse semantic layers -If you have trusted metrics in [Snowflake semantic views](https://docs.snowflake.com/en/user-guide/views-semantic/overview) or [Databricks metric views](https://docs.databricks.com/aws/en/uc-semantics/metric-views), you can use those calculations directly with commons. It can also group or filter metrics by approved fields from the warehouse. Answers based on these warehouse-defined metrics have the `Verified answer` provenance outcome and display the corresponding provenance marker. +If you have trusted metrics in [Snowflake semantic views](https://docs.snowflake.com/en/user-guide/views-semantic/overview) or [Databricks metric views](https://docs.databricks.com/aws/en/uc-semantics/metric-views), you can use those calculations directly with commons. It can also group or filter metrics by approved fields from the warehouse. Answers based on these warehouse-defined metrics have the `Verified answer` provenance outcome and display the green check-shield provenance marker. ## Data sources diff --git a/pkg-r/vignettes/commons.css b/pkg-r/vignettes/commons.css index bab7eb9..b23a2d7 100644 --- a/pkg-r/vignettes/commons.css +++ b/pkg-r/vignettes/commons.css @@ -1,9 +1,12 @@ -.commons-answer-pill-icon { +.commons-documentation-marker { background: transparent; border: 0; border-radius: 0; + height: 1.5em; margin: 0; padding: 0; + vertical-align: text-bottom; + width: 1.5em; } .commons-flow { @@ -21,20 +24,12 @@ .commons-trust-segment-high { background: #f8fcf9; - border-color: #cfeedd; + border-color: #a9d9bf; } .commons-trust-segment-lower { background: #fffcf7; - border-color: #f2ddbb; -} - -.commons-trust-segment-high .commons-trust-label { - color: #397957; -} - -.commons-trust-segment-lower .commons-trust-label { - color: #7b5c27; + border-color: #dfc28a; } .commons-branch-flow { @@ -78,25 +73,15 @@ .commons-flow-results { align-items: center; display: inline-flex; - gap: 0.3rem; + gap: 0.55rem; white-space: nowrap; } -.commons-flow-answer { - font-size: 0.9rem; -} - .commons-flow-or { color: #666; font-size: 0.8rem; } -.commons-citation-ref { - color: #397f91; - font-size: 0.7em; - font-weight: 700; -} - .commons-flow-arrow { color: #777; font-size: 1.2em; @@ -115,6 +100,7 @@ } .commons-trust-label { + color: #4a4a4a; font-size: 0.85rem; font-weight: 600; margin-bottom: 0.5rem;