Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ vignette.

<!-- Diagram source: Introduction to commons vignette. Update it there, then save the image. https://github.com/posit-dev/commons/blob/a29ac09c39c8edb99f2a9ea0ecc1836e6538bb25/vignettes/commons.Rmd#L76 -->

<img src="pkg-r/man/figures/README-trust-flow.png" alt="A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and produces a verified answer. The lower-trust path searches context and writes custom SQL or R, producing either a cited or untrusted answer." width="100%" />
<img src="pkg-r/man/figures/README-trust-flow.png" alt="Flow diagram. A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and ends with a green check-shield marker for the Verified answer outcome. The lower-trust path searches context and writes custom SQL or R, ending with either a blue quote-mark citation marker for the Cited outcome or a yellow exclamation marker for the Untrusted outcome." width="684" />

## Evaluation

Expand Down
3 changes: 2 additions & 1 deletion pkg-r/README.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,8 @@ For more information, see the [Introduction to commons](https://posit-dev.github

```{r trust-flow}
#| echo: false
#| fig-alt: "A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and produces a verified answer. The lower-trust path searches context and writes custom SQL or R, producing either a cited or untrusted answer."
#| fig-alt: "Flow diagram. A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and ends with a green check-shield marker for the Verified answer outcome. The lower-trust path searches context and writes custom SQL or R, ending with either a blue quote-mark citation marker for the Cited outcome or a yellow exclamation marker for the Untrusted outcome."
#| out-width: 684

knitr::include_graphics("man/figures/README-trust-flow.png")
```
Expand Down
2 changes: 1 addition & 1 deletion pkg-r/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ vignette.

<!-- Diagram source: Introduction to commons vignette. Update it there, then save the image. https://github.com/posit-dev/commons/blob/a29ac09c39c8edb99f2a9ea0ecc1836e6538bb25/vignettes/commons.Rmd#L76 -->

<img src="man/figures/README-trust-flow.png" alt="A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and produces a verified answer. The lower-trust path searches context and writes custom SQL or R, producing either a cited or untrusted answer." width="100%" />
<img src="man/figures/README-trust-flow.png" alt="Flow diagram. A question first searches trusted calculations. The high-trust path runs a relevant trusted calculation and ends with a green check-shield marker for the Verified answer outcome. The lower-trust path searches context and writes custom SQL or R, ending with either a blue quote-mark citation marker for the Cited outcome or a yellow exclamation marker for the Untrusted outcome." width="684" />

## Evaluation

Expand Down
Binary file modified pkg-r/man/figures/README-trust-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
56 changes: 35 additions & 21 deletions pkg-r/vignettes/commons.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,25 @@ knitr::opts_chunk$set(
comment = "#>",
eval = FALSE
)
knitr::knit_meta_add(list(commons:::commons_chat_dependency()))

documentation_marker <- function(
tag,
alt = commons:::provenance_display[[tag]]$label
) {
icon <- c(
A = "trusted-icon.svg",
B = "citation-mark.svg",
C = "warning-icon.svg"
)[[tag]]

htmltools::tags$img(
class = "commons-documentation-marker",
src = knitr::image_uri(
system.file("www", "commons-chat", "figs", icon, package = "commons")
),
alt = alt
)
}
```

This document explains the structure of a commons project and how to start building a commons agent.
Expand All @@ -37,15 +55,13 @@ The high-trust "happy path" occurs when the user asks a question that correspond

> How many total animals were observed at Oak Bluff?

The agent then searches for a trusted calculation that can answer the question. If it finds one, it runs that code and then reports the result with a `Verified answer` provenance marker.
The agent then searches for a trusted calculation that can answer the question. If it finds one, it runs that code and then reports the result with the green check-shield provenance marker for the `Verified answer` outcome.

> At Oak Bluff, 59 individual animals were observed across 5 species, based on 28 hours of survey effort. Note this reflects observed individuals during surveys, not necessarily a full census of every animal present at the site.
>
> `r commons:::commons_answer_pill("A")`
> At Oak Bluff, 59 individual animals were observed across 5 species, based on 28 hours of survey effort. Note this reflects observed individuals during surveys, not necessarily a full census of every animal present at the site. `r documentation_marker("A")`

Although the agent had to decide which trusted calculation to run, it did not have to decide _what code to write_, reducing degrees of freedom and allowing it to take advantage of pre-vetted code.

However, we also expect users to ask questions that stray from the "happy path." For those, the agent searches for additional context and writes custom code (either SQL or R). Answers from this path either include a verified citation or display an `Untrusted` provenance marker.
However, we also expect users to ask questions that stray from the "happy path." For those, the agent searches for additional context and writes custom code (either SQL or R). Answers from this path either include a blue quote-mark citation marker or display the yellow exclamation provenance marker for an `Untrusted` outcome.

## Working with the agent skill

Expand All @@ -69,14 +85,14 @@ file.copy(skill, ".claude/skills", recursive = TRUE)

## Trust flow

commons agents will use trusted calculations whenever possible. When the user asks a question, the agent first searches the semantic layer for a trusted calculation. If it finds one, it then calls that calculation and the resulting answer displays a green provenance marker labeled `Verified answer`.
commons agents will use trusted calculations whenever possible. When the user asks a question, the agent first searches the semantic layer for a trusted calculation. If it finds one, it then calls that calculation and the resulting answer displays the green check-shield provenance marker for the `Verified answer` outcome.

If a relevant trusted calculation is not found, the agent proceeds down the lower-trust path. It searches through the context for additional information, then uses that information to write custom SQL or R code to answer the user's question. These answers either include a verified citation in a footnote or display an `Untrusted` provenance marker.
If a relevant trusted calculation is not found, the agent proceeds down the lower-trust path. It searches through the context for additional information, then uses that information to write custom SQL or R code to answer the user's question. These answers either include blue quote-mark citation markers that open details about verified sources or display the yellow exclamation provenance marker for an `Untrusted` outcome.

<figure class="commons-flow commons-flow-trusted-calculations column-page">
<div class="commons-branch-flow"
role="img"
aria-label="Search trusted calculations, then follow either the high-trust path through a relevant calculation or the lower-trust path through context and custom SQL or R">
aria-label="Search trusted calculations, then follow either the high-trust path through a relevant calculation to a verified answer marker, or the lower-trust path through context and custom SQL or R to a citation or untrusted marker">
<div class="commons-branch-origin">
<span class="commons-flow-node">Search trusted<br>calculations</span>
</div>
Expand All @@ -92,7 +108,7 @@ If a relevant trusted calculation is not found, the agent proceeds down the lowe
<span class="commons-flow-arrow">&rarr;</span>
<span class="commons-flow-node">Run trusted<br>calculation</span>
<span class="commons-flow-arrow">&rarr;</span>
`r commons:::commons_answer_pill("A")`
`r documentation_marker("A")`
</div>
</div>
<div class="commons-trust-segment commons-trust-segment-lower">
Expand All @@ -105,29 +121,27 @@ If a relevant trusted calculation is not found, the agent proceeds down the lowe
<span class="commons-flow-node">Write<br>SQL/R</span>
<span class="commons-flow-arrow">&rarr;</span>
<span class="commons-flow-results">
<span class="commons-flow-answer">
Cited answer<sup class="commons-citation-ref">1</sup>
</span>
`r documentation_marker("B")`
<span class="commons-flow-or">or</span>
`r commons:::commons_answer_pill("C")`
`r documentation_marker("C")`
</span>
</div>
</div>
</div>
</div>
</figure>

The lower-trust path has two possible provenance outcomes. When the agent writes custom SQL or R, it can also include supporting text quoted from a trusted source. If commons verifies that the quoted text appears in that source, the provenance outcome is `Cited` and the verified citations are displayed as footnotes. If no citation verifies, the provenance outcome is `Untrusted` and the answer displays a provenance marker.
The lower-trust path has two possible provenance outcomes. When the agent writes custom SQL or R, it can also include supporting text quoted from a trusted source. If commons verifies that the quoted text appears in that source, the provenance outcome is `Cited` and the answer displays blue quote-mark citation markers that open the source details. If no citation verifies, the provenance outcome is `Untrusted` and the answer displays the yellow exclamation provenance marker.

The following table details the various ways each provenance outcome can occur:

| How the answer is produced | Provenance outcome |
|---|---|
| A trusted R [measure](#semantic-layer) | `r commons:::commons_answer_pill("A")` |
| A [data dictionary metric](#definitions), possibly grouped or filtered with [definitions](#data-dictionaries) | `r commons:::commons_answer_pill("A")` |
| A [Snowflake semantic-view or Databricks metric-view metric](#warehouse-semantic-layers) | `r commons:::commons_answer_pill("A")` |
| Custom SQL, including SQL that uses [data dictionary definitions](#definitions) | Cited or `r commons:::commons_answer_pill("C")` |
| Custom R | Cited or `r commons:::commons_answer_pill("C")` |
| A trusted R [measure](#semantic-layer) | Verified answer `r documentation_marker("A", alt = "")` |
| A [data dictionary metric](#definitions), possibly grouped or filtered with [definitions](#data-dictionaries) | Verified answer `r documentation_marker("A", alt = "")` |
| A [Snowflake semantic-view or Databricks metric-view metric](#warehouse-semantic-layers) | Verified answer `r documentation_marker("A", alt = "")` |
| Custom SQL, including SQL that uses [data dictionary definitions](#definitions) | Cited `r documentation_marker("B", alt = "")` or Untrusted `r documentation_marker("C", alt = "")` |
| Custom R | Cited `r documentation_marker("B", alt = "")` or Untrusted `r documentation_marker("C", alt = "")` |
| No data tool used (e.g., because the agent already had sufficient information or the question could not be answered from accessible information) | No provenance outcome |

The agent itself does not determine the provenance outcome. commons derives it deterministically from the agent's behavior.
Expand Down Expand Up @@ -198,7 +212,7 @@ Observed individuals reflect organisms recorded during surveys. They should not

### Warehouse semantic layers

If you have trusted metrics in [Snowflake semantic views](https://docs.snowflake.com/en/user-guide/views-semantic/overview) or [Databricks metric views](https://docs.databricks.com/aws/en/uc-semantics/metric-views), you can use those calculations directly with commons. It can also group or filter metrics by approved fields from the warehouse. Answers based on these warehouse-defined metrics have the `Verified answer` provenance outcome and display the corresponding provenance marker.
If you have trusted metrics in [Snowflake semantic views](https://docs.snowflake.com/en/user-guide/views-semantic/overview) or [Databricks metric views](https://docs.databricks.com/aws/en/uc-semantics/metric-views), you can use those calculations directly with commons. It can also group or filter metrics by approved fields from the warehouse. Answers based on these warehouse-defined metrics have the `Verified answer` provenance outcome and display the green check-shield provenance marker.

## Data sources

Expand Down
30 changes: 8 additions & 22 deletions pkg-r/vignettes/commons.css
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
.commons-answer-pill-icon {
.commons-documentation-marker {
background: transparent;
border: 0;
border-radius: 0;
height: 1.5em;
margin: 0;
padding: 0;
vertical-align: text-bottom;
width: 1.5em;
}

.commons-flow {
Expand All @@ -21,20 +24,12 @@

.commons-trust-segment-high {
background: #f8fcf9;
border-color: #cfeedd;
border-color: #a9d9bf;
}

.commons-trust-segment-lower {
background: #fffcf7;
border-color: #f2ddbb;
}

.commons-trust-segment-high .commons-trust-label {
color: #397957;
}

.commons-trust-segment-lower .commons-trust-label {
color: #7b5c27;
border-color: #dfc28a;
}

.commons-branch-flow {
Expand Down Expand Up @@ -78,25 +73,15 @@
.commons-flow-results {
align-items: center;
display: inline-flex;
gap: 0.3rem;
gap: 0.55rem;
white-space: nowrap;
}

.commons-flow-answer {
font-size: 0.9rem;
}

.commons-flow-or {
color: #666;
font-size: 0.8rem;
}

.commons-citation-ref {
color: #397f91;
font-size: 0.7em;
font-weight: 700;
}

.commons-flow-arrow {
color: #777;
font-size: 1.2em;
Expand All @@ -115,6 +100,7 @@
}

.commons-trust-label {
color: #4a4a4a;
font-size: 0.85rem;
font-weight: 600;
margin-bottom: 0.5rem;
Expand Down
Loading