From 27cd8e585fa030ec0bf0422d1a1671f6a21cc9ca Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Fri, 19 Jun 2026 21:11:21 -0700 Subject: [PATCH 1/9] docs: onboarding guide for federated private S3 backends Adds a "Connect a Private S3 Bucket" guide under Using Source: how Source serves a private bucket via OIDC federation (Source stores only a role ARN, no credentials), the federation contract (issuer / audience / subject), step-by-step IAM setup, and copy-paste CloudFormation + Terraform parameterized by connection id, bucket, and prefix. Addresses source-cooperative/source.coop#330. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/using-source/connect-private-s3.md | 293 ++++++++++++++++++++++++ sidebars.ts | 1 + 2 files changed, 294 insertions(+) create mode 100644 docs/using-source/connect-private-s3.md diff --git a/docs/using-source/connect-private-s3.md b/docs/using-source/connect-private-s3.md new file mode 100644 index 0000000..1ca6e54 --- /dev/null +++ b/docs/using-source/connect-private-s3.md @@ -0,0 +1,293 @@ +--- +title: Connect a Private S3 Bucket +id: connect-private-s3 +slug: /connect-private-s3 +sidebar_position: 4 +--- + +:::info Beta +Federated backends are rolling out. Source provisions your data connection and +gives you its **connection ID**; you create the AWS resources below and send the +**role ARN** back. To get started, contact [hello@source.coop](mailto:hello@source.coop). +::: + +Source can serve data from a **private** S3 bucket you own — without ever holding +your bucket credentials. Instead of handing Source long-lived access keys, you +create an IAM role that the Source data proxy assumes on demand via OpenID +Connect (OIDC) federation. Source stores only the role's ARN; there is no secret +at rest, on either side. + +## How it works + +1. The Source data proxy (`https://data.source.coop`) is an OIDC identity provider. +2. When a request needs your bucket, the proxy mints a short-lived OIDC token + whose **subject** identifies the Source connection, account, and product. +3. The proxy calls `sts:AssumeRoleWithWebIdentity` on your role. Your role's + **trust policy** decides whether to allow it; its **permission policy** caps + what it can read. +4. AWS returns temporary, prefix-scoped credentials that the proxy uses to read + your objects. Nothing long-lived is stored. + +You stay in control: the trust policy says *who* (which Source connection) may +assume the role, and the permission policy says *what* (which bucket and prefix) +they may read. + +## What you'll need + +- An AWS account containing the private S3 bucket. +- Your **connection ID** from Source (for example, `acme-private`). If you have + admin access to the connection in Source, it's shown on the connection's page + along with the exact subject pattern; otherwise Source provides it. +- The bucket name and the key prefix Source should read under. + +## The federation contract + +The proxy presents these values; your AWS resources must match them exactly. + +| Field | Value | +| --- | --- | +| OIDC provider URL (issuer) | `https://data.source.coop` | +| Audience (`aud`) | `source-coop-data-proxy` | +| Subject (`sub`) | `scv1:conn::/` | + +Because the subject is product-grained, your trust policy matches it with a +wildcard at the connection level: `scv1:conn::*`. This lets the +proxy assume the role for any product served by that one connection, and nothing +else. + +## Step 1 — Create the OIDC identity provider + +You need exactly **one** OIDC provider for `https://data.source.coop` per AWS +account, no matter how many buckets you connect. The CLI and console retrieve the +TLS thumbprint automatically: + +```bash +aws iam create-open-id-connect-provider \ + --url https://data.source.coop \ + --client-id-list source-coop-data-proxy +``` + +This returns the provider ARN +(`arn:aws:iam:::oidc-provider/data.source.coop`), which you'll +reference below. If the provider already exists, reuse it. + +## Step 2 — Create the IAM role + +Create a role with the **trust policy** below. Replace `` with your +AWS account ID and `` with your Source connection ID. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam:::oidc-provider/data.source.coop" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "data.source.coop:aud": "source-coop-data-proxy" + }, + "StringLike": { + "data.source.coop:sub": "scv1:conn::*" + } + } + } + ] +} +``` + +Attach a **permission policy** scoping read access to your bucket and prefix. +Replace `` and `` (leave `` empty to allow the whole +bucket). This is your blast-radius cap — Source can never read outside it. + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "ListBucketPrefix", + "Effect": "Allow", + "Action": "s3:ListBucket", + "Resource": "arn:aws:s3:::", + "Condition": { "StringLike": { "s3:prefix": "*" } } + }, + { + "Sid": "GetObjects", + "Effect": "Allow", + "Action": "s3:GetObject", + "Resource": "arn:aws:s3:::/*" + } + ] +} +``` + +## Step 3 — Give Source the role ARN + +Send the role ARN to Source. If you manage the connection yourself, paste it into +the connection's **Role ARN** field in the admin UI. Source fills in the rest; +no key or secret is ever exchanged. + +## Infrastructure as code + +Both templates create the IAM role and its policies, parameterized by connection +ID, bucket, and prefix. + +### CloudFormation + +This template creates the role and takes the OIDC provider ARN from Step 1 as a +parameter (so the provider — one per account — is managed separately). + +```yaml +AWSTemplateFormatVersion: "2010-09-09" +Description: IAM role for Source Cooperative federated read access to a private S3 bucket. + +Parameters: + ConnectionId: + Type: String + Description: Your Source data connection ID (provided by Source). + BucketName: + Type: String + Description: The private S3 bucket Source should read. + Prefix: + Type: String + Default: "" + Description: Key prefix Source may read under (leave blank for the whole bucket). + OidcProviderArn: + Type: String + Description: ARN of the data.source.coop OIDC provider in this account (see Step 1). + +Resources: + SourceFederatedRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: + Federated: !Ref OidcProviderArn + Action: sts:AssumeRoleWithWebIdentity + Condition: + StringEquals: + "data.source.coop:aud": "source-coop-data-proxy" + StringLike: + "data.source.coop:sub": !Sub "scv1:conn:${ConnectionId}:*" + Policies: + - PolicyName: SourceReadAccess + PolicyDocument: + Version: "2012-10-17" + Statement: + - Sid: ListBucketPrefix + Effect: Allow + Action: s3:ListBucket + Resource: !Sub "arn:aws:s3:::${BucketName}" + Condition: + StringLike: + "s3:prefix": !Sub "${Prefix}*" + - Sid: GetObjects + Effect: Allow + Action: s3:GetObject + Resource: !Sub "arn:aws:s3:::${BucketName}/${Prefix}*" + +Outputs: + RoleArn: + Description: Send this ARN to Source. + Value: !GetAtt SourceFederatedRole.Arn +``` + +### Terraform + +This creates the OIDC provider too (fetching the TLS thumbprint via the `tls` +provider). If the provider already exists in your account, drop the +`aws_iam_openid_connect_provider` resource and reference the existing one. + +```hcl +variable "connection_id" { type = string } +variable "bucket_name" { type = string } +variable "prefix" { + type = string + default = "" +} + +data "tls_certificate" "source" { + url = "https://data.source.coop" +} + +resource "aws_iam_openid_connect_provider" "source" { + url = "https://data.source.coop" + client_id_list = ["source-coop-data-proxy"] + thumbprint_list = [data.tls_certificate.source.certificates[0].sha1_fingerprint] +} + +data "aws_iam_policy_document" "trust" { + statement { + effect = "Allow" + actions = ["sts:AssumeRoleWithWebIdentity"] + principals { + type = "Federated" + identifiers = [aws_iam_openid_connect_provider.source.arn] + } + condition { + test = "StringEquals" + variable = "data.source.coop:aud" + values = ["source-coop-data-proxy"] + } + condition { + test = "StringLike" + variable = "data.source.coop:sub" + values = ["scv1:conn:${var.connection_id}:*"] + } + } +} + +data "aws_iam_policy_document" "read" { + statement { + sid = "ListBucketPrefix" + effect = "Allow" + actions = ["s3:ListBucket"] + resources = ["arn:aws:s3:::${var.bucket_name}"] + condition { + test = "StringLike" + variable = "s3:prefix" + values = ["${var.prefix}*"] + } + } + statement { + sid = "GetObjects" + effect = "Allow" + actions = ["s3:GetObject"] + resources = ["arn:aws:s3:::${var.bucket_name}/${var.prefix}*"] + } +} + +resource "aws_iam_role" "source_federated" { + name = "source-coop-${var.connection_id}" + assume_role_policy = data.aws_iam_policy_document.trust.json +} + +resource "aws_iam_role_policy" "read" { + name = "source-read-access" + role = aws_iam_role.source_federated.id + policy = data.aws_iam_policy_document.read.json +} + +output "role_arn" { + description = "Send this ARN to Source." + value = aws_iam_role.source_federated.arn +} +``` + +## Troubleshooting + +- **`AccessDenied` on assume:** the `data.source.coop:sub` or + `data.source.coop:aud` condition doesn't match. Confirm the connection ID in + your `sub` wildcard matches the one Source gave you, and that the audience is + exactly `source-coop-data-proxy`. +- **`AccessDenied` reading objects:** the permission policy's bucket or prefix + doesn't cover the requested keys. Check `` and ``. +- **Provider already exists:** an account can have only one OIDC provider per + URL. Reuse the existing `data.source.coop` provider rather than creating + another. diff --git a/sidebars.ts b/sidebars.ts index de6cd95..007ebfe 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -31,6 +31,7 @@ const sidebars: SidebarsConfig = { 'using-source/create-a-data-product', 'using-source/data-upload', 'using-source/data-proxy', + 'using-source/connect-private-s3', ], }, { From 7a1eabfaee20ef1d8778a060bc5b99b462344d4d Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 20 Jul 2026 12:54:09 -0700 Subject: [PATCH 2/9] docs: reframe federated S3 guide as Bring Your Own Bucket Generalize the guide to cover public or private buckets, lead with the beta/staff-enablement note, and split into "add the backend" then "set up private access". Move CloudFormation/Terraform into details tags. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...private-s3.md => bring-your-own-bucket.md} | 99 ++++++++++++------- sidebars.ts | 2 +- 2 files changed, 66 insertions(+), 35 deletions(-) rename docs/using-source/{connect-private-s3.md => bring-your-own-bucket.md} (75%) diff --git a/docs/using-source/connect-private-s3.md b/docs/using-source/bring-your-own-bucket.md similarity index 75% rename from docs/using-source/connect-private-s3.md rename to docs/using-source/bring-your-own-bucket.md index 1ca6e54..60393f2 100644 --- a/docs/using-source/connect-private-s3.md +++ b/docs/using-source/bring-your-own-bucket.md @@ -1,24 +1,57 @@ --- -title: Connect a Private S3 Bucket -id: connect-private-s3 -slug: /connect-private-s3 +title: Bring Your Own Bucket +id: bring-your-own-bucket +slug: /bring-your-own-bucket sidebar_position: 4 --- :::info Beta -Federated backends are rolling out. Source provisions your data connection and -gives you its **connection ID**; you create the AWS resources below and send the -**role ARN** back. To get started, contact [hello@source.coop](mailto:hello@source.coop). +Bring Your Own Bucket (BYOB) is a beta feature that Source staff must enable for +your account. To get started, contact [hello@source.coop](mailto:hello@source.coop). ::: -Source can serve data from a **private** S3 bucket you own — without ever holding -your bucket credentials. Instead of handing Source long-lived access keys, you -create an IAM role that the Source data proxy assumes on demand via OpenID -Connect (OIDC) federation. Source stores only the role's ARN; there is no secret -at rest, on either side. +Normally Source stores your data in buckets it manages. With **Bring Your Own +Bucket (BYOB)**, Source instead serves data from an S3 bucket **you own**. The +bucket can be: + +- **Public** — anyone can already read it. Source just points at it. +- **Private** — Source reads it on demand through OIDC federation, without ever + holding your bucket credentials. + +Either way you keep control of the storage: the data lives in your account, on +your terms, and Source serves it through the [data proxy](/data-proxy) like any +other product. ## How it works +Source provisions a **data connection** for your bucket and gives you its +**connection ID**. From there: + +- For a **public** bucket, that's it — Source reads it directly. +- For a **private** bucket, you create an IAM role that the Source data proxy + assumes on demand. Source stores only the role's ARN; there is no secret at + rest, on either side. + +## Step 1 — Ask Source to add the backend + +Contact [hello@source.coop](mailto:hello@source.coop) with: + +- The **bucket name** and its **AWS region**. +- The **key prefix** Source should read under (or the whole bucket). +- Whether the bucket is **public** or **private**. + +Source enables BYOB for your account, provisions the connection, and gives you a +**connection ID** (for example, `acme-data`). + +If the bucket is **public**, you're done — Source can serve it now. + +If the bucket is **private**, continue to Step 2 to grant federated read access. + +## Setting up private access + +For a private bucket, Source reads your objects through OpenID Connect (OIDC) +federation: + 1. The Source data proxy (`https://data.source.coop`) is an OIDC identity provider. 2. When a request needs your bucket, the proxy mints a short-lived OIDC token whose **subject** identifies the Source connection, account, and product. @@ -32,15 +65,7 @@ You stay in control: the trust policy says *who* (which Source connection) may assume the role, and the permission policy says *what* (which bucket and prefix) they may read. -## What you'll need - -- An AWS account containing the private S3 bucket. -- Your **connection ID** from Source (for example, `acme-private`). If you have - admin access to the connection in Source, it's shown on the connection's page - along with the exact subject pattern; otherwise Source provides it. -- The bucket name and the key prefix Source should read under. - -## The federation contract +### The federation contract The proxy presents these values; your AWS resources must match them exactly. @@ -55,7 +80,7 @@ wildcard at the connection level: `scv1:conn::*`. This lets the proxy assume the role for any product served by that one connection, and nothing else. -## Step 1 — Create the OIDC identity provider +### Step 2 — Create the OIDC identity provider You need exactly **one** OIDC provider for `https://data.source.coop` per AWS account, no matter how many buckets you connect. The CLI and console retrieve the @@ -71,7 +96,7 @@ This returns the provider ARN (`arn:aws:iam:::oidc-provider/data.source.coop`), which you'll reference below. If the provider already exists, reuse it. -## Step 2 — Create the IAM role +### Step 3 — Create the IAM role Create a role with the **trust policy** below. Replace `` with your AWS account ID and `` with your Source connection ID. @@ -124,20 +149,21 @@ bucket). This is your blast-radius cap — Source can never read outside it. } ``` -## Step 3 — Give Source the role ARN +### Step 4 — Give Source the role ARN -Send the role ARN to Source. If you manage the connection yourself, paste it into -the connection's **Role ARN** field in the admin UI. Source fills in the rest; -no key or secret is ever exchanged. +Send the role ARN back to Source. If you manage the connection yourself, paste it +into the connection's **Role ARN** field in the admin UI. Source fills in the +rest; no key or secret is ever exchanged. -## Infrastructure as code +### Infrastructure as code -Both templates create the IAM role and its policies, parameterized by connection -ID, bucket, and prefix. +Prefer to manage the IAM resources declaratively? The templates below create the +role and its policies, parameterized by connection ID, bucket, and prefix. -### CloudFormation +
+CloudFormation -This template creates the role and takes the OIDC provider ARN from Step 1 as a +This template creates the role and takes the OIDC provider ARN from Step 2 as a parameter (so the provider — one per account — is managed separately). ```yaml @@ -157,7 +183,7 @@ Parameters: Description: Key prefix Source may read under (leave blank for the whole bucket). OidcProviderArn: Type: String - Description: ARN of the data.source.coop OIDC provider in this account (see Step 1). + Description: ARN of the data.source.coop OIDC provider in this account (see Step 2). Resources: SourceFederatedRole: @@ -198,7 +224,10 @@ Outputs: Value: !GetAtt SourceFederatedRole.Arn ``` -### Terraform +
+ +
+Terraform This creates the OIDC provider too (fetching the TLS thumbprint via the `tls` provider). If the provider already exists in your account, drop the @@ -280,7 +309,9 @@ output "role_arn" { } ``` -## Troubleshooting +
+ +### Troubleshooting - **`AccessDenied` on assume:** the `data.source.coop:sub` or `data.source.coop:aud` condition doesn't match. Confirm the connection ID in diff --git a/sidebars.ts b/sidebars.ts index 007ebfe..15ebf98 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -31,7 +31,7 @@ const sidebars: SidebarsConfig = { 'using-source/create-a-data-product', 'using-source/data-upload', 'using-source/data-proxy', - 'using-source/connect-private-s3', + 'using-source/bring-your-own-bucket', ], }, { From 398dd5cf4d730c19c50ebf9ba871bcaa0e4bd5b0 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 20 Jul 2026 13:04:29 -0700 Subject: [PATCH 3/9] docs: move Bring Your Own Bucket above the data proxy in the sidebar Co-Authored-By: Claude Opus 4.8 (1M context) --- sidebars.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sidebars.ts b/sidebars.ts index 15ebf98..9399719 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -30,8 +30,8 @@ const sidebars: SidebarsConfig = { 'using-source/create-an-account', 'using-source/create-a-data-product', 'using-source/data-upload', - 'using-source/data-proxy', 'using-source/bring-your-own-bucket', + 'using-source/data-proxy', ], }, { From 3138a3ed1ac32817a240417e9a1add743564aaf2 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 20 Jul 2026 13:06:22 -0700 Subject: [PATCH 4/9] docs: note Bring Your Own Bucket and data connection in Create a Data Product Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/using-source/create-a-data-product.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/using-source/create-a-data-product.md b/docs/using-source/create-a-data-product.md index 4daa2e9..fd084d8 100644 --- a/docs/using-source/create-a-data-product.md +++ b/docs/using-source/create-a-data-product.md @@ -17,6 +17,9 @@ Data products can be owned by an organization or an individual. You will see a d - **Description**: Optional; maximum 500 characters. Use it for a short overview; put detailed documentation in the README. - **Visibility**: New data products are created **Unlisted** (not shown in search). When ready to publish, open the data product page, click the gear icon, and set the state to **Listed**. - **Tags**: Comma-separated, up to 20 tags. They help others discover your data. +- **Data Connection**: Choose where the product's data lives. By default it uses Source-managed storage. If you've set up [Bring Your Own Bucket](/bring-your-own-bucket), select the data connection for your bucket instead. + +You can bring an existing S3 bucket you own rather than storing data on Source. This is a beta feature—see [Bring Your Own Bucket](/bring-your-own-bucket) to set up a data connection before creating the product, then select it here. ## Editing a data product From 51f77c96d83c19ab6d88a82cd3eb2be21e398942 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 20 Jul 2026 13:06:55 -0700 Subject: [PATCH 5/9] docs: recommend us-west-2 region for BYOB buckets Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/using-source/bring-your-own-bucket.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/using-source/bring-your-own-bucket.md b/docs/using-source/bring-your-own-bucket.md index 60393f2..68590c4 100644 --- a/docs/using-source/bring-your-own-bucket.md +++ b/docs/using-source/bring-your-own-bucket.md @@ -36,7 +36,9 @@ Source provisions a **data connection** for your bucket and gives you its Contact [hello@source.coop](mailto:hello@source.coop) with: -- The **bucket name** and its **AWS region**. +- The **bucket name** and its **AWS region**. We informally recommend + `us-west-2`, where most Source data lives—colocating keeps access fast and + avoids cross-region transfer costs. - The **key prefix** Source should read under (or the whole bucket). - Whether the bucket is **public** or **private**. From 7b15c9dc6d09e89ae616495dd04d74675db2b476 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 20 Jul 2026 13:08:01 -0700 Subject: [PATCH 6/9] docs: recommend us-west-2 when selecting a data connection Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/using-source/bring-your-own-bucket.md | 4 +--- docs/using-source/create-a-data-product.md | 2 +- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/using-source/bring-your-own-bucket.md b/docs/using-source/bring-your-own-bucket.md index 68590c4..60393f2 100644 --- a/docs/using-source/bring-your-own-bucket.md +++ b/docs/using-source/bring-your-own-bucket.md @@ -36,9 +36,7 @@ Source provisions a **data connection** for your bucket and gives you its Contact [hello@source.coop](mailto:hello@source.coop) with: -- The **bucket name** and its **AWS region**. We informally recommend - `us-west-2`, where most Source data lives—colocating keeps access fast and - avoids cross-region transfer costs. +- The **bucket name** and its **AWS region**. - The **key prefix** Source should read under (or the whole bucket). - Whether the bucket is **public** or **private**. diff --git a/docs/using-source/create-a-data-product.md b/docs/using-source/create-a-data-product.md index fd084d8..2f87fd2 100644 --- a/docs/using-source/create-a-data-product.md +++ b/docs/using-source/create-a-data-product.md @@ -17,7 +17,7 @@ Data products can be owned by an organization or an individual. You will see a d - **Description**: Optional; maximum 500 characters. Use it for a short overview; put detailed documentation in the README. - **Visibility**: New data products are created **Unlisted** (not shown in search). When ready to publish, open the data product page, click the gear icon, and set the state to **Listed**. - **Tags**: Comma-separated, up to 20 tags. They help others discover your data. -- **Data Connection**: Choose where the product's data lives. By default it uses Source-managed storage. If you've set up [Bring Your Own Bucket](/bring-your-own-bucket), select the data connection for your bucket instead. +- **Data Connection**: Choose where the product's data lives. By default it uses Source-managed storage. If you've set up [Bring Your Own Bucket](/bring-your-own-bucket), select the data connection for your bucket instead. If you don't have a strong preference, choose a connection in the `us-west-2` region—this is where most Source data lives, and colocating keeps access fast and avoids cross-region transfer costs. You can bring an existing S3 bucket you own rather than storing data on Source. This is a beta feature—see [Bring Your Own Bucket](/bring-your-own-bucket) to set up a data connection before creating the product, then select it here. From 26705fece737330fbd0fc180203673a67c5c82c6 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 20 Jul 2026 13:31:19 -0700 Subject: [PATCH 7/9] docs: use self-service Data Connection form for adding a BYOB backend Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/using-source/bring-your-own-bucket.md | 64 +++++++++++++++------- 1 file changed, 44 insertions(+), 20 deletions(-) diff --git a/docs/using-source/bring-your-own-bucket.md b/docs/using-source/bring-your-own-bucket.md index 60393f2..9aef21a 100644 --- a/docs/using-source/bring-your-own-bucket.md +++ b/docs/using-source/bring-your-own-bucket.md @@ -24,26 +24,46 @@ other product. ## How it works -Source provisions a **data connection** for your bucket and gives you its -**connection ID**. From there: - -- For a **public** bucket, that's it — Source reads it directly. -- For a **private** bucket, you create an IAM role that the Source data proxy - assumes on demand. Source stores only the role's ARN; there is no secret at - rest, on either side. - -## Step 1 — Ask Source to add the backend - -Contact [hello@source.coop](mailto:hello@source.coop) with: - -- The **bucket name** and its **AWS region**. -- The **key prefix** Source should read under (or the whole bucket). -- Whether the bucket is **public** or **private**. - -Source enables BYOB for your account, provisions the connection, and gives you a -**connection ID** (for example, `acme-data`). - -If the bucket is **public**, you're done — Source can serve it now. +Once Source enables BYOB for your account, you create a **data connection** that +describes your bucket. Products can then mirror to it instead of Source-managed +storage. From there: + +- For a **public** bucket, that's it — Source reads it directly (unsigned). +- For a **private** bucket, you also create an IAM role that the Source data + proxy assumes on demand. Source stores only the role's ARN; there is no secret + at rest, on either side. + +## Step 1 — Create a data connection + +In your account or organization admin, open **Data Connections** and choose +**Create Data Connection**. Fill in the form: + +- **Connection ID**: lowercase letters, numbers, and hyphens. It's stored as + `--` and **cannot be changed after creation**. +- **Name**: a human-readable label shown in admin lists and the product mirror + picker. +- **Prefix Template**: the object-key prefix each product gets within the bucket. + `{{repository.account_id}}` and `{{repository.repository_id}}` are substituted + when a product attaches. The default + (`{{repository.account_id}}/{{repository.repository_id}}/`) is usually right. +- **Read Only**: check this to allow browse/download only. **Required for a + public (unsigned) connection.** +- **Allowed Visibilities**: product visibilities permitted to use this connection. +- **Provider**: `S3 / S3-compatible (R2, MinIO)`. +- **Bucket**: your bucket name. +- **Base Prefix**: optional shared root folder prepended to every object path; + leave blank for the bucket root. +- **Region**: the bucket's AWS region (use `auto` for S3-compatible backends + like Cloudflare R2). +- **Endpoint**: a custom S3-compatible endpoint for non-AWS backends; leave blank + for AWS S3. +- **Authentication Type**: how the data proxy reaches your bucket. + - **Public bucket** → **None (unsigned)** (and check **Read Only**). + - **Private bucket** → the role-based/federated option, then set up the IAM + role in Step 2. + +Click **Create Connection**. If the bucket is **public**, you're done — attach +the connection when you [create a data product](/create-a-data-product). If the bucket is **private**, continue to Step 2 to grant federated read access. @@ -75,6 +95,10 @@ The proxy presents these values; your AWS resources must match them exactly. | Audience (`aud`) | `source-coop-data-proxy` | | Subject (`sub`) | `scv1:conn::/` | +Here `` is the stored ID of the connection you created in Step 1 +(the full `--` value). The connection's page in the admin shows its +exact `sub` pattern. + Because the subject is product-grained, your trust policy matches it with a wildcard at the connection level: `scv1:conn::*`. This lets the proxy assume the role for any product served by that one connection, and nothing From cebe14550ad2b533e66959cfa65e6e10d20408f9 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 20 Jul 2026 13:40:30 -0700 Subject: [PATCH 8/9] docs: recommend blank Prefix Template for custom BYOB connections Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/using-source/bring-your-own-bucket.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/using-source/bring-your-own-bucket.md b/docs/using-source/bring-your-own-bucket.md index 9aef21a..9f2296b 100644 --- a/docs/using-source/bring-your-own-bucket.md +++ b/docs/using-source/bring-your-own-bucket.md @@ -44,8 +44,8 @@ In your account or organization admin, open **Data Connections** and choose picker. - **Prefix Template**: the object-key prefix each product gets within the bucket. `{{repository.account_id}}` and `{{repository.repository_id}}` are substituted - when a product attaches. The default - (`{{repository.account_id}}/{{repository.repository_id}}/`) is usually right. + when a product attaches. For a custom bucket, leave this **blank** so products + read from the paths already in your bucket rather than a Source-style prefix. - **Read Only**: check this to allow browse/download only. **Required for a public (unsigned) connection.** - **Allowed Visibilities**: product visibilities permitted to use this connection. From 5685def1f53bd3b4650f4121b1884d4feb910e75 Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Mon, 20 Jul 2026 15:54:37 -0700 Subject: [PATCH 9/9] docs: trim Data Connection form notes to the non-obvious fields Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/using-source/bring-your-own-bucket.md | 33 ++++++++-------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/docs/using-source/bring-your-own-bucket.md b/docs/using-source/bring-your-own-bucket.md index 9f2296b..dc43154 100644 --- a/docs/using-source/bring-your-own-bucket.md +++ b/docs/using-source/bring-your-own-bucket.md @@ -36,27 +36,18 @@ storage. From there: ## Step 1 — Create a data connection In your account or organization admin, open **Data Connections** and choose -**Create Data Connection**. Fill in the form: - -- **Connection ID**: lowercase letters, numbers, and hyphens. It's stored as - `--` and **cannot be changed after creation**. -- **Name**: a human-readable label shown in admin lists and the product mirror - picker. -- **Prefix Template**: the object-key prefix each product gets within the bucket. - `{{repository.account_id}}` and `{{repository.repository_id}}` are substituted - when a product attaches. For a custom bucket, leave this **blank** so products - read from the paths already in your bucket rather than a Source-style prefix. -- **Read Only**: check this to allow browse/download only. **Required for a - public (unsigned) connection.** -- **Allowed Visibilities**: product visibilities permitted to use this connection. -- **Provider**: `S3 / S3-compatible (R2, MinIO)`. -- **Bucket**: your bucket name. -- **Base Prefix**: optional shared root folder prepended to every object path; - leave blank for the bucket root. -- **Region**: the bucket's AWS region (use `auto` for S3-compatible backends - like Cloudflare R2). -- **Endpoint**: a custom S3-compatible endpoint for non-AWS backends; leave blank - for AWS S3. +**Create Data Connection**. Most fields are self-explanatory; a few are worth +calling out: + +- **Read Only**: allows browse/download only, blocking writes through the + connection. **Required for a public (unsigned) connection.** +- **Base Prefix**: an optional shared root folder prepended to every object path + (leave blank for the bucket root). Individual products can read from a child + prefix under this via the **Prefix Template**, which substitutes + `{{repository.account_id}}` and `{{repository.repository_id}}` when a product + attaches. +- **Allowed Visibilities**: the product visibilities (public, unlisted, + restricted) permitted to use this connection. - **Authentication Type**: how the data proxy reaches your bucket. - **Public bucket** → **None (unsigned)** (and check **Read Only**). - **Private bucket** → the role-based/federated option, then set up the IAM