From e6d9afcad93a90a265f0e15273cf130ba23636bc Mon Sep 17 00:00:00 2001 From: Anthony Lukach Date: Fri, 25 Sep 2026 14:11:48 -0700 Subject: [PATCH] docs: add an automated access guide for service accounts Adds Automated Access (docs/using-source/automated-access.md), the guide for software that reaches Source Cooperative with nobody at the keyboard: what a service account is; creating one and granting it products; issuing an API key and the five variables the AWS CLI or an SDK needs to exchange it at the data proxy; the refusal users see and the request id to quote; keeping the key out of URLs and debug output; rotation and editable expiry; tools that keep their first credentials; GDAL; revocation, the emergency stop and revoking a found key; and GitHub Actions as coming soon. The page is added to sidebars.ts. Upload Your Data's Option 3 walked through trusting an IAM role in the uploader's own AWS account to write straight to Source's bucket. Service accounts supersede it, so the section is now a pointer to the new page, as are the two references to it earlier on that page and the "contact us" line for automated access. Access Data gains one sentence pointing unattended software at service accounts. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01R1eiTse4416N6uTgAy4Ddd --- docs/using-source/automated-access.md | 265 ++++++++++++++++++++++++++ docs/using-source/data-proxy.md | 2 + docs/using-source/data-upload.md | 219 +-------------------- sidebars.ts | 1 + 4 files changed, 275 insertions(+), 212 deletions(-) create mode 100644 docs/using-source/automated-access.md diff --git a/docs/using-source/automated-access.md b/docs/using-source/automated-access.md new file mode 100644 index 0000000..d8d0fe2 --- /dev/null +++ b/docs/using-source/automated-access.md @@ -0,0 +1,265 @@ +--- +title: Automated Access +id: automated-access +slug: /automated-access +--- + +Some software reaches Source Cooperative with nobody at the keyboard: a nightly +sync, a publishing pipeline, an instrument that uploads its readings. Give it a +**service account** and an **API key**. The AWS CLI and SDKs exchange the key +for short-lived credentials at the [data proxy](/data-proxy) and renew them on +their own, so nothing else has to run on the machine. + +## What a service account is + +A service account is a login for software. It belongs to one account, yours or +an organization's, and whoever manages that account manages it. + +- **It has its own access.** You grant it products one at a time, to read or to + read and write. It can reach only products its owner owns, and it never + inherits your access or anyone else's. +- **There's no person behind it.** It has no profile, can't sign in to the + website, and can't create products or manage people. +- **You can stop it without touching a person.** Revoke its keys, or disable or + delete it, and nobody's own account changes. + +An organization can own service accounts, so a pipeline doesn't depend on the +account of whoever set it up. + +A service account's ID is its owner's ID, two hyphens, and a name of its own, +such as `your-org--nightly-sync`. It can't be changed. + +## Create a service account + +1. Open the profile page of the account that will own it, yours or an + organization's, click the gear icon, and choose **Service Accounts**. +2. Click **New service account**. +3. Under **Who it is**, enter a **Name**, such as `Nightly Sync`. The **Account + ID** is made from the name; click **Edit** beside it to choose another. +4. Leave **How software signs in** empty. It is for GitHub Actions, which + [isn't available yet](#github-actions). +5. Under **What it can reach**, click **Grant a product**, choose a product and + **Read** or **Read and write**, and click the check mark. Repeat for each + product the job needs, and no more: a job that only downloads needs **Read**. +6. Click **Create service account**. + +You land on the service account's page. You can change what it reaches at any +time under **Can reach**; each change is saved as you make it. + +## API keys + +An API key lets software on your own server, VM, cluster or instrument sign in +as the service account. + +### Issue a key + +1. On the service account's page, under **API keys**, click **Issue an API + key**. +2. Enter a **Label** that says where the key will live, such as `HPC cron job`, + so you know which key to revoke later. +3. Choose when it **Expires**: in 30 days, 90 days or a year, or never (until + you revoke it). +4. Click **Issue key**. + +The key is shown once. Copy it now: Source stores only a hash of it, so nobody +can show it to you again. If you lose it, issue another and revoke the lost one. +A key starts with `sck_`. + +You can't issue a key to a disabled service account. + +### Set up the machine + +Save the key in a file that only the job's user can read. A trailing newline is +fine. + +```bash +mkdir -p -m 700 ~/.source-coop +cat > ~/.source-coop/nightly-sync.key # paste the key, press Enter, then Ctrl-D +chmod 600 ~/.source-coop/nightly-sync.key +``` + +Then set five environment variables wherever the job runs. The dialog that +showed you the key lists them too, filled in for your service account apart from +the key file's path: + +```bash +export AWS_ROLE_ARN=arn:aws:iam::your-org--nightly-sync:role/FullAccess +export AWS_WEB_IDENTITY_TOKEN_FILE=$HOME/.source-coop/nightly-sync.key +export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts +export AWS_ENDPOINT_URL_S3=https://data.source.coop +export AWS_REGION=us-west-2 +``` + +| Variable | What it's for | +| --- | --- | +| `AWS_ROLE_ARN` | How much the credentials may do. `FullAccess` is everything the service account may do; `ReadOnly` is reads only. The value has the shape AWS tools expect, with the service account's ID where an AWS account number would be. | +| `AWS_WEB_IDENTITY_TOKEN_FILE` | The file that holds the key, as an absolute path. | +| `AWS_ENDPOINT_URL_STS` | Where to exchange the key for credentials: the data proxy. | +| `AWS_ENDPOINT_URL_S3` | Where to send S3 requests: the data proxy. | +| `AWS_REGION` | Required by S3 clients. It doesn't say where your data is stored. | + +That's all. The AWS CLI and SDKs read the key from the file, exchange it at the +data proxy for credentials that last an hour, and exchange it again before those +run out. Nothing else runs on the machine, and the key file never changes. + +The two endpoint variables need **AWS CLI 2.13** or later, or **boto3 1.28** +(**botocore 1.31**) or later. Older releases ignore them and send the key to AWS +instead, which refuses it. + +With the AWS CLI: + +```bash +aws --version # aws-cli/2.13.0 or later +aws s3 ls s3://your-org/your-product/ +aws s3 sync ./outgoing s3://your-org/your-product/outgoing/ +``` + +With boto3: + +```python +import boto3 # boto3 1.28 (botocore 1.31) or later + +# No keys, endpoint or region here: boto3 reads the five variables. +s3 = boto3.client("s3") +s3.upload_file("mydata.csv", "your-org", "your-product/mydata.csv") +``` + +Other AWS SDKs work the same way, provided their web identity credential +provider reads `AWS_ENDPOINT_URL_STS`. + +A cron job or a system service doesn't read your shell's profile, so set the +variables in its own environment. + +### If the key is refused + +A key that has been revoked or has expired, a key whose service account is +disabled, and a value that isn't a Source Cooperative key at all are all refused +with the same error: + +```text +An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: API key was not accepted (request id 8f3a1c2b9d4e5f60-SEA) +``` + +The service account's page shows whether it is disabled, and each key's row +shows whether the key has been revoked or has expired. If none of those explains +it, email [hello@source.coop](mailto:hello@source.coop) and quote the request +id: it lets us find the reason in our logs. + +### Keep the key secret + +- Treat the key as a password for the service account. Keep it out of source + control, scripts and command lines, and give it to the job as a file. +- `aws --debug` prints the key, because it logs the request that carries it. + Don't share debug output from a machine that has a key set up. +- A key goes in a file or in the body of a request, never in a URL, because URLs + end up in logs. The data proxy refuses a key sent in a URL, with + `API key must be sent in the request body, not the URL`. If that ever happens, + replace the key. + +### Rotate a key, or change when it expires + +A service account can have several keys at once, so you can replace one without +stopping the job: issue a new key, deploy it, and revoke the old one once +nothing uses it. Each key's row shows when it was last used. + +To change when a key expires, click **Change expiry** on its row: later, for a +job that runs longer than planned, or sooner, during an incident. The new expiry +counts from today. + +## Tools that keep their first credentials + +The AWS CLI and SDKs renew credentials on their own. Tools that manage +credentials themselves often don't: given a fixed set of credentials +(`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN`, or the +same values in their own settings), they read it once and use it until they +exit. GDAL's `/vsis3/`, DuckDB and rclone are common examples; DuckDB reads a +secret's credentials when you run `CREATE SECRET`. + +Credentials from the data proxy last an hour, so with these tools a transfer +that runs longer than that fails partway through with an `ExpiredToken` (HTTP +403) error, although the key is still good. To avoid it: + +- Split long transfers into runs that each finish within the hour, and give + each run fresh credentials: start a new process, or in DuckDB, replace the + secret (`CREATE OR REPLACE SECRET`) before each batch. +- Do large copies with the AWS CLI (`aws s3 cp`, `aws s3 sync`) or an AWS SDK, + which renew credentials mid-transfer. +- Where a tool can run a credential helper each time its credentials run out, + use one. GDAL can; see the next section. + +## GDAL + +GDAL's `/vsis3/` can't use an API key directly. Given `AWS_ROLE_ARN` and +`AWS_WEB_IDENTITY_TOKEN_FILE` and no other credentials, GDAL 3.6 and later do +the exchange themselves, in a way that can't work for a key: + +- They don't read `AWS_ENDPOINT_URL_STS`, so they send the key to AWS, not to + the data proxy. +- They put the key in the URL of the request. The data proxy refuses a key in a + URL, and a key sent to AWS in a URL should be treated as exposed. + +So where GDAL runs with the five variables set, also set +`CPL_AWS_WEB_IDENTITY_ENABLE=NO` to stop it trying. If GDAL has already run with +them, replace the key. + +:::info Coming soon + +The Source CLI will be able to exchange an API key without a browser +([source-coop-cli#17](https://github.com/source-cooperative/source-coop-cli/issues/17)). +GDAL 3.12 and later can then get credentials from it as a `credential_process`, +and renew them the same way. This page will show the setup once that release is +out. + +::: + +## Revoke a key, or stop a service account + +| To | Do this | What happens | +| --- | --- | --- | +| Stop one key | Click **Revoke** on the key's row. | New exchanges with the key are refused within about a minute. | +| Stop everything the service account does | Click **Disable** under **Danger zone**. | New exchanges with any of its keys are refused within about a minute. Writes stop within about a minute, and reads of restricted products within about five minutes. | + +Revoking a key doesn't recall credentials already issued with it: they keep +working until they expire, an hour after they were issued, or up to 12 hours if +the client asked for longer. Disabling the service account cuts those off too, +which makes it the emergency stop for a leaked key. Public products stay +readable by anyone, as always. + +Disabling keeps the service account's keys and grants. Enabling it again makes +every key that hasn't been revoked or expired work again, so revoke a leaked key +before you enable the account. + +### Revoke a key you found + +Anyone who holds a key can revoke it, without an account. If you come across one +that has leaked, in a repository, a log or a message, send it in the body of +this request: + +```bash +curl -X POST https://source.coop/api/v1/service-account-keys/revocations \ + -H 'content-type: application/json' \ + -d '{"key":"sck_…"}' +``` + +For a well-formed key the answer is always `204 No Content`, whether the key was +live, already revoked or unknown. A live key is revoked, and new exchanges with +it stop within about a minute. Send the key in the body, never in the URL. + +We plan to revoke keys pushed to public GitHub repositories automatically, +through GitHub's secret scanning. + +## GitHub Actions + +:::info Coming soon + +A service account can also trust a GitHub Actions workflow, which then signs in +with a token GitHub issues for each run, with no key to store. The service +account page already lets you add a workflow, but the data proxy doesn't accept +those sign-ins yet +([data.source.coop#222](https://github.com/source-cooperative/data.source.coop/issues/222), +[data.source.coop#223](https://github.com/source-cooperative/data.source.coop/issues/223)). +Until it does, a workflow can use an API key like any other machine: keep the +key in a GitHub Actions secret, write it to a file at the start of the job, and +set the variables above. + +::: diff --git a/docs/using-source/data-proxy.md b/docs/using-source/data-proxy.md index aef0a44..98348d9 100644 --- a/docs/using-source/data-proxy.md +++ b/docs/using-source/data-proxy.md @@ -58,6 +58,8 @@ aws s3 ls s3://your-org/your-product --profile source-coop See [Upload Your Data](/data-upload#get-credentials-with-the-source-cli-recommended) for the one-time install and profile setup. +For software that runs on its own, with nobody there to log in, use a [service account](/automated-access) instead. + ## Getting Started with AWS CLI If you don't have the AWS CLI installed, follow the [AWS CLI Getting Started Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html). diff --git a/docs/using-source/data-upload.md b/docs/using-source/data-upload.md index 119ac6a..2231195 100644 --- a/docs/using-source/data-upload.md +++ b/docs/using-source/data-upload.md @@ -14,8 +14,7 @@ If you do not see the option to upload (for example, Edit Mode or View Credentia ## How uploads work: the Source data proxy -Unless you set up [direct bucket access](#option-3-longstanding-or-automated-access-advanced), -every upload goes through the **Source data proxy** at `https://data.source.coop`. +Every upload goes through the **Source data proxy** at `https://data.source.coop`. The proxy speaks the **S3 API**, the object storage protocol that Amazon S3 introduced and that most storage tools now support. It is not itself an AWS service. @@ -36,10 +35,6 @@ That is why this guide uses AWS tools: `your-product` product in the `your-org` account on Source Cooperative. It is not an Amazon S3 bucket. Where your data is physically stored doesn't change it. -The one exception is [Option 3](#option-3-longstanding-or-automated-access-advanced), -which writes straight to Source's Amazon S3 bucket with your own AWS identity. -In that section, "AWS" really does mean AWS. - --- ## The short version (what you need to do) @@ -49,7 +44,7 @@ You have two main ways to upload data: 1. **Upload directly in the Source Cooperative User Interface (UI)** (drag-and-drop or file selector), or 2. **Use temporary Source credentials** to upload through the data proxy with an S3 client such as the AWS Command Line Interface (CLI) or an AWS Software Development Kit (SDK) -For long‑term or automated access, contact the Source Cooperative team at `hello@source.coop`. +For long‑term or automated access, use a [service account](/automated-access). --- @@ -235,211 +230,11 @@ aws s3 sync ./data s3://your-org/your-product/ --endpoint-url https://data.sourc ## Option 3: Long‑standing or automated access (ADVANCED) -If you need: - -- Continuous uploads -- Scheduled pipelines -- Integration from your own AWS account - -You can use your own IAM role to write to the Source Cooperative bucket. - -:::info This option really is AWS - -Unlike Options 1 and 2, this option bypasses the data proxy. You write -directly to Source Cooperative's Amazon S3 bucket (e.g. `us-west-2.opendata.source.coop`, `eu-west-1.opendata.source.coop`) -using an identity in **your own AWS account**. It only applies to products stored -in that bucket. If your product is stored with another provider, or you don't have -an AWS account, use [Option 2](#option-2-upload-through-the-data-proxy-with-temporary-credentials-recommended-for-larger-uploads) -or contact us. - -::: - -### How this works - -- You create an IAM role (or use an existing IAM user/account) in **your own** AWS account -- You send us its ARN, and we grant it write access to your account's prefix in the Source Cooperative bucket - -No credentials are shared, and no role chaining is required. - ---- - -### Step 1: Create an IAM role (or pick an identity to use) - -Which identity you send us depends on what is doing the uploading: - -| Uploading from | Send us | -| --- | --- | -| A service (ingestion pipeline, ECS task, Lambda, EC2, GitHub Actions with OIDC) | An **IAM role** ARN | -| A person running the AWS CLI locally | An **IAM user** ARN | -| Many identities in one account | The **AWS account** ARN (we trust the whole account; your account controls who may use it) | - -For a service, create a role in your account with a trust policy for whatever assumes it, then attach a policy granting it read/write access under your account prefix. Replace `your-org` with your Source Cooperative account ID. - -:::tip - -The [IAM policy wizard](/tools/iam-policy-wizard) generates this policy for you — enter your account ID and, optionally, a product ID. - -::: - -
-Example trust policy (ECS tasks) - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { "Service": "ecs-tasks.amazonaws.com" }, - "Action": "sts:AssumeRole" - } - ] -} -``` - -
- -
-Example access policy - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "s3:PutObject", - "s3:GetObject", - "s3:DeleteObject", - "s3:AbortMultipartUpload", - "s3:ListMultipartUploadParts" - ], - "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop/your-org/*" - }, - { - "Effect": "Allow", - "Action": "s3:ListBucket", - "Resource": "arn:aws:s3:::us-west-2.opendata.source.coop", - "Condition": { - "StringLike": { "s3:prefix": "your-org/*" } - } - } - ] -} -``` - -`s3:AbortMultipartUpload` and `s3:ListMultipartUploadParts` cover the multipart -uploads the AWS CLI and SDKs use automatically for large files. `s3:PutObject` -alone authorizes starting an upload and sending its parts, but without those two -a failed or resumed transfer cannot clean up after itself. Both are object-level -actions, so the prefix in `Resource` scopes them like the rest. - -:::note - -This policy deliberately omits `s3:ListBucketMultipartUploads`, which lists -in-progress uploads across the *whole* bucket. AWS does not support the -`s3:prefix` condition key on it, so it cannot be limited to your data, and no -upload path needs it — incomplete uploads are cleaned up automatically after 7 -days. Only `aws s3api list-multipart-uploads` requires it; contact us if you -have a workflow that requires this policy. - -::: - -
- -
-Creating the role with the AWS CLI - -```bash -aws iam create-role \ - --role-name source-coop-upload \ - --assume-role-policy-document file://trust-policy.json - -aws iam put-role-policy \ - --role-name source-coop-upload \ - --policy-name source-coop-write \ - --policy-document file://s3-policy.json -``` - -
- -:::note - -This policy only grants permission on *your* side. Uploads will still fail with `AccessDenied` until we grant the same role access on the bucket side (Step 2). - -::: - ---- - -### Step 2: Send us the ARN - -Email [hello@source.coop](mailto:hello@source.coop) with: - -- The ARN of the role, user, or account you want us to trust, for example: - - Role: `arn:aws:iam::123456789012:role/source-coop-upload` - - User: `arn:aws:iam::123456789012:user/data-uploader` - - Account: `arn:aws:iam::123456789012:root` -- Your Source Cooperative account ID and product ID (the `your-org/your-product` prefix you will write to) -- A short description of the workflow (e.g. "nightly ingestion pipeline running on ECS") - -We will add the ARN to the bucket policy and confirm when it is active. - ---- - -### Step 3: Upload using that identity - -Once we confirm, upload with credentials for that role, user, or account: - -```bash -aws s3 cp mydata.csv s3://us-west-2.opendata.source.coop/your-org/your-product/mydata.csv -``` - -Services that already run as the role (ECS tasks, Lambda, EC2 instance profiles) need no assume-role step — the SDK picks up the role automatically. - -
-Uploading a directory - -```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ -``` - -
- -
-Assuming the role from a workstation or CI job - -Let the AWS CLI do the assume-role for you — add a profile to `~/.aws/config`: - -```ini -[profile source-coop-upload] -role_arn = arn:aws:iam::123456789012:role/source-coop-upload -source_profile = default -region = us-west-2 -``` - -```bash -aws s3 sync ./data s3://us-west-2.opendata.source.coop/your-org/your-product/ \ - --profile source-coop-upload -``` - -
- -
-Uploading with boto3 - -```python -import boto3 - -s3 = boto3.client("s3") -s3.upload_file( - "mydata.csv", - "us-west-2.opendata.source.coop", - "your-org/your-product/mydata.csv", -) -``` - -
+For continuous uploads, scheduled pipelines and anything else that runs without +a person at the keyboard, create a **service account**: a login for software, +with its own access to your products. Give it an API key, and the AWS CLI and +SDKs upload through the data proxy with it, renewing their credentials on their +own. See [Automated Access](/automated-access). --- diff --git a/sidebars.ts b/sidebars.ts index 9a34c5a..00cba44 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -33,6 +33,7 @@ const sidebars: SidebarsConfig = { 'using-source/data-upload', 'using-source/bring-your-own-bucket', 'using-source/data-proxy', + 'using-source/automated-access', ], }, {