docs: add an automated access guide for service accounts

[alukach]

What

A new page, Automated Access (/automated-access, last under Using Source), for software that reaches Source Cooperative with nobody at the keyboard. It covers what a service account is; creating one and granting it products, as the UI flow; issuing an API key (shown once) and the five variables the AWS CLI or an SDK needs to exchange it at the data proxy and refresh on its own, with AWS CLI and boto3 examples and their minimum versions; the refusal a revoked, expired or unknown key gets and the request id to quote to support; keeping the key out of URLs and aws --debug output; several keys per service account for rotation, and Change expiry; tools that keep their first credentials, so long transfers fail mid-flight; GDAL; revocation and disabling as the emergency stop, with their timings; revoking a key you found; and GitHub Actions as a clearly marked "coming soon".

Upload Your Data's Option 3 walked through trusting an IAM role in the uploader's own AWS account to write straight to Source's bucket. Service accounts supersede that flow, so the section is now a short pointer to the new page, as are the two earlier references to it on that page and the "contact us" line for automated access. The heading is kept as it was so #option-3-longstanding-or-automated-access-advanced still lands. Access Data gains one sentence pointing unattended software at service accounts.

This documents features whose code is still in open PRs, so it merges only after they deploy: source-cooperative/source.coop#570 (key lifecycle), source-cooperative/source.coop#580 (opaque keys, the variables printed at issue, Change expiry, the Disable wording), source-cooperative/source.coop#581 (the public self-revoke route), source-cooperative/data.source.coop#235 (the proxy's key exchange), and source-cooperative/data.source.coop#221 (the FullAccess and ReadOnly role names).

Decisions to flag

• Role ARN and region. The page uses AWS_ROLE_ARN=arn:aws:iam::<service-account-id>:role/FullAccess and AWS_REGION=us-west-2, the form the issue dialog in feat(accounts): opaque API keys resolved by hash, and a working key UI source.coop#580 prints and the GitHub snippet on source.coop main already uses. It documents FullAccess as everything the service account may do and ReadOnly as reads only. Both names need Add the ReadOnly Role alongside FullAccess data.source.coop#221, which is being built; until then the proxy serves only _default, which stays as an alias of FullAccess and which the page doesn't mention.
• GDAL sends the key to AWS, not to the proxy. The brief said GDAL's own STS call puts the token in the URL, which the proxy refuses. GDAL's source (port/cpl_aws.cpp, checked at 3.6.0, 3.12.0 and master) shows it is worse than that. GDAL never reads AWS_ENDPOINT_URL_STS; its STS root is CPL_AWS_STS_ROOT_URL, defaulting to https://sts.<AWS_REGION>.amazonaws.com. So on a machine with the five variables and no other credentials, GDAL 3.6 or later sends the key to AWS in a GET query string. The page says to set CPL_AWS_WEB_IDENTITY_ENABLE=NO (present since 3.6.0) wherever GDAL runs beside the variables, and to replace a key GDAL has already seen. The coming-soon note for the Source CLI (Unattended refresh: exchange an API key without a browser and keep the token file fresh source-coop-cli#17) names GDAL 3.12, the first release that reads credential_process and refreshes it on expiry.
• Tools that keep their first credentials. The issue says GDAL, DuckDB and rclone "capture environment variables at process start". The page puts it as fixed credentials (the three AWS_* values, or the same values in a tool's own settings), because that is where the mid-transfer failure is certain. GDAL 3.12+ with credential_process refreshes, and so can recent DuckDB. The failure is named as the proxy returns it: ExpiredToken, HTTP 403 (multistore 0.7.2).
• DuckDB's credential_chain isn't recommended with a key. The page says DuckDB reads a secret's credentials at CREATE SECRET and suggests CREATE OR REPLACE SECRET between batches. It doesn't suggest PROVIDER credential_chain with the five variables. Whether DuckDB's bundled AWS C++ SDK sends that exchange to AWS_ENDPOINT_URL_STS depends on its version: the older internal STS client hard-codes sts.<region>.amazonaws.com, while the CRT-based provider reads an endpoint override. If it doesn't, the key goes to AWS. Verifying that is Verify an unmodified AWS SDK acquires and refreshes credentials from the environment alone data.source.coop#229's work. rclone likewise appears only in the fixed-credentials framing.
• Old SDKs. Releases before AWS CLI 2.13.0 and botocore 1.31.0 (boto3 1.28.0) ignore the endpoint variables and send the key to AWS. The page says so; the versions come from the three projects' changelogs.
• Lifetimes. "An hour by default, up to 12 hours if the client asked" comes from the proxy's 3600-second default and STS_MAX_SESSION_DURATION_SECS = "43200" in production on feat(sts): exchange opaque API keys at /.sts by hash lookup data.source.coop#235's branch.
• Self-revoke and secret scanning. Revoking a found key with POST https://source.coop/api/v1/service-account-keys/revocations lands with feat(accounts): revoke leaked API keys via self-revoke and GitHub secret scanning source.coop#581. It takes the key in a JSON body and answers 204 for any well-formed key. Automatic revocation of keys pushed to public GitHub also lands with feat(accounts): revoke leaked API keys via self-revoke and GitHub secret scanning source.coop#581, but it needs GitHub partner registration, so the page says only that it is planned.
• GitHub Actions is a short "coming soon" note, not instructions, because trusted workflows need Namespace the credential subject by verified issuer data.source.coop#222 and Trust GitHub Actions alongside the Source issuer data.source.coop#223. It warns that the service account page already offers adding a workflow, and says a workflow can use an API key meanwhile.
• Left in place: the IAM and bucket policy wizards. /tools/iam-policy-wizard and the internal /tools/bucket-policy-wizard served the superseded Option 3 flow. Nothing in the docs links to them now, and the IAM wizard's intro still sends readers to Upload Your Data "for the full walkthrough". Whether to retire them depends on whether anyone is still onboarded through IAM roles, so that is a separate change.
• Scope against the issue text. This follows the revised scope for Unattended workflow guide #34 (not yet applied on GitHub) where the issue text differs. It covers the server, VM and HPC path, with one line for cron and systemd. It has no Kubernetes snippet, since the five variables don't change by environment. GitHub Actions is "coming soon" rather than documented, and the SDK path doesn't wait on Unattended refresh: exchange an API key without a browser and keep the token file fresh source-coop-cli#17; only GDAL does. The Access Data sentence goes beyond the brief; it is there because that page is where readers wanting programmatic access land.

How I tested it

• docusaurus build (Docusaurus 3.10.2, Node 22.9.0) succeeds with the site's onBrokenLinks: 'throw', and prints no broken-link or broken-anchor warnings. I built after the final edits and checked the output: the page renders at /automated-access, the sidebar lists it after Access Data, Access Data's "Next" goes to it, #github-actions resolves, both admonitions render, and the Option 3 anchor still exists. npm ci in the worktree failed with ENOSPC (the disk is nearly full), so the build ran against the main checkout's existing pnpm install through a symlink, with DOCUSAURUS_NO_PERSISTENT_CACHE=1 so nothing was written into it. I removed the symlink and the build output afterwards.
• Not run: the page's commands against a live proxy. That needs the PRs above deployed.
• I checked the facts against their sources: GDAL's source at 3.6.0 through 3.12.0 and master; the AWS CLI v2, botocore and boto3 changelogs; multistore 0.7.2's error mapping; feat(sts): exchange opaque API keys at /.sts by hash lookup data.source.coop#235 for the refusal messages and the request id (the Cloudflare ray id); and the UI labels on source.coop main and on feat/opaque-api-keys at c85aff23.

Docs and ADRs

This is the docs half of source-cooperative/source.coop#570, source-cooperative/source.coop#580, source-cooperative/source.coop#581 and source-cooperative/data.source.coop#235. I checked ADR-013 as revised (source-cooperative/data.source.coop#234) and ADR-014 (source-cooperative/data.source.coop#232). The page describes what they decide: the five variables, a key only in a request body, one refusal carrying a request id, revocation within the 60-second cache, disabling as the emergency stop (writes within a minute, restricted reads within five), and FullAccess/ReadOnly. Neither needs a change. The other Using Source pages (Create an Account, Create a Data Product, Bring Your Own Bucket) still hold.

Related

Part of #34. It isn't Closes: the GitHub Actions section and the GDAL setup wait on source-cooperative/data.source.coop#222, source-cooperative/data.source.coop#223 and source-cooperative/source-coop-cli#17. #32 ("Document unattended credential acquisition") covers the same ground and looks like a duplicate of #34; I left it untouched. Part of source-cooperative/source.coop#491.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R1eiTse4416N6uTgAy4Ddd

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs-source-coop	Ready	Preview	Sep 25, 2026 9:13pm UTC