feat: exchange a service account's API key without a browser

[alukach]

Closes #17, as rescoped on 2026-09-25 for the revised ADR-013 (see the last decision below). Part of source-cooperative/source.coop#491. The server side is source-cooperative/data.source.coop#235 (the proxy's key exchange at /.sts) plus source-cooperative/source.coop#580 (issuing keys and answering the proxy's lookup).

What and why

A service account's API key is an opaque sck_ secret (ADR-013 as revised in source-cooperative/data.source.coop#234). The proxy exchanges it at /.sts as it does an ID token, but only from a POST form body. This PR lets the CLI make that exchange for software that runs unattended.

• Key mode for creds. source-coop creds --api-key-file <path> (or SOURCE_API_KEY_FILE, or SOURCE_API_KEY holding the key itself) exchanges the key at /.sts without a browser and prints the usual output. That output is credential_process JSON by default, so credential_process = source-coop creds --api-key-file … in ~/.aws/config serves AWS SDKs and GDAL 3.12+. Credentials are cached as login's are and replaced before they expire, so a daemon keeps working across expiry. I extended creds rather than adding a subcommand, because creds already is the credential_process mode.
• Credentials are replaced before SDKs start asking. botocore refreshes credential_process credentials once they have under 15 minutes left, and reruns the process on every credential lookup until it gets ones with more. The Go and JavaScript SDKs do the same at 5 minutes. creds therefore replaces credentials once they have less than 16 minutes left (the extra minute is for clock skew), or less than half the requested session if that is shorter. The half-session limit keeps a 15-minute session from being replaced on every call. This covers a key's exchange and login's refresh-token refresh. On main, creds serves credentials until 60 seconds before expiry, so it runs once per S3 request for the last 15 minutes of every session.
• Token in the body, never the URL. Every exchange, the Ory ID token's included, moves from the query string to a form-encoded POST body. The proxy refuses a key in a URL, URLs land in access logs, and reqwest puts the URL in its connection errors, so a network failure used to print the ID token to stderr. The deployed proxy already reads form bodies at /.sts (checked against staging below), so nothing changes for login users, and this can merge before the server side lands.
• Role. RoleArn defaults to arn:aws:iam::000000000000:role/_default. A bare name such as --role-arn ReadOnly is sent as arn:aws:iam::000000000000:role/ReadOnly, and a full ARN is sent as given.
• Errors. A refusal prints the proxy's Code and Message verbatim and exits 1, for example Error: STS error (InvalidIdentityToken): API key was not accepted (request id a40cee47fef1c4b4). When the message carries no request id (the 500 the proxy returns when it fails closed, or any Ory-token error), the id from the x-request-id header is appended, so every failure has one to quote to support.
• README. The key mode, the refresh window, the credential_process snippet, GDAL, and the stock-SDK setup that needs no CLI (AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN, AWS_ENDPOINT_URL_STS, AWS_ENDPOINT_URL_S3, AWS_REGION). It also corrects the stale --role-arn default in the login options table (source-coop-user → _default).

Decisions to flag

• Names. --api-key-file / SOURCE_API_KEY_FILE and SOURCE_API_KEY follow the CLI's pairing of each flag with a SOURCE_* variable (--proxy-url / SOURCE_PROXY_URL, --role-arn / SOURCE_ROLE_ARN). The _FILE suffix mirrors AWS_WEB_IDENTITY_TOKEN_FILE, and one key file serves both the CLI and a stock SDK. No flag takes the key itself, because a command line is visible to other users through ps and ends up in shell history and ~/.aws/config. When both are set the file wins, since it is the explicit choice a profile makes.
• The CLI checks the key's shape before sending it: trimmed, then exactly sck_ plus 36 base62 characters whose last six are the CRC-32 of the thirty before them (ADR-013, docs(adr): ADR-013 keys end in a CRC-32 checksum data.source.coop#242), the same rule as the proxy's parse_api_key. A key cut short or mistyped therefore fails locally too, with "check that it was copied whole". Anything else fails locally, so pointing --api-key-file at the wrong file never sends that file's contents anywhere. The error names the source (the path, or SOURCE_API_KEY) and never the value.
• Cache slot. A key's credentials are cached under <role>+key-<first 16 hex digits of sha256(key)>. They never share a slot with a login session or another key, and the key itself is never written down.
• In key mode the cache is best effort. A cache that can't be read (a corrupt entry, a locked keychain) counts as a miss, and a failed write is a warning on stderr, so the exchange is the only hard dependency. login mode stays strict, because there the cache holds the refresh token.
• Every key-mode call takes a per-slot lock, so SDK threads or a batch of jobs started together share one exchange rather than each spending one against the proxy's limit of 100 a minute per IP.
• Refreshing early never ends a session sooner. Sometimes the early exchange or refresh fails, through a proxy outage, a revoked key or an expired refresh token. If the cached credentials are still valid, creds then serves them with a warning on stderr, as SDKs keep their credentials when an advisory refresh fails. Failing instead would make a one-shot aws s3 cp in a cron job fail during a proxy blip whenever the cache was in its last 16 minutes, not just its last minute. The one-minute buffer remains the floor of the refresh window.
• login credentials without a refresh token keep the one-minute rule. Only an interactive login can replace them, so refreshing them early would only turn valid credentials into an error sooner.
• Role expansion applies to the Ory-token exchange too. It lives in sts::assume_role, which both paths share. The proxy's is_default_role treats _default and arn:…:role/_default alike, so login behaves as before, and its cache is still keyed by the role as given, so existing caches stay valid.
• The #580 dialog suggests a role #235 refuses. feat(accounts): opaque API keys resolved by hash, and a working key UI source.coop#580's issue dialog suggests AWS_ROLE_ARN=arn:aws:iam::<account>:role/FullAccess, but feat(sts): exchange opaque API keys at /.sts by hash lookup data.source.coop#235 serves only _default until Add the ReadOnly Role alongside FullAccess data.source.coop#221 lands, and answers any other role with MalformedPolicyDocument: role not found. This README uses role/_default. I'm flagging it here rather than changing either PR.
• The issue text predates the rescope. Unattended refresh: exchange an API key without a browser and keep the token file fresh #17 on GitHub asks for a mode that "keeps AWS_WEB_IDENTITY_TOKEN_FILE fresh". Under the revised ADR-013 that file holds the key itself and never changes, so there is nothing to refresh. The rescoped issue asks for credential_process plus an env export for tools that send STS as a GET, which is what this delivers (--format env already existed). The rescoped text wins where the two disagree.

How I tested

• cargo fmt --check and cargo clippy --all-targets -- -D warnings are clean, as is CI's cargo clippy -- -D warnings. cargo test: 33 passed, 1 ignored (the existing keyring test). All re-run on a39541d, after the checksum commit.
• New unit tests, using wiremock:
  • An API-key exchange is a form-encoded POST with exactly Action, RoleArn (the bare ReadOnly expanded), WebIdentityToken and DurationSeconds, and no query string.
  • Refresh window: credentials with 10 minutes left are due at the default hour but not for a 15-minute session, and credentials with 30 minutes left are not due at an hour.
  • Credentials with 5 minutes left of a 15-minute session are exchanged again before they expire. If that exchange fails they are served, and once they have expired a failure is an error. Fresh credentials are served without a request, and an unusable cache still yields credentials.
  • The proxy's refusal body, byte for byte, surfaces verbatim. A header request id is appended only when the message lacks one.
  • Key parsing trims \n and \r\n, accepts keys whose checksums were computed independently with Python's zlib, and rejects empty, JWT, short, long, wrong-case, mistyped, non-base62 and checksum-less 47-character input.
  • The file wins over the environment variable, and errors never echo the key.
  • The credential_process JSON has its five fields.
  • Key cache slots differ per key and per role, and never equal login's.
• To confirm the new tests fail when the logic breaks, I put back the one-minute rule, which failed two of them, and then disabled the fallback, which failed the third.
• The existing refresh tests now expect a POST body, with the ID token absent from the query string.
• End to end, I ran a Linux release build of fc37931 in ghcr.io/osgeo/gdal:ubuntu-small-latest. The image is headless with no keyring, so the cache used its file fallback. The build ran against a local stand-in for the proxy that answers /.sts as #235 does and issues credentials for the requested duration, here 300 seconds, which puts the refresh point 150 seconds before expiry:
  • The first creds --api-key-file exchanged the key and printed only the credential_process JSON: stdout parsed as JSON, with 0 bytes on stderr. An immediate second call was served from the cache.
  • 155 seconds later, 145 seconds before expiry, I made the stand-in answer 500. creds exited 0, served the cached credentials, and warned on stderr: Warning: STS error (InternalError): internal error (request id mockreq0002); using cached credentials until they expire. With the stand-in back, the next call exchanged the key again before the credentials expired.
  • The cache file was ReadOnly_key-c8b4449fef56b9ee.json, mode 0600.
  • A refused key printed Error: STS error (InvalidIdentityToken): API key was not accepted (request id mockreq0004), exited 1, and wrote nothing to stdout.
  • GDAL 3.14dev, with credential_process = source-coop creds --api-key-file … in ~/.aws/config, ran the CLI and logged "Successfully obtained credentials from credential_process". It then read /vsis3/my-account/my-product/empty.geojson with a request signed by the exchanged access key and session token.
  • Every /.sts request was an application/x-www-form-urlencoded POST with an empty query string.
• Live, with a throwaway well-formed key, on 9270c1a (fc37931 changes when credentials are replaced, not the request):
  • data.staging.source.coop runs main's proxy, which knows no keys, so the token took the same STS route an Ory ID token now takes. The form body reached that route, which answered STS error (InvalidIdentityToken): malformed JWT (request id a40d0b771cbed85f).
  • The #235 preview failed closed, as expected while staging doesn't serve the key route: STS error (InternalError): internal error (request id a40d0b783fc2b4a4).
• Not run:
  • A successful key exchange against a real deployment. That needs feat(accounts): opaque API keys resolved by hash, and a working key UI source.coop#580 deployed where the #235 preview points.
  • A real browser login. Its refresh is covered by the updated unit tests. Its decision of when to refresh reads the OS keyring, so it has no unit test, as on main, but it uses the same needs_refresh and fallback as key mode, which do.
  • The stock-SDK section of the README restates what feat(sts): exchange opaque API keys at /.sts by hash lookup data.source.coop#235 verified with boto3; I did not rerun it.

Docs and ADRs

• ADR-013 as revised (docs(adr): revise ADR-013 — API keys are opaque secrets resolved by the platform data.source.coop#234) says the CLI "offers the same exchange with the key in the request body and can serve as an AWS credential_process", which is how GDAL gets credentials. This PR implements that, so the ADR still holds. ADR-014 is untouched.
• docs.source.coop's docs/using-source/data-upload.md documents login with credential_process = source-coop creds. Setup is unchanged, and the page's claim that the AWS CLI and SDKs refresh credentials automatically still holds; they now just stop rerunning creds on every request. User docs for the key workflow belong to Unattended workflow guide docs.source.coop#34, which relies on this PR for GDAL.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R1eiTse4416N6uTgAy4Ddd