Skip to content

docs: clarify that uploads go through the S3-compatible data proxy - #35

Merged
alukach merged 5 commits into
mainfrom
docs/clarify-s3-proxy
Sep 25, 2026
Merged

alukach merged 5 commits into
mainfrom
docs/clarify-s3-proxy

Conversation

@alukach

@alukach alukach commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Follow-up to the discussion in source-cooperative/source-coop-cli#16. The upload guide mentions AWS dozens of times and never says the backing store may be something other than AWS, so it reads as if uploading requires AWS.

Changes to /data-upload

  • New How uploads work: the Source data proxy section explains that:
    • data.source.coop speaks the S3 API but is not an AWS service.
    • Products may be stored on S3, GCS, Azure, R2, and others.
    • The AWS CLI and SDKs are only S3 clients here.
    • Credentials come from Source and use the AWS credential format.
    • s3://your-org/your-product/ is a proxy address, not an S3 bucket.
  • Option 2 now describes uploading through the data proxy with temporary credentials. It explains what endpoint_url does, adds AWS_ENDPOINT_URL and a boto3 endpoint_url example for credentials copied from the UI, and notes that the region value does not show where the data is stored.
  • Fixes stale s3://us-west-2.opendata.source.coop/... addresses in "Where to upload" and the AWS CLI example (missed in f614344). They now use s3://your-org/your-product/ with --endpoint-url.
  • Option 3 gets a callout: it bypasses the proxy, writes directly to Source's Amazon S3 bucket with the uploader's own AWS identity, and only applies to products stored there.
  • Small fixes: "ilke" → "like", "selectory" → "selector", "permanent AWS access keys" → "permanent access keys".

docusaurus build passes with no broken links or anchors.

🤖 Generated with Claude Code

The upload guide read as if data lived in AWS. Explain that the data
proxy speaks the S3 API and may be backed by any object store, that AWS
tools are only S3 clients, and that credentials come from Source. Fix
stale s3://us-west-2.opendata.source.coop addresses in Option 2 and call
out that Option 3 is the one path that is genuinely AWS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs-source-coop Ready Ready Preview Sep 25, 2026 3:54am UTC

Request Review

@tylere tylere left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this is much clearer.
I left a few minor comments.

Comment thread docs/using-source/data-upload.md
Comment thread docs/using-source/data-upload.md Outdated
Comment thread docs/using-source/data-upload.md Outdated
@@ -408,7 +465,7 @@ This protects both you and Source Cooperative.

Please do not:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a user tries any of these actions they will fail, correct?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the actions that aren't human behavior, yes, they will be denied.

Clarified the S3 bucket examples for data upload.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread docs/using-source/data-upload.md Outdated
@alukach
alukach merged commit 6265a95 into main Sep 25, 2026
2 checks passed
@alukach
alukach deleted the docs/clarify-s3-proxy branch September 25, 2026 03:56

This branch was successfully deployed

1 active deployment
Preview — 0eb2eab5 Deployed Sep 25, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants