Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 53 additions & 1 deletion docs/using-source/data-proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,62 @@ See [Upload Your Data](/data-upload#get-credentials-with-the-source-cli-recommen

If you don't have the AWS CLI installed, follow the [AWS CLI Getting Started Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html).

## Using GeoPandas

For working with spatial data using Python, you can use [GeoPandas](https://geopandas.org/). You'll need Python and the `geopandas` and `s3fs` packages.

```sh
pip install geopandas s3fs
```

Then, in your code:

```python
import geopandas

# Use the Data Proxy
storage_options = {
"endpoint_url": "https://data.source.coop",
"anon": True
}

gdf = geopandas.read_parquet(
"s3://kerner-lab/fields-of-the-world/vietnam/chips_vietnam.parquet",
storage_options=storage_options,
)
```

The code for using pandas (for non-geospatial data) is similar.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want to introduce another example dataset, but I can if you like.


## Using [DuckDB](https://duckdb.org/)

```sql
-- https://duckdb.org/docs/current/core_extensions/httpfs/overview#installation-and-loading
INSTALL httpfs;
LOAD httpfs;

-- https://duckdb.org/docs/current/core_extensions/spatial/overview#installing-and-loading
INSTALL spatial;
LOAD spatial;

CREATE OR REPLACE SECRET source_coop (
TYPE s3,
PROVIDER config,
ENDPOINT 'data.source.coop',
URL_STYLE 'path'
-- If you're working with multiple S3-compatible sources (beyond the Data Proxy), you'll want to only apply this SECRET to the Source Cooperative account name(s).
-- https://duckdb.org/docs/current/configuration/secrets_manager#creating-multiple-secrets-for-the-same-service-type
-- SCOPE 's3://kerner-lab'
);

SELECT *
FROM 's3://kerner-lab/fields-of-the-world/vietnam/chips_vietnam.parquet';
```

## Current Status

The Source Data Proxy is currently in beta. We're working to make performance indistinguishable from accessing objects directly via AWS S3.

## Uploading Data Through the Source Data Proxy

Uploading data through the Source Data Proxy is currently disabled while we work on new, easier, and more secure ways to upload data. If you need to upload data, please contact hello@source.coop.
Uploading data through the Source Data Proxy is currently disabled while we work on new, easier, and more secure ways to upload data. If you need to upload data, please contact hello@source.coop.