-
Notifications
You must be signed in to change notification settings - Fork 39
DOC-428: Update remainder of Snowflake docs to push lstk #908
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,12 +8,30 @@ label: | |
|
|
||
| LocalStack exposes various configuration options to control its behaviour. | ||
|
|
||
| These options can be passed to LocalStack as environment variables like so: | ||
| With `lstk`, these options can be passed as `LOCALSTACK_`-prefixed environment variables when starting the container: | ||
|
|
||
| ```bash | ||
| DEBUG=1 localstack start --stack snowflake | ||
| LOCALSTACK_DEBUG=1 lstk start | ||
| ``` | ||
|
|
||
| Alternatively, set them as named environment profiles in your config file and reference them from the container block: | ||
|
|
||
| ```toml | ||
| # .lstk/config.toml | ||
| [[containers]] | ||
| type = "snowflake" | ||
| env = ["debug"] | ||
|
|
||
| [env.debug] | ||
| DEBUG = "1" | ||
| ``` | ||
|
|
||
| ```bash | ||
| lstk start | ||
| ``` | ||
|
|
||
| See [Passing environment variables to the container](/aws/developer-tools/running-localstack/lstk#passing-environment-variables-to-the-container) for details. | ||
|
|
||
| ## Core | ||
|
|
||
| Options that affect the core Snowflake emulator functionality. | ||
|
|
@@ -35,11 +53,11 @@ Options that affect the core Snowflake emulator functionality. | |
|
|
||
| By default, the Snowflake emulator accepts requests for hostnames such as `snowflake.localhost.localstack.cloud` and other `*.snowflake.*` hostnames. | ||
| If you expose the emulator through a custom DNS name, for example in Kubernetes or behind an ingress, set `SF_HOSTNAMES` to the exact hostnames clients use to reach the emulator. | ||
| When you use the `localstack` CLI, add the `LOCALSTACK_` prefix so the CLI passes the variable to the container: | ||
| When you use `lstk`, add the `LOCALSTACK_` prefix so the CLI passes the variable to the container: | ||
|
|
||
| ```bash | ||
| LOCALSTACK_SF_HOSTNAMES=snowflake.internal.example.com,snowflake.internal,snowflake.localhost.localstack.cloud \ | ||
| localstack start --stack snowflake | ||
| lstk start | ||
| ``` | ||
|
|
||
| The first hostname in `SF_HOSTNAMES` is used as the primary hostname for local connection defaults and generated URLs. | ||
|
|
@@ -75,77 +93,27 @@ If your custom hostname also needs a matching TLS certificate, use LocalStack's | |
| LOCALSTACK_SF_HOSTNAMES=snowflake.internal.example.com \ | ||
| CUSTOM_SSL_CERT_PATH=/var/lib/localstack/custom/cert.pem \ | ||
| SKIP_SSL_CERT_DOWNLOAD=1 \ | ||
| localstack start --stack snowflake | ||
| lstk start | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Similarly, use the config I provided in https://linear.app/localstack/issue/PRO-327/verify-lstk-behavior-in-capabilitiesconfigurationmd |
||
| ``` | ||
|
|
||
| The file referenced by `CUSTOM_SSL_CERT_PATH` must contain a certificate and private key that match the hostname used by your Snowflake clients. | ||
| For more general guidance on adding trusted certificates to LocalStack, see [Custom TLS certificates](/aws/developer-tools/security-testing/custom-tls-certificates/). | ||
|
|
||
| ## CLI | ||
|
|
||
| These options are applicable when using the CLI to start LocalStack. | ||
|
|
||
| | Variable | Example Values | Description | | ||
| | - | - | - | | ||
| | `LOCALSTACK_VOLUME_DIR` | `~/.cache/localstack/volume` (on Linux) | The location on the host of the LocalStack volume directory mount. | | ||
| | `CONFIG_PROFILE` | | The configuration profile to load. See [Profiles](#profiles) | | ||
| | `CONFIG_DIR` | `~/.localstack` | The path where LocalStack can find configuration profiles and other CLI-specific configuration | | ||
| `lstk` is configured through its config file rather than through environment variables. | ||
| See [Configuration](/aws/developer-tools/running-localstack/lstk#configuration) on the `lstk` page for the config file search order, the field reference, and how to define named environment profiles. | ||
|
|
||
| ## Docker | ||
|
|
||
| Options to configure how LocalStack interacts with Docker. | ||
|
|
||
| | Variable | Example Values | Description | | ||
| | - | - | - | | ||
| | `LOCALSTACK_VOLUME_DIR` | `~/.cache/localstack/volume` (on Linux) | The location on the host of the LocalStack volume directory mount. | | ||
| | `DOCKER_FLAGS` | | Allows to pass custom flags (e.g., volume mounts) to "docker run" when running LocalStack in Docker. | | ||
| | `DOCKER_SOCK` | `/var/run/docker.sock` | Path to local Docker UNIX domain socket | | ||
| | `DOCKER_BRIDGE_IP` | `172.17.0.1` | IP of the Docker bridge used to enable access between containers | | ||
| | `LEGACY_DOCKER_CLIENT` | `0`\|`1` | Whether LocalStack should use the command-line Docker client and subprocess execution to run Docker commands, rather than the Docker SDK. | | ||
| | `DOCKER_CMD` | `docker` (default), `sudo docker`| Shell command used to run Docker containers (only used in combination with `LEGACY_DOCKER_CLIENT`) | | ||
| | `FORCE_NONINTERACTIVE` | | When running with Docker, disables the `--interactive` and `--tty` flags. Useful when running headless. | | ||
|
|
||
| ## Profiles | ||
|
|
||
| LocalStack supports configuration profiles which are stored in the `~/.localstack` config directory. | ||
| A configuration profile is a set of environment variables stored in a `*.env` file in the LocalStack config directory. | ||
|
|
||
| Here is an example of what configuration profiles might look like: | ||
|
|
||
| ```bash | ||
| tree ~/.localstack | ||
| /home/username/.localstack | ||
| ├── default.env | ||
| ├── dev.env | ||
| └── pro.env | ||
| ``` | ||
|
|
||
| Here is an example of what a specific environment profile looks like | ||
|
|
||
| ```bash | ||
| cat ~/.localstack/pro-debug.env | ||
| LOCALSTACK_AUTH_TOKEN=XXXXX | ||
| SF_LOG=trace | ||
| SF_S3_ENDPOINT=s3.localhost.localstack.cloud:4566 | ||
| ``` | ||
|
|
||
| You can load a profile by either setting the environment variable `CONFIG_PROFILE=<profile>` or the `--profile=<profile>` CLI flag when using the CLI. | ||
| Let's take an example to load the `dev.env` profile file if it exists: | ||
|
|
||
| ```bash | ||
| localstack --profile=dev start --stack snowflake | ||
| ``` | ||
|
|
||
| If no profile is specified, the `default.env` profile will be loaded. | ||
| If explicitly specified, any environment variables will overwrite the configurations defined in the profile. | ||
|
|
||
| To display the config environment variables, you can use the following command: | ||
|
|
||
| ```bash | ||
| localstack --profile=dev config show | ||
| ``` | ||
|
|
||
| :::note | ||
| The `CONFIG_PROFILE` is a CLI feature and cannot be used directly with a docker-compose setup. | ||
| You can look at [alternative means of setting environment variables](https://docs.docker.com/compose/environment-variables/set-environment-variables/) for your Docker Compose setups. | ||
| For Docker setups, we recommend passing the environment variables directly to the `docker run` command. | ||
| ::: | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -27,7 +27,7 @@ CREATE DATABASE test123; | |
| SHOW DATABASES; | ||
| ``` | ||
|
|
||
| Mount the script into `/etc/localstack/init/ready.d/` using Docker Compose or the `localstack` CLI: | ||
| Mount the script into `/etc/localstack/init/ready.d/` using Docker Compose or `lstk`: | ||
|
|
||
| <Tabs> | ||
| <TabItem label="docker-compose.yml"> | ||
|
|
@@ -51,12 +51,22 @@ services: | |
| - "/var/run/docker.sock:/var/run/docker.sock" | ||
| ``` | ||
| </TabItem> | ||
| <TabItem label="CLI"> | ||
| <TabItem label="lstk"> | ||
| Declare the bind mount and the `DEBUG` profile in your config file, then start LocalStack: | ||
|
|
||
| ```toml | ||
| # .lstk/config.toml | ||
| [[containers]] | ||
| type = "snowflake" | ||
| env = ["debug"] | ||
| volumes = ["/path/to/test.sf.sql:/etc/localstack/init/ready.d/test.sf.sql"] | ||
|
|
||
| [env.debug] | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd add an inline comment here saying: "Optionally enable DEBUG" as DEBUG is not required for init hook but helps to diagnose init hook issues |
||
| DEBUG = "1" | ||
| ``` | ||
|
|
||
| ```bash | ||
| # DOCKER_FLAGS are additional parameters to the `docker run` command of localstack start | ||
| DOCKER_FLAGS='-v /path/to/test.sf.sql:/etc/localstack/init/ready.d/test.sf.sql' \ | ||
| DEBUG=1 \ | ||
| localstack start --stack snowflake | ||
| lstk start | ||
| ``` | ||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,14 +21,13 @@ State Management is an essential feature that supports various use-cases, such a | |
|
|
||
| LocalStack’s Persistence mechanism enables the saving and restoration of the entire LocalStack state. It functions as a **pause and resume** feature, allowing you to take a snapshot of your LocalStack instance and save this data to disk. This mechanism ensures a quick and efficient way to preserve and continue your work with Snowflake resources locally. | ||
|
|
||
| To start snapshot-based persistence, launch LocalStack with the configuration option `PERSISTENCE=1`. This setting instructs LocalStack to save all local Snowflake resources and their respective application states into the LocalStack Volume Directory. Upon restarting LocalStack, you'll be able to resume your activities exactly where you left off. | ||
| To start snapshot-based persistence, launch LocalStack with the `--persist` command-line option, or the configuration option `PERSISTENCE=1`. This setting instructs LocalStack to save all local Snowflake resources and their respective application states into the LocalStack Volume Directory. Upon restarting LocalStack, you'll be able to resume your activities exactly where you left off. | ||
|
|
||
| <Tabs> | ||
| <TabItem label="LocalStack CLI"> | ||
| <TabItem label="lstk"> | ||
| ```bash | ||
| export LOCALSTACK_AUTH_TOKEN=<your_auth_token> | ||
| PERSISTENCE=1 \ | ||
| localstack start --stack snowflake | ||
| lstk start --persist | ||
| ``` | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's worth adding a note that to resume a persisted session after the emulator stopped, or to restart a session while keeping persistence on, you also need this flag, i.e. and |
||
| </TabItem> | ||
| <TabItem label="Docker Compose"> | ||
|
|
@@ -69,17 +68,17 @@ The Export/Import State feature enables you to export the state of your LocalSta | |
| To export the state, you can run the following command: | ||
|
|
||
| ```bash | ||
| localstack state export '<file-name>' | ||
| lstk snapshot save '<file-name>' | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since file-name here mostly associates with a path, I'd suggest |
||
| ``` | ||
|
|
||
| You can use the `<file-name>` argument to specify a file path to export the state to. If you do not specify a file path, the state will be exported to the current working directory into a file named `ls-state-export`. | ||
| You can use the `<file-name>` argument to specify a file path to export the state to. If you do not specify a file path, the state will be exported to the current working directory into an auto-named snapshot file. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's helpful to link to the snapshot section of the lstk doc for more info on supported save destinations (local and remote). |
||
|
|
||
| ### Import the State | ||
|
|
||
| To import the state, you can run the following command: | ||
|
|
||
| ```bash | ||
| localstack state import '<file-name>' | ||
| lstk snapshot load '<file-name>' | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you agree with using above then this also changes to |
||
| ``` | ||
|
|
||
| The `<file-name>` argument is required and specifies the file path to import the state from. The file should be generated from a previous export. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,7 +14,7 @@ The Jupyter Notebook and the dataset used in this tutorial are available on [Git | |
|
|
||
| ## Prerequisites | ||
|
|
||
| - [`localstack` CLI](/snowflake/getting-started/) with a [`LOCALSTACK_AUTH_TOKEN`](/snowflake/getting-started/auth-token/) | ||
| - [`lstk`](/snowflake/getting-started/) with a [`LOCALSTACK_AUTH_TOKEN`](/snowflake/getting-started/auth-token/) | ||
| - [LocalStack for Snowflake](/snowflake/getting-started/) | ||
| - [Snowpark](/snowflake/integrations/snowpark) with other Python libraries | ||
| - [Jupyter Notebook](https://jupyter.org/install#jupyter-notebook) | ||
|
|
@@ -27,7 +27,7 @@ Start your LocalStack container in your preferred terminal/shell. | |
|
|
||
| ```bash | ||
| export LOCALSTACK_AUTH_TOKEN=<your_auth_token> | ||
| localstack start --stack snowflake | ||
| lstk start --type snowflake | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Simply |
||
| ``` | ||
|
|
||
| ## Create a Snowpark session | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,7 +18,7 @@ With LocalStack's Snowflake emulator, you can create catalog integrations that c | |
|
|
||
| ## Prerequisites | ||
|
|
||
| - [`localstack` CLI](/snowflake/getting-started/) with a [`LOCALSTACK_AUTH_TOKEN`](/snowflake/getting-started/auth-token/) | ||
| - [`lstk`](/snowflake/getting-started/) with a [`LOCALSTACK_AUTH_TOKEN`](/snowflake/getting-started/auth-token/) | ||
| - [LocalStack for Snowflake](/snowflake/getting-started/) | ||
| - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) & [`awslocal` wrapper](/aws/connecting/aws-cli/#localstack-aws-cli-awslocal) | ||
| - Python 3.10+ with `pyiceberg` and `pyarrow` installed | ||
|
|
@@ -29,7 +29,7 @@ Start your LocalStack container with the Snowflake emulator enabled. | |
|
|
||
| ```bash | ||
| export LOCALSTACK_AUTH_TOKEN=<your_auth_token> | ||
| localstack start --stack snowflake | ||
| lstk start --type snowflake | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Simply |
||
| ``` | ||
|
|
||
| ## Create S3 Tables resources | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd recommend using the lstk config file to do this: see my reply in https://linear.app/localstack/issue/PRO-327/verify-lstk-behavior-in-capabilitiesconfigurationmd