diff --git a/docs/how-tos/airflow/send-emails.md b/docs/how-tos/airflow/send-emails.md index 02eee0ad..e17ebc35 100644 --- a/docs/how-tos/airflow/send-emails.md +++ b/docs/how-tos/airflow/send-emails.md @@ -96,6 +96,53 @@ Once you created the `SMTP` integration, it's time to add it to the Airflow serv - Click `Save Changes`. +## Test the integration with a simple DAG + +Datacoves turns the SMTP integration into the `smtp_default` Airflow connection (you can see it in Airflow under `Admin > Connections`), so no extra configuration is needed in your DAGs. Before wiring up failure notifications, you can verify the setup end to end with a DAG that just sends an email: + +```python +from pendulum import datetime + +try: + # Airflow 3 + from airflow.sdk import dag +except ImportError: + # Airflow 2 + from airflow.decorators import dag + +try: + # Airflow 3 + from airflow.providers.smtp.operators.smtp import EmailOperator +except ImportError: + # Airflow 2 + from airflow.operators.email import EmailOperator + + +@dag( + default_args={ + "start_date": datetime(2024, 1, 1), + "owner": "Noel Gomez", # Replace with your name + }, + description="Send a test email through the SMTP integration", + schedule=None, + tags=["sample"], + catchup=False, +) +def smtp_test(): + + EmailOperator( + task_id="send_test_email", + to="gomezn@example.com", # Replace with the recipient + subject="SMTP test from Airflow", + html_content="It works! Sent through the smtp_default connection.", + ) + + +dag = smtp_test() +``` + +Trigger it manually from the Airflow UI; the recipient should get the test email within a few seconds. + ## Implement in a DAG If you have already created a DAG it's time to modify your DAG to make use of our newly set up SMTP integration on Airflow. diff --git a/docs/how-tos/airflow/use-aws-secrets-manager.mdx b/docs/how-tos/airflow/use-aws-secrets-manager.mdx index 9f466b0e..874ee79c 100644 --- a/docs/how-tos/airflow/use-aws-secrets-manager.mdx +++ b/docs/how-tos/airflow/use-aws-secrets-manager.mdx @@ -6,21 +6,19 @@ sidebar_position: 36 --- # How to use AWS Secrets Manager in Airflow -Datacoves integrates with the Airflow Secrets Backend Interface, offering support for its native Datacoves Secrets Backend, AWS Secrets Manager, and Azure Key Vault. For other Airflow-compatible Secrets Managers, please reach out to us. +Datacoves lets you pick the Airflow secrets backend for each project or environment: the built-in Datacoves Secrets Manager (the default), AWS Secrets Manager, Azure Key Vault, [GCP Secret Manager](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/secrets-backends/google-cloud-secret-manager-backend.html) or [HashiCorp Vault](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/secrets-backends/hashicorp-vault.html). The selected backend is queried directly by Airflow. Secrets backends can be configured at the project level, at the environment level, or both. See [configure your AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details. ## Read variable from AWS Secrets Manager -Airflow's `Variable.get` searches multiple places: +With AWS Secrets Manager selected as the secrets backend, Airflow's `Variable.get` searches multiple places: -1. AWS Secrets Manager (if configured) -2. Datacoves Secrets Manager -3. Airflow environment variables +1. AWS Secrets Manager +2. Airflow variables and environment variables Once a variable is found, Airflow stops searching. -![Secrets flowchart](assets/variablle_flow.png) ### Best practices @@ -64,7 +62,7 @@ To auto mask your secret you can use `secret` or `password` in the secret name s ## Using a secrets manager directly from Airflow -While not recommended, you can bypass the Datacoves secrets manager integration by configuring an Airflow connection and using the `SecretsManagerHook` in an Airflow DAG. +While not recommended, you can bypass the secrets backend integration by configuring an Airflow connection and using the `SecretsManagerHook` in an Airflow DAG. ### Configure an Airflow Connection Create a new Airflow Connection with the following parameters: diff --git a/docs/how-tos/airflow/use-azure-key-vault.mdx b/docs/how-tos/airflow/use-azure-key-vault.mdx index fc7fac12..44d92eff 100644 --- a/docs/how-tos/airflow/use-azure-key-vault.mdx +++ b/docs/how-tos/airflow/use-azure-key-vault.mdx @@ -6,7 +6,7 @@ sidebar_position: 37 --- # How to use Azure Key Vault in Airflow -Datacoves integrates with the Airflow Secrets Backend Interface, offering support for its native Datacoves Secrets Backend, AWS Secrets Manager, and Azure Key Vault. For other Airflow-compatible Secrets Managers, please reach out to us. +Datacoves lets you pick the Airflow secrets backend for each project or environment: the built-in Datacoves Secrets Manager (the default), AWS Secrets Manager, Azure Key Vault, [GCP Secret Manager](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/secrets-backends/google-cloud-secret-manager-backend.html) or [HashiCorp Vault](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/secrets-backends/hashicorp-vault.html). The selected backend is queried directly by Airflow. Secrets backends can be configured at the project level, at the environment level, or both. See [configure your Azure Key Vault](/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault) for details. @@ -14,28 +14,27 @@ When Datacoves runs in your Azure subscription, Airflow can authenticate to Key ## Read variable from Azure Key Vault -Airflow's `Variable.get` searches multiple places: +With Azure Key Vault selected as the secrets backend, Airflow's `Variable.get` searches multiple places: -1. Azure Key Vault (if configured) -2. Datacoves Secrets Manager -3. Airflow variables and environment variables +1. Azure Key Vault +2. Airflow variables and environment variables Once a variable is found, Airflow stops searching. ### Secret naming -Variable keys and connection ids must start with `datacoves-` for the lookup to be sent to Azure Key Vault. The Key Vault secret name is the variable key with the `airflow-variables-` prefix (or `airflow-connections-` for connections), and underscores are translated to dashes because Key Vault secret names only allow letters, numbers and dashes: +The Key Vault secret name is the variable key with the `airflow-variables-` prefix (or `airflow-connections-` for connections), and underscores are translated to dashes because Key Vault secret names only allow letters, numbers and dashes. No special prefix is required on the variable key itself: | In your DAG | Key Vault secret name | | --- | --- | -| `Variable.get("datacoves-my-secret")` | `airflow-variables-datacoves-my-secret` | -| `Variable.get("datacoves-my_secret")` | `airflow-variables-datacoves-my-secret` | -| Connection `datacoves-warehouse` | `airflow-connections-datacoves-warehouse` | +| `Variable.get("my-secret")` | `airflow-variables-my-secret` | +| `Variable.get("my_secret")` | `airflow-variables-my-secret` | +| Connection `warehouse` | `airflow-connections-warehouse` | ### Best practices 1. Call `Variable.get` from within an Airflow/Datacoves decorator to fetch at runtime only. Fetching at the top level of a DAG file would query Key Vault on every DAG parse. -2. Keep the `datacoves-` prefix on everything you store in Key Vault for Airflow; lookups without it never reach the vault. +2. Every variable and connection lookup reaches the vault before falling back to Airflow's own storage; see the [Azure Key Vault backend documentation](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/secrets-backends/azure-key-vault.html) for options like `connections_prefix` and `variables_prefix` if you want to scope what is looked up. ### Example DAG using Azure Key Vault @@ -68,7 +67,7 @@ def azure_key_vault_example(): def read_secret_from_key_vault(): # Fetch at runtime (inside the task), never at the top level of the # DAG file, so Key Vault is only called when the task runs. - my_var = Variable.get("datacoves-my-secret") + my_var = Variable.get("my-secret") print(f"Fetched a {len(my_var)} character value from Azure Key Vault") read_secret_from_key_vault() @@ -77,14 +76,14 @@ dag = azure_key_vault_example() ``` :::tip -To auto mask your secret you can use `secret` or `password` in the variable name since this will honor `hide_sensitive_var_conn_fields`. eg `datacoves-my-password`. Please see [this documentation](https://www.astronomer.io/docs/learn/airflow-variables#hide-sensitive-information-in-airflow-variables) for a full list of masking words. +To auto mask your secret you can use `secret` or `password` in the variable name since this will honor `hide_sensitive_var_conn_fields`. eg `my-password`. Please see [this documentation](https://www.astronomer.io/docs/learn/airflow-variables#hide-sensitive-information-in-airflow-variables) for a full list of masking words. ::: ## Using Azure Key Vault directly from Airflow -While not recommended, you can bypass the Datacoves secrets manager integration by configuring an Airflow connection and reading secrets with the Azure Key Vault SDK. The SDK (`azure-identity` and `azure-keyvault-secrets`) is already installed in Datacoves Airflow images as part of the Microsoft Azure provider. +While not recommended, you can bypass the secrets backend integration by configuring an Airflow connection and reading secrets with the Azure Key Vault SDK. The SDK (`azure-identity` and `azure-keyvault-secrets`) is already installed in Datacoves Airflow images as part of the Microsoft Azure provider. -When reading secrets this way, the [secret naming](#secret-naming) rules above do not apply: you fetch any Key Vault secret by its exact name, with no `airflow-variables-` or `datacoves-` prefix required. +When reading secrets this way, the [secret naming](#secret-naming) rules above do not apply: you fetch any Key Vault secret by its exact name, with no `airflow-variables-` prefix required. ### Configure an Airflow Connection diff --git a/docs/how-tos/airflow/use-datacoves-secrets-manager.mdx b/docs/how-tos/airflow/use-datacoves-secrets-manager.mdx index 531216d1..91e48397 100644 --- a/docs/how-tos/airflow/use-datacoves-secrets-manager.mdx +++ b/docs/how-tos/airflow/use-datacoves-secrets-manager.mdx @@ -7,7 +7,7 @@ sidebar_position: 38 # How to use Datacoves Secrets Manager in Airflow -Datacoves includes a built-in [Secrets Manager](/docs/reference/admin-menu/secrets) that allows you to securely store and manage secrets for both administrators and developers. Secrets can be stored at the project or environment level and easily shared across other tools in your stack, ensuring seamless integration and enhanced security. [Creating or editing a secret](/docs/how-tos/datacoves/how_to_secrets) in the Datacoves Secret Manager is straightforward. Be sure to prefix all secrets stored in Datacoves Secrets Manager with `datacoves-`. +Datacoves includes a built-in [Secrets Manager](/docs/reference/admin-menu/secrets), the default Airflow secrets backend, that allows you to securely store and manage secrets for both administrators and developers. Secrets can be stored at the project or environment level and easily shared across other tools in your stack, ensuring seamless integration and enhanced security. [Creating or editing a secret](/docs/how-tos/datacoves/how_to_secrets) in the Datacoves Secret Manager is straightforward. Be sure to prefix all secrets stored in Datacoves Secrets Manager with `datacoves-`. ## Read variable from Datacoves Secrets Manager diff --git a/docs/how-tos/datacoves/how_to_environments.md b/docs/how-tos/datacoves/how_to_environments.md index d682d017..fceeaebb 100644 --- a/docs/how-tos/datacoves/how_to_environments.md +++ b/docs/how-tos/datacoves/how_to_environments.md @@ -51,7 +51,7 @@ The services enabled for the environment may require additional configurations. - **dbt profiles path:** The location where Airflow will find dbt profiles.yml file to use during a dbt run. This should be `automate/dbt`. Please be aware that you will need to create the `automate` and`dbt` folders as well as the `profiles.yml` in your repository. - **YAML DAGs path:** When using yml based Airflow DAGs Airflow will look for the yml files in this location. We recommend this be set to `orchestrate/dags`. Please be aware that you will need to create the `orchestrate` and `dags` folders in your repository. - **Python DAGs path:** This is the location Airflow will look for the DAG definition files. We recommend this be set to `orchestrate/dag_yml_definitions`. Please be aware that you will need to create the `orchestrate` and `dag_yml_definitions` folders in your repository. - - **Additional Secrets Backend:** Allows you to configure AWS Secrets Manager for this specific environment. This can be set independently of the project-level configuration, or left as `Use Project Settings` to inherit it if one exists. See [Configure AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details. + - **Airflow Secrets Backend:** Selects the secrets backend Airflow uses for this specific environment: the Datacoves Secrets Manager (the default), AWS Secrets Manager, Azure Key Vault, GCP Secret Manager or HashiCorp Vault. Leave it as `Use Project Settings` to inherit the project-level selection. See [Configure AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) or [Configure Azure Key Vault](/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault) for details. - OBSERVE (Docs) requires: - **branch:** Here we specify the branch that will be synchronized for production dbt docs. This branch must exist in your git repository. diff --git a/docs/how-tos/datacoves/how_to_projects/README.md b/docs/how-tos/datacoves/how_to_projects/README.md index 51231c4d..92e757ee 100644 --- a/docs/how-tos/datacoves/how_to_projects/README.md +++ b/docs/how-tos/datacoves/how_to_projects/README.md @@ -71,6 +71,6 @@ See this [how-to guide on configuring Azure DevOps](/docs/how-tos/datacoves/how_ - **CI/CD Provider:** When provided, this will display a link to your CI/CD jobs on the Observe tab of a Datacoves environment. Once you choose your provider, you will be able to specify your `CI jobs home URL`. -- **Secrets Backend:** Datacoves provides a Secrets Backend out of the box; you can also configure additional Secrets Backends at the project level, at the environment level, or both. See [AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details. +- **Airflow Secrets Backend:** Datacoves provides a Secrets Backend out of the box (the Datacoves Secrets Manager); alternatively you can select AWS Secrets Manager, Azure Key Vault, GCP Secret Manager or HashiCorp Vault as the backend Airflow queries directly, at the project level, at the environment level, or both. See [AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) or [Azure Key Vault](/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault) for details. ![Project Secrets Backend](../assets/edit_project_secrets_backend.jpg) diff --git a/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md b/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md index 82851dc5..a3f839fa 100644 --- a/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md +++ b/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md @@ -1,11 +1,17 @@ --- -title: "Add AWS Secrets Manager as a Datacoves Backend" +title: "Use AWS Secrets Manager as the Airflow Secrets Backend" sidebar_label: "Configure AWS Secrets Manager" -description: "Connect AWS Secrets Manager to Datacoves as a project-level secrets backend to securely inject credentials into Airflow and VS Code environments." +description: "Select AWS Secrets Manager as the Airflow secrets backend in Datacoves so Airflow reads variables and connections directly from AWS, with access keys or an IAM role." sidebar_position: 50 --- # Configuring AWS Secrets Manager +Datacoves can use AWS Secrets Manager as the Airflow secrets backend so that Airflow fetches variables and connections directly from AWS at runtime. Secret values never pass through or get stored in Datacoves. + +:::note +Selecting an external backend replaces the Datacoves Secrets Manager for Airflow: secrets created in the Datacoves Secrets admin are no longer served to Airflow, and if no project in your account uses the Datacoves Secrets Manager, the `Secrets` item is hidden from the admin menu. +::: + ## Table of Contents - [Prereqs](#prereqs) - [Create your Secret in AWS Secrets Manager](#create-your-secret-in-aws-secrets-manager) @@ -82,7 +88,7 @@ This configuration applies to all environments under the project unless overridd ![Project](../assets/menu_projects.gif) -**Step 2:** Scroll down to the `Secrets` section and select `AWS Secrets Manager` from the `Additional Secrets Backend` dropdown. +**Step 2:** Scroll down to the `Airflow Secrets Backend` section and select `AWS Secrets Manager` from the `Secrets Backend` dropdown (the default selection, `Datacoves Secrets Manager`, is the built-in backend). ![Project Secrets Backend](../assets/edit_project_secrets_backend.jpg) @@ -104,7 +110,7 @@ This configuration applies to all environments under the project unless overridd To learn how to read a variable from the AWS Secrets Manager check out our [How To](/docs/category/how-tos) :::tip -For security purposes, once this has been saved you will not be able to view the values. To modify the Secrets backend you will need to set the Secrets backend to `None` and save the changes. Then start the setup again. +For security purposes, once this has been saved you will not be able to view the values. To modify the configuration, switch the Secrets Backend back to `Datacoves Secrets Manager`, save the changes, and then start the setup again. ::: ### Environment-level configuration @@ -115,7 +121,7 @@ AWS Secrets Manager can also be configured directly at the environment level, in **Step 2:** Go to **Services Configuration**, then select **Airflow settings**. -**Step 3:** Scroll down to the **Additional Secrets Backend** section. Select `AWS Secrets Manager` to configure it for this environment. If a project-level configuration exists and you want this environment to use it, leave the field set to `Use Project Settings`. +**Step 3:** Scroll down to the **Airflow Secrets Backend** section. Select `AWS Secrets Manager` to configure it for this environment. Leave the field set to `Use Project Settings` to inherit the project-level selection, or pick `Datacoves Secrets Manager` to force the built-in backend on this environment regardless of the project setting. ![Environment Secrets Backend Override](../assets/edit_environment_secrets_backend.png) @@ -156,7 +162,7 @@ IRSA requires the Datacoves platform team to enable workload identity for your e The Datacoves team can provide your cluster's OIDC issuer and confirm the environment slug. -**Step 2:** Once the Datacoves team confirms workload identity is enabled with your role, configure the `Additional Secrets Backend` exactly as described above but **omit** `aws_access_key_id` and `aws_secret_access_key`: +**Step 2:** Once the Datacoves team confirms workload identity is enabled with your role, configure the `Airflow Secrets Backend` exactly as described above but **omit** `aws_access_key_id` and `aws_secret_access_key`: ```json { diff --git a/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault.md b/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault.md index 9d185513..05270bf5 100644 --- a/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault.md +++ b/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault.md @@ -1,12 +1,16 @@ --- -title: "Add Azure Key Vault as a Datacoves Backend" +title: "Use Azure Key Vault as the Airflow Secrets Backend" sidebar_label: "Configure Azure Key Vault" -description: "Connect Azure Key Vault to Datacoves as a project-level secrets backend so Airflow reads variables and connections directly from your vault, using Managed Identity or a service principal." +description: "Select Azure Key Vault as the Airflow secrets backend in Datacoves so Airflow reads variables and connections directly from your vault, using Managed Identity or a service principal." sidebar_position: 52 --- # Configuring Azure Key Vault -Datacoves can chain Azure Key Vault behind the Datacoves Secrets Backend so that Airflow fetches variables and connections directly from your vault at runtime. Secret values never pass through or get stored in Datacoves; Airflow talks to Azure Key Vault directly from within your environment. +Datacoves can use Azure Key Vault as the Airflow secrets backend so that Airflow fetches variables and connections directly from your vault at runtime. Secret values never pass through or get stored in Datacoves; Airflow talks to Azure Key Vault directly from within your environment. + +:::note +Selecting an external backend replaces the Datacoves Secrets Manager for Airflow: secrets created in the Datacoves Secrets admin are no longer served to Airflow, and if no project in your account uses the Datacoves Secrets Manager, the `Secrets` item is hidden from the admin menu. +::: ## Table of Contents - [Prereqs](#prereqs) @@ -80,8 +84,8 @@ Airflow maps variables and connections to Key Vault secret names using a prefix: ### Things to note: -1. Variable keys and connection ids must start with `datacoves-`. The Datacoves Secrets Backend only forwards lookups with that prefix to the additional backend; anything else is resolved from Airflow's own variables and connections. For example, to use `Variable.get("datacoves-my-secret")` in a DAG, create a Key Vault secret named `airflow-variables-datacoves-my-secret`. -2. Azure Key Vault secret names only allow letters, numbers and dashes. Underscores in a variable key or connection id are automatically translated to dashes when building the secret name, so `Variable.get("datacoves-my_secret")` also reads the secret `airflow-variables-datacoves-my-secret`. +1. Any variable key or connection id is looked up in Key Vault first, and falls back to Airflow's own variables and connections when not found; no special prefix is required. For example, to use `Variable.get("my-secret")` in a DAG, create a Key Vault secret named `airflow-variables-my-secret`. +2. Azure Key Vault secret names only allow letters, numbers and dashes. Underscores in a variable key or connection id are automatically translated to dashes when building the secret name, so `Variable.get("my_secret")` also reads the secret `airflow-variables-my-secret`. 3. Store the value as plain text. For connections, store either an Airflow connection URI (for example `snowflake://user:password@account/`) or a JSON object with the connection fields. ## Configure your Secrets Backend @@ -94,7 +98,7 @@ This configuration applies to all environments under the project unless overridd **Step 1:** Navigate to the Projects Admin page and click on the edit icon for the desired project. -**Step 2:** Scroll down to the `Secrets` section and select `Azure Key Vault` from the `Additional Secrets Backend` dropdown. +**Step 2:** Scroll down to the `Airflow Secrets Backend` section and select `Azure Key Vault` from the `Secrets Backend` dropdown (the default selection, `Datacoves Secrets Manager`, is the built-in backend). **Step 3:** Paste the JSON configuration for the authentication method you chose above. @@ -103,7 +107,7 @@ See the [Azure Key Vault secrets backend documentation](https://airflow.apache.o ::: :::tip -For security purposes, once this has been saved you will not be able to view the values. To modify the Secrets backend you will need to set the Secrets backend to `None` and save the changes. Then start the setup again. +For security purposes, once this has been saved you will not be able to view the values. To modify the configuration, switch the Secrets Backend back to `Datacoves Secrets Manager`, save the changes, and then start the setup again. ::: ### Environment-level configuration @@ -114,7 +118,7 @@ Azure Key Vault can also be configured directly at the environment level, indepe **Step 2:** Go to **Services Configuration**, then select **Airflow settings**. -**Step 3:** Scroll down to the **Additional Secrets Backend** section. Select `Azure Key Vault` to configure it for this environment. If a project-level configuration exists and you want this environment to use it, leave the field set to `Use Project Settings`. +**Step 3:** Scroll down to the **Airflow Secrets Backend** section. Select `Azure Key Vault` to configure it for this environment. Leave the field set to `Use Project Settings` to inherit the project-level selection, or pick `Datacoves Secrets Manager` to force the built-in backend on this environment regardless of the project setting. :::note The configuration fields available at the environment level are the same as those at the project level. Any values entered here will take precedence over the project settings for this environment only. diff --git a/docs/reference/admin-menu/secrets.md b/docs/reference/admin-menu/secrets.md index 928c9382..8a091ae1 100644 --- a/docs/reference/admin-menu/secrets.md +++ b/docs/reference/admin-menu/secrets.md @@ -10,6 +10,10 @@ sidebar_position: 80 Secrets are used to manage confidential information that are used by tools running in VSCode, or services like Airflow. +:::note +The `Secrets` menu item is only shown when at least one project in the account uses the Datacoves Secrets Manager as its Airflow secrets backend. When every project uses an external backend (such as AWS Secrets Manager or Azure Key Vault), Datacoves secrets are not served to Airflow and the menu is hidden. +::: + Some uses could be: - Storing Airbyte connections credentials using `dbt-coves extract` and `dbt-coves load` commands.