[DCV-3996] Update secrets backend docs for direct backend selection

docs/how-tos/airflow/send-emails.md

@@ -96,6 +96,53 @@ Once you created the `SMTP` integration, it's time to add it to the Airflow serv
 
 - Click `Save Changes`. 
 
+## Test the integration with a simple DAG
+
+Datacoves turns the SMTP integration into the `smtp_default` Airflow connection (you can see it in Airflow under `Admin > Connections`), so no extra configuration is needed in your DAGs. Before wiring up failure notifications, you can verify the setup end to end with a DAG that just sends an email:
+
+```python
+from pendulum import datetime
+
+try:
+    # Airflow 3
+    from airflow.sdk import dag
+except ImportError:
+    # Airflow 2
+    from airflow.decorators import dag
+
+try:
+    # Airflow 3
+    from airflow.providers.smtp.operators.smtp import EmailOperator
+except ImportError:
+    # Airflow 2
+    from airflow.operators.email import EmailOperator
+
+
+@dag(
+    default_args={
+        "start_date": datetime(2024, 1, 1),
+        "owner": "Noel Gomez",  # Replace with your name
+    },
+    description="Send a test email through the SMTP integration",
+    schedule=None,
+    tags=["sample"],
+    catchup=False,
+)
+def smtp_test():
+
+    EmailOperator(
+        task_id="send_test_email",
+        to="gomezn@example.com",  # Replace with the recipient
+        subject="SMTP test from Airflow",
+        html_content="<b>It works!</b> Sent through the smtp_default connection.",
+    )
+
+
+dag = smtp_test()
+```
+
+Trigger it manually from the Airflow UI; the recipient should get the test email within a few seconds.
+
 ## Implement in a DAG
 
 If you have already created a DAG it's time to modify your DAG to make use of our newly set up SMTP integration on Airflow. 

docs/how-tos/airflow/use-aws-secrets-manager.mdx

@@ -6,21 +6,19 @@ sidebar_position: 36
 ---
 # How to use AWS Secrets Manager in Airflow
 
-Datacoves integrates with the Airflow Secrets Backend Interface, offering support for its native Datacoves Secrets Backend, AWS Secrets Manager, and Azure Key Vault. For other Airflow-compatible Secrets Managers, please reach out to us.
+Datacoves lets you pick the Airflow secrets backend for each project or environment: the built-in Datacoves Secrets Manager (the default), AWS Secrets Manager, Azure Key Vault, [GCP Secret Manager](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/secrets-backends/google-cloud-secret-manager-backend.html) or [HashiCorp Vault](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/secrets-backends/hashicorp-vault.html). The selected backend is queried directly by Airflow.
 
 Secrets backends can be configured at the project level, at the environment level, or both. See [configure your AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details.
 
 ## Read variable from AWS Secrets Manager
 
-Airflow's `Variable.get` searches multiple places:
+With AWS Secrets Manager selected as the secrets backend, Airflow's `Variable.get` searches multiple places:
 
-1. AWS Secrets Manager (if configured)  
-2. Datacoves Secrets Manager  
-3. Airflow environment variables  
+1. AWS Secrets Manager
+2. Airflow variables and environment variables
 
 Once a variable is found, Airflow stops searching.
 
-![Secrets flowchart](assets/variablle_flow.png)
 
 ### Best practices
 
@@ -64,7 +62,7 @@ To auto mask your secret you can use `secret` or `password` in the secret name s
 
 ## Using a secrets manager directly from Airflow
 
-While not recommended, you can bypass the Datacoves secrets manager integration by configuring an Airflow connection and using the `SecretsManagerHook` in an Airflow DAG.
+While not recommended, you can bypass the secrets backend integration by configuring an Airflow connection and using the `SecretsManagerHook` in an Airflow DAG.
 
 ### Configure an Airflow Connection
 Create a new Airflow Connection with the following parameters:

docs/how-tos/airflow/use-azure-key-vault.mdx

@@ -6,36 +6,35 @@ sidebar_position: 37
 ---
 # How to use Azure Key Vault in Airflow
 
-Datacoves integrates with the Airflow Secrets Backend Interface, offering support for its native Datacoves Secrets Backend, AWS Secrets Manager, and Azure Key Vault. For other Airflow-compatible Secrets Managers, please reach out to us.
+Datacoves lets you pick the Airflow secrets backend for each project or environment: the built-in Datacoves Secrets Manager (the default), AWS Secrets Manager, Azure Key Vault, [GCP Secret Manager](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/secrets-backends/google-cloud-secret-manager-backend.html) or [HashiCorp Vault](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/secrets-backends/hashicorp-vault.html). The selected backend is queried directly by Airflow.
 
 Secrets backends can be configured at the project level, at the environment level, or both. See [configure your Azure Key Vault](/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault) for details.
 
 When Datacoves runs in your Azure subscription, Airflow can authenticate to Key Vault with the cluster's Managed Identity, so no credentials need to be stored anywhere. Alternatively, a service principal with a client secret can be used on any Datacoves deployment.
 
 ## Read variable from Azure Key Vault
 
-Airflow's `Variable.get` searches multiple places:
+With Azure Key Vault selected as the secrets backend, Airflow's `Variable.get` searches multiple places:
 
-1. Azure Key Vault (if configured)
-2. Datacoves Secrets Manager
-3. Airflow variables and environment variables
+1. Azure Key Vault
+2. Airflow variables and environment variables
 
 Once a variable is found, Airflow stops searching.
 
 ### Secret naming
 
-Variable keys and connection ids must start with `datacoves-` for the lookup to be sent to Azure Key Vault. The Key Vault secret name is the variable key with the `airflow-variables-` prefix (or `airflow-connections-` for connections), and underscores are translated to dashes because Key Vault secret names only allow letters, numbers and dashes:
+The Key Vault secret name is the variable key with the `airflow-variables-` prefix (or `airflow-connections-` for connections), and underscores are translated to dashes because Key Vault secret names only allow letters, numbers and dashes. No special prefix is required on the variable key itself:
 
 | In your DAG | Key Vault secret name |
 | --- | --- |
-| `Variable.get("datacoves-my-secret")` | `airflow-variables-datacoves-my-secret` |
-| `Variable.get("datacoves-my_secret")` | `airflow-variables-datacoves-my-secret` |
-| Connection `datacoves-warehouse` | `airflow-connections-datacoves-warehouse` |
+| `Variable.get("my-secret")` | `airflow-variables-my-secret` |
+| `Variable.get("my_secret")` | `airflow-variables-my-secret` |
+| Connection `warehouse` | `airflow-connections-warehouse` |
 
 ### Best practices
 
 1. Call `Variable.get` from within an Airflow/Datacoves decorator to fetch at runtime only. Fetching at the top level of a DAG file would query Key Vault on every DAG parse.
-2. Keep the `datacoves-` prefix on everything you store in Key Vault for Airflow; lookups without it never reach the vault.
+2. Every variable and connection lookup reaches the vault before falling back to Airflow's own storage; see the [Azure Key Vault backend documentation](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/secrets-backends/azure-key-vault.html) for options like `connections_prefix` and `variables_prefix` if you want to scope what is looked up.
 
 ### Example DAG using Azure Key Vault
 
@@ -68,7 +67,7 @@ def azure_key_vault_example():
     def read_secret_from_key_vault():
         # Fetch at runtime (inside the task), never at the top level of the
         # DAG file, so Key Vault is only called when the task runs.
-        my_var = Variable.get("datacoves-my-secret")
+        my_var = Variable.get("my-secret")
         print(f"Fetched a {len(my_var)} character value from Azure Key Vault")
 
     read_secret_from_key_vault()
@@ -77,14 +76,14 @@ dag = azure_key_vault_example()
 ```
 
 :::tip
-To auto mask your secret you can use `secret` or `password` in the variable name since this will honor `hide_sensitive_var_conn_fields`. eg `datacoves-my-password`. Please see [this documentation](https://www.astronomer.io/docs/learn/airflow-variables#hide-sensitive-information-in-airflow-variables) for a full list of masking words.
+To auto mask your secret you can use `secret` or `password` in the variable name since this will honor `hide_sensitive_var_conn_fields`. eg `my-password`. Please see [this documentation](https://www.astronomer.io/docs/learn/airflow-variables#hide-sensitive-information-in-airflow-variables) for a full list of masking words.
 :::
 
 ## Using Azure Key Vault directly from Airflow
 
-While not recommended, you can bypass the Datacoves secrets manager integration by configuring an Airflow connection and reading secrets with the Azure Key Vault SDK. The SDK (`azure-identity` and `azure-keyvault-secrets`) is already installed in Datacoves Airflow images as part of the Microsoft Azure provider.
+While not recommended, you can bypass the secrets backend integration by configuring an Airflow connection and reading secrets with the Azure Key Vault SDK. The SDK (`azure-identity` and `azure-keyvault-secrets`) is already installed in Datacoves Airflow images as part of the Microsoft Azure provider.
 
-When reading secrets this way, the [secret naming](#secret-naming) rules above do not apply: you fetch any Key Vault secret by its exact name, with no `airflow-variables-` or `datacoves-` prefix required.
+When reading secrets this way, the [secret naming](#secret-naming) rules above do not apply: you fetch any Key Vault secret by its exact name, with no `airflow-variables-` prefix required.
 
 ### Configure an Airflow Connection
 

docs/how-tos/airflow/use-datacoves-secrets-manager.mdx

@@ -7,7 +7,7 @@ sidebar_position: 38
 
 # How to use Datacoves Secrets Manager in Airflow
 
-Datacoves includes a built-in [Secrets Manager](/docs/reference/admin-menu/secrets) that allows you to securely store and manage secrets for both administrators and developers. Secrets can be stored at the project or environment level and easily shared across other tools in your stack, ensuring seamless integration and enhanced security. [Creating or editing a secret](/docs/how-tos/datacoves/how_to_secrets) in the Datacoves Secret Manager is straightforward. Be sure to prefix all secrets stored in Datacoves Secrets Manager with `datacoves-`.
+Datacoves includes a built-in [Secrets Manager](/docs/reference/admin-menu/secrets), the default Airflow secrets backend, that allows you to securely store and manage secrets for both administrators and developers. Secrets can be stored at the project or environment level and easily shared across other tools in your stack, ensuring seamless integration and enhanced security. [Creating or editing a secret](/docs/how-tos/datacoves/how_to_secrets) in the Datacoves Secret Manager is straightforward. Be sure to prefix all secrets stored in Datacoves Secrets Manager with `datacoves-`.
 
 ## Read variable from Datacoves Secrets Manager
 

docs/how-tos/datacoves/how_to_environments.md

@@ -51,7 +51,7 @@ The services enabled for the environment may require additional configurations.
   - **dbt profiles path:** The location where Airflow will find dbt profiles.yml file to use during a dbt run. This should be `automate/dbt`. Please be aware that you will need to create the `automate` and`dbt` folders as well as the `profiles.yml` in your repository. 
   - **YAML DAGs path:** When using yml based Airflow DAGs Airflow will look for the yml files in this location. We recommend this be set to `orchestrate/dags`. Please be aware that you will need to create the `orchestrate` and `dags` folders in your repository.
   - **Python DAGs path:** This is the location Airflow will look for the DAG definition files.  We recommend this be set to `orchestrate/dag_yml_definitions`. Please be aware that you will need to create the `orchestrate` and `dag_yml_definitions` folders in your repository.
-  - **Additional Secrets Backend:** Allows you to configure AWS Secrets Manager for this specific environment. This can be set independently of the project-level configuration, or left as `Use Project Settings` to inherit it if one exists. See [Configure AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details.
+  - **Airflow Secrets Backend:** Selects the secrets backend Airflow uses for this specific environment: the Datacoves Secrets Manager (the default), AWS Secrets Manager, Azure Key Vault, GCP Secret Manager or HashiCorp Vault. Leave it as `Use Project Settings` to inherit the project-level selection. See [Configure AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) or [Configure Azure Key Vault](/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault) for details.
 - OBSERVE (Docs) requires:
   -  **branch:** Here we specify the branch that will be synchronized for production dbt docs. This branch must exist in your git repository.
 

docs/how-tos/datacoves/how_to_projects/README.md

@@ -71,6 +71,6 @@ See this [how-to guide on configuring Azure DevOps](/docs/how-tos/datacoves/how_
 </Tabs>
 
 - **CI/CD Provider:** When provided, this will display a link to your CI/CD jobs on the Observe tab of a Datacoves environment. Once you choose your provider, you will be able to specify your `CI jobs home URL`.
-- **Secrets Backend:** Datacoves provides a Secrets Backend out of the box; you can also configure additional Secrets Backends at the project level, at the environment level, or both. See [AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details.
+- **Airflow Secrets Backend:** Datacoves provides a Secrets Backend out of the box (the Datacoves Secrets Manager); alternatively you can select AWS Secrets Manager, Azure Key Vault, GCP Secret Manager or HashiCorp Vault as the backend Airflow queries directly, at the project level, at the environment level, or both. See [AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) or [Azure Key Vault](/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault) for details.
 
 ![Project Secrets Backend](../assets/edit_project_secrets_backend.jpg)

docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager.md

@@ -1,11 +1,17 @@
 ---
-title: "Add AWS Secrets Manager as a Datacoves Backend"
+title: "Use AWS Secrets Manager as the Airflow Secrets Backend"
 sidebar_label: "Configure AWS Secrets Manager"
-description: "Connect AWS Secrets Manager to Datacoves as a project-level secrets backend to securely inject credentials into Airflow and VS Code environments."
+description: "Select AWS Secrets Manager as the Airflow secrets backend in Datacoves so Airflow reads variables and connections directly from AWS, with access keys or an IAM role."
 sidebar_position: 50
 ---
 # Configuring AWS Secrets Manager
 
+Datacoves can use AWS Secrets Manager as the Airflow secrets backend so that Airflow fetches variables and connections directly from AWS at runtime. Secret values never pass through or get stored in Datacoves.
+
+:::note
+Selecting an external backend replaces the Datacoves Secrets Manager for Airflow: secrets created in the Datacoves Secrets admin are no longer served to Airflow, and if no project in your account uses the Datacoves Secrets Manager, the `Secrets` item is hidden from the admin menu.
+:::
+
 ## Table of Contents
 - [Prereqs](#prereqs)
 - [Create your Secret in AWS Secrets Manager](#create-your-secret-in-aws-secrets-manager)
@@ -82,7 +88,7 @@ This configuration applies to all environments under the project unless overridd
 
 ![Project](../assets/menu_projects.gif)
 
-**Step 2:** Scroll down to the `Secrets` section and select `AWS Secrets Manager` from the `Additional Secrets Backend` dropdown.
+**Step 2:** Scroll down to the `Airflow Secrets Backend` section and select `AWS Secrets Manager` from the `Secrets Backend` dropdown (the default selection, `Datacoves Secrets Manager`, is the built-in backend).
 
 ![Project Secrets Backend](../assets/edit_project_secrets_backend.jpg)
 
@@ -104,7 +110,7 @@ This configuration applies to all environments under the project unless overridd
 To learn how to read a variable from the AWS Secrets Manager check out our [How To](/docs/category/how-tos)
 
 :::tip
-For security purposes, once this has been saved you will not be able to view the values. To modify the Secrets backend you will need to set the Secrets backend to `None` and save the changes. Then start the setup again.
+For security purposes, once this has been saved you will not be able to view the values. To modify the configuration, switch the Secrets Backend back to `Datacoves Secrets Manager`, save the changes, and then start the setup again.
 :::
 
 ### Environment-level configuration
@@ -115,7 +121,7 @@ AWS Secrets Manager can also be configured directly at the environment level, in
 
 **Step 2:** Go to **Services Configuration**, then select **Airflow settings**.
 
-**Step 3:** Scroll down to the **Additional Secrets Backend** section. Select `AWS Secrets Manager` to configure it for this environment. If a project-level configuration exists and you want this environment to use it, leave the field set to `Use Project Settings`.
+**Step 3:** Scroll down to the **Airflow Secrets Backend** section. Select `AWS Secrets Manager` to configure it for this environment. Leave the field set to `Use Project Settings` to inherit the project-level selection, or pick `Datacoves Secrets Manager` to force the built-in backend on this environment regardless of the project setting.
 
 ![Environment Secrets Backend Override](../assets/edit_environment_secrets_backend.png) 
 
@@ -156,7 +162,7 @@ IRSA requires the Datacoves platform team to enable workload identity for your e
 
 The Datacoves team can provide your cluster's OIDC issuer and confirm the environment slug.
 
-**Step 2:** Once the Datacoves team confirms workload identity is enabled with your role, configure the `Additional Secrets Backend` exactly as described above but **omit** `aws_access_key_id` and `aws_secret_access_key`:
+**Step 2:** Once the Datacoves team confirms workload identity is enabled with your role, configure the `Airflow Secrets Backend` exactly as described above but **omit** `aws_access_key_id` and `aws_secret_access_key`:
 
 ```json
 {