diff --git a/docs/content/docs/examples/NetBox/Export Template/_index.md b/docs/content/docs/examples/NetBox/Export Template/_index.md new file mode 100644 index 00000000..2cc48b3b --- /dev/null +++ b/docs/content/docs/examples/NetBox/Export Template/_index.md @@ -0,0 +1,399 @@ +--- +title: "NetBox (Export Template)" +linkTitle: "NetBox Export Template" +weight: 1 +description: > + Discover targets from NetBox using HTTP provider with NetBox Export Template +--- + +This guide shows how to use **NetBox Export Templates** with the HTTP provider to discover and sync targets. + +Export Templates offer powerful filtering, transformation, and formatting directly in NetBox, reducing the load on the operator. + +## Overview + +An **Export Template** is a Jinja2 template defined in NetBox that: + +1. **Queries** NetBox's internal database (devices, interfaces, etc.) +2. **Filters** results based on custom criteria +3. **Transforms** data into your desired output format (JSON, YAML, CSV, etc.) +4. **Returns** the formatted output via a custom REST API endpoint + +When used with gNMIc's HTTP provider, the operator simply fetches the rendered JSON template and parses the result — no additional gNMIc Operator transformation needed if done correctly. + +--- + +## Prerequisites + +- A running Kubernetes cluster with gNMIc Operator installed +- `kubectl` access to your cluster +- A reachable NetBox instance with permissions to create Export Templates +- A NetBox API token +- Familiarity with Jinja2 templates + +--- + +## Step 1: Create a NetBox API Token and Store It Securely + +### Step 1a: Create the API Token in NetBox + +Create a dedicated API token in NetBox for gNMIc Operator access. + +1. Log in to NetBox. +2. Open your user profile or go to **User > API Tokens**. +3. Click **Add** or **Add token**. +4. Enter a descriptive name such as `gNMIc Operator`. +5. Grant the minimum permissions required for read-only device discovery. +6. Copy the token value and store it safely; NetBox will not show it again. + +### Step 1b: Store the Token in a Kubernetes Secret + +Create a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) containing the token. + +```bash +# Substitute YOUR_NETBOX_API_TOKEN with your actual token +# Bearer Token Format (v2): nbt_. +kubectl create secret generic netbox-api-token \ + --from-literal=token=YOUR_NETBOX_API_TOKEN \ + -n gnmic-system +``` + +Verify the Secret was created: + +```bash +kubectl get secret netbox-api-token -n gnmic-system -o yaml +``` + +--- + +## Step 2: Create an Export Template in NetBox + +Log in to your NetBox instance and navigate to **Customization > Export Templates**. + +### Step 2a: Create a New Template + +Click **Add Export Template** and fill in the details: + +| Field | Value | Notes | +|-------|-------|-------| +| **Name** | `gNMIc Device Export` | Descriptive name for your template | +| **Content Type** | `dcim > device` | Export template applies to Device objects | +| **Template Code** | (see below) | Jinja2 template | +| **File Extension** | `json` | Output format | +| **Mime Type** | `application/json` | Correct MIME type for JSON | + +### Step 2b: Template Code Example + +The following Export Templates only work for devices that have a primary IPv4 address set in NetBox. If primary_ip4 is missing, the expression returns '', so those devices will not yield a valid target address. For NetBox data model details, see the [NetBox Devices Data Model](https://netboxlabs.com/docs/netbox/models/dcim/device/) documentation. + +See the HTTP provider's "Default Response Format" section for the expected JSON structure: [HTTP provider docs](../../user-guide/targetsource/providers/http.md) + +#### Basic Template (All Devices) + +```jinja2 +[ + {% for device in queryset %} + { + "name": "{{ device.name }}", + "address": "{{ device.primary_ip4.address.ip }}", + "labels": { + "site": "{{ device.site.name }}", + "role": "{{ device.role.name }}", + "region": "{{ device.site.region.name }}", + "type": "{{ device.device_type.model }}" + } + }{{ "," if not loop.last }} + {% endfor %} +] +``` + +#### Advanced Template (Filtered by Status and Role) + +```jinja2 +[ + {% for device in queryset.filter(status='active', role__name__in=['leaf', 'spine']) %} + { + "name": "{{ device.name }}", + "address": "{{ device.primary_ip4.address.ip }}", + "labels": { + "site": "{{ device.site.name }}", + "role": "{{ device.role.name }}", + "region": "{{ device.site.region.name }}", + "model": "{{ device.device_type.model }}", + "serial": "{{ device.serial }}", + "asset_tag": "{{ device.asset_tag }}" + } + }{{ "," if not loop.last }} + {% endfor %} +] +``` + +**Key template elements:** + +- `queryset`: The filtered set of devices (all unless you add `.filter()`) +- `device.name`: Device hostname +- `device.primary_ip4.address.ip`: Primary IPv4 address +- `device.site.name`, `device.device_role.name`: NetBox relationships (site, role, etc.) +- `loop.last`: Jinja2 loop variable to avoid trailing comma on last item + +### Step 2c: Save and Access the Template + +Once saved, NetBox exposes the template via: + +``` +http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc+Device+Export +``` + +Or fetch it directly: + +```bash +# Replace with your NetBox URL and template name +# Substitute YOUR_NETBOX_API_TOKEN with your actual token +# Bearer Token Format (v2): nbt_. +curl -H "Authorization: Bearer YOUR_NETBOX_API_TOKEN" \ + "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" +``` + +The response should be a JSON array of targets ready for the gNMIc Operator. + +Sample JSON output produced by the basic export template: + +```json +[ + + { + "name": "edge-rtr-01.dc1.example.com", + "address": "203.0.113.1", + "labels": { + "site": "DC1", + "role": "edge", + "region": "eu-central-1", + "type": "router" + } + }, + +] +``` + +> Ensure the response is valid JSON and contains no hidden or invalid characters, otherwise the gNMIc Operator will fail to parse it. + +> If you instead return a JSON object with a nested array, add a mapping section such as `targetsField: "self.targets"` to the TargetSource CR. + +--- + +## Step 3: Create a TargetProfile + +Define how discovered targets should be configured. The `TargetProfile` points to a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) containing device credentials, such as username/password or client certificates. + +Create a credentials Secret first, then reference it from the TargetProfile. + +```yaml +# Replace YOUR_DEVICE_USERNAME and YOUR_DEVICE_PASSWORD with your corresponding default device username and password +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: gnmic-system +type: Opaque +stringData: + username: YOUR_DEVICE_USERNAME + password: YOUR_DEVICE_PASSWORD +``` + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetProfile +metadata: + name: netbox-device + namespace: gnmic-system +spec: + credentialsRef: device-credentials + timeout: 10s +``` + +For more TargetProfile options and credential handling, see the operator documentation for `TargetProfile`. + +--- + +## Step 4: Create a TargetSource Using Export Template + +Create a `TargetSource` that references your NetBox export template endpoint: + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: netbox-export-source + namespace: gnmic-system +spec: + targetPort: 57400 + targetProfile: netbox-device + targetLabels: + inventory: netbox + sync-source: export-template + provider: + http: + url: "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" + method: GET + interval: 30m + timeout: 30s + authentication: + token: + scheme: Token + tokenSecretRef: + name: netbox-api-token + key: token +``` + +--- + +## Step 5: Verify Target Discovery + +Once the `TargetSource` is deployed, verify that targets are being discovered: + +```bash +# List discovered targets +kubectl get targets -n gnmic-system + +# Check TargetSource status and sync details +kubectl describe targetsource netbox-export-source -n gnmic-system +``` + +Successful sync shows: + +- `status.status`: "success" (or similar) +- `status.targetsCount`: number of devices +- `status.lastSync`: recent timestamp + +--- + +## Example: Complete Setup + +Here's a full example combining all components: + +```yaml +--- +# Secret for NetBox API token +apiVersion: v1 +kind: Secret +metadata: + name: netbox-api-token + namespace: gnmic-system +type: Opaque +data: + # base64-encoded token (echo -n "YOUR_TOKEN" | base64) + token: YOUR_BASE64_ENCODED_TOKEN + +--- +# Secret for Target Credential +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: gnmic-system +type: Opaque +stringData: + username: YOUR_DEVICE_USERNAME + password: YOUR_DEVICE_PASSWORD + +--- +# TargetProfile +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetProfile +metadata: + name: netbox-device + namespace: gnmic-system +spec: + credentialsRef: device-credentials + timeout: 10s + + +--- +# TargetSource using Export Template +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: netbox-export-source + namespace: gnmic-system +spec: + targetPort: 57400 + targetProfile: netbox-device + targetLabels: + inventory: netbox + sync-source: export-template + provider: + http: + url: "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" + method: GET + interval: 30m + timeout: 30s + authentication: + token: + scheme: Token + tokenSecretRef: + name: netbox-api-token + key: token +``` + +--- + +## Advantages of Export Templates + +- **Powerful Filtering**: Filter devices by site, status, role, tags, etc. directly in NetBox +- **Reduced Operator Load**: NetBox handles data transformation; operator just fetches JSON +- **Reusability**: One template can serve multiple consumers +- **Maintainability**: Update discovery logic in NetBox without changing Kubernetes manifests +- **Performance**: Avoids REST API pagination for large inventories + +--- + +## Limitations & Considerations + +### 1. Reverse Proxy and URL Path Rewriting + +If NetBox is behind a reverse proxy with URL path rewriting: + +- **Issue**: The export template endpoint uses query parameters that may not survive proxy transformation. +- **Solution**: + - Ensure the proxy preserves query strings exactly. + - Test the export URL directly: + ```bash + curl -H "Authorization: Token YOUR_TOKEN" \ + "http://netbox.example.com:8000/api/dcim/devices/?export=gNMIc%20Device%20Export" + ``` + - If the proxy blocks or modifies parameters, consider using a direct NetBox endpoint without proxying. + +### 2. Large Inventory Rendering + +- Very large device counts can cause NetBox to take time rendering the template. +- **Solution**: + - Use `.filter()` in your template to limit results. + - Create separate export templates for different device groups (e.g., by site or role). + +### 3. Complex Jinja2 Logic + +- NetBox's Jinja2 sandbox restricts some Python functions for security. +- **Solution**: Keep templates simple and use NetBox's built-in filters and objects. Test the URL with curl or similar before deploying. + +--- + +## Template Troubleshooting + +### Missing Targets in Kubernetes + +- **Check**: Are all required fields populated in NetBox? (e.g., `primary_ip4` may be `None` if not set) +- **Solution**: Add conditional checks: + ```jinja2 + {% if device.primary_ip4 %} + "address": "{{ device.primary_ip4.address.ip }}" + {% endif %} + ``` + +### Authorization Fails + +If you get a 403 error: + +- Verify the token is valid and not expired. +- Ensure the API token is enabled. + +--- diff --git a/docs/content/docs/examples/NetBox/REST API/_index.md b/docs/content/docs/examples/NetBox/REST API/_index.md new file mode 100644 index 00000000..18111d75 --- /dev/null +++ b/docs/content/docs/examples/NetBox/REST API/_index.md @@ -0,0 +1,319 @@ +--- +title: "NetBox (REST API)" +linkTitle: "NetBox REST" +weight: 2 +description: > + Discover targets from NetBox using the HTTP provider and NetBox REST API +--- + +This guide shows how to configure the HTTP provider to discover targets from NetBox using its REST API. + +The REST API approach is direct and straightforward — query NetBox's standard API endpoints to retrieve devices that match your criteria. + +## Prerequisites + +- A running Kubernetes cluster with gNMIc Operator installed +- `kubectl` access to your cluster +- A reachable NetBox instance (inside or outside the cluster) +- A NetBox API token + +## Overview + +The HTTP `TargetSource` loader performs these steps: + +1. **Fetch** JSON device data from a NetBox REST API endpoint (`/api/dcim/devices/`) +2. **Transform** each device record into a gNMIc target using CEL expressions +3. **Create** or **update** `Target` resources in Kubernetes with the extracted data + +--- + +## Step 1: Create a NetBox API Token and Store It Securely + +### Step 1a: Create the API Token in NetBox + +Create a dedicated API token in NetBox for gNMIc Operator access. + +1. Log in to NetBox. +2. Open your user profile or go to **User > API Tokens**. +3. Click **Add** or **Add token**. +4. Enter a descriptive name such as `gNMIc Operator`. +5. Grant the minimum permissions required for read-only device discovery. +6. Copy the token value and store it safely; NetBox will not show it again. + +### Step 1b: Store the Token in a Kubernetes Secret + +Create a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) containing the token so it is not embedded in manifests. + +```bash +# Substitute YOUR_NETBOX_API_TOKEN with your actual token +# Bearer Token Format (v2): nbt_. +kubectl create secret generic netbox-api-token \ + --from-literal=token=YOUR_NETBOX_API_TOKEN \ + -n gnmic-system +``` + +Verify the Secret was created: + +```bash +kubectl get secret netbox-api-token -n gnmic-system -o yaml +``` + +--- + +## Step 2: Create a TargetProfile + +Define how discovered targets should be configured. The `TargetProfile` points to a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) containing device credentials, such as username/password or client certificates. + +Create a credentials Secret first, then reference it from the profile. + +```yaml +# Replace YOUR_DEVICE_USERNAME and YOUR_DEVICE_PASSWORD with your corresponding default device username and password +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: gnmic-system +type: Opaque +stringData: + username: YOUR_DEVICE_USERNAME + password: YOUR_DEVICE_PASSWORD +``` + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetProfile +metadata: + name: netbox-device + namespace: gnmic-system +spec: + credentialsRef: device-credentials + timeout: 10s +``` + +For more TargetProfile options and credential handling, see the operator documentation for `TargetProfile`. + +--- + +## Step 3: Create a TargetSource Using REST API + +The following `TargetSource` queries NetBox's REST API to discover devices: + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: netbox-rest-source + namespace: gnmic-system +spec: + targetPort: 57400 + targetProfile: netbox-device + targetLabels: + inventory: netbox + sync-source: rest-api + provider: + http: + url: "http://netbox.example.com:8000/api/dcim/devices/?limit=1000" + method: GET + interval: 5m + timeout: 30s + authentication: + token: + scheme: Bearer + tokenSecretRef: + name: netbox-api-token + key: token + pagination: + nextField: "next" + mapping: + targetsField: "self.results" + address: "item.primary_ip4 != null ? item.primary_ip4.address.split('/')[0] : ''" + labels: | + { + "site": item.site.name, + "role": item.device_role.name, + "model": item.device_type.model, + "status": item.status.value + } +``` + +> This mapping only works for devices that have a primary IPv4 address set in NetBox. If primary_ip4 is missing, the expression returns '', so those devices will not yield a valid target address. For NetBox API details, see the [NetBox REST API](https://netboxlabs.com/docs/netbox/integrations/rest-api/) documentation. + +The HTTP loader supports `targetsField` and individual CEL expressions for `name`, `address`, `port`, `labels`, and `targetProfile`. See the HTTP Provider docs "Response Mapping via CEL" section for more details: [HTTP provider docs](../../user-guide/targetsource/providers/http.md) + +Use `self` for the full response and `item` for each candidate object. + +--- + +## Step 4: Apply and Verify Target Discovery + +Deploy the `TargetSource` and check that targets are being discovered and synced: + +```bash +# List discovered targets +kubectl apply -f /path/to/targetsource.yaml -n gnmic-system + +# List discovered targets +kubectl get targets -n gnmic-system + +# Check TargetSource status +kubectl describe targetsource netbox-rest-source -n gnmic-system +``` + +Look for: +- `status.status`: "success" (or similar) +- `status.targetsCount`: number of discovered devices +- `status.lastSync`: recent timestamp + +--- + +## Example: Complete Setup + +Here's a complete example combining all resources: + +```yaml +--- +# Secret for NetBox API token +apiVersion: v1 +kind: Secret +metadata: + name: netbox-api-token + namespace: gnmic-system +type: Opaque +data: + # base64-encoded token (echo -n "YOUR_TOKEN" | base64) + token: YOUR_BASE64_ENCODED_TOKEN + +--- +# Secret for Target Credential +apiVersion: v1 +kind: Secret +metadata: + name: device-credentials + namespace: gnmic-system +type: Opaque +stringData: + username: YOUR_DEVICE_USERNAME + password: YOUR_DEVICE_PASSWORD + +--- +# TargetProfile +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetProfile +metadata: + name: netbox-device + namespace: gnmic-system +spec: + credentialsRef: device-credentials + timeout: 10s + +--- +# TargetSource with REST API +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: netbox-rest-source + namespace: gnmic-system +spec: + targetPort: 57400 + targetProfile: netbox-device + targetLabels: + inventory: netbox + sync-source: rest-api + provider: + http: + url: "http://netbox.example.com:8000/api/dcim/devices/?limit=1000" + method: GET + interval: 5m + timeout: 30s + authentication: + token: + scheme: Bearer + tokenSecretRef: + name: netbox-api-token + key: token + pagination: + nextField: "next" + mapping: + targetsField: "self.results" + address: "item.primary_ip4 != null ? item.primary_ip4.address.split('/')[0] : ''" + labels: | + { + "site": item.site.name, + "role": item.device_role.name, + "model": item.device_type.model, + "status": item.status.value + } +``` + +--- + +## Performance Considerations & Limitations + +### REST API Query Limits + +- **Query Size**: The example uses `limit=1000`. Adjust based on your NetBox instance's pagination settings and response size limits. +- **Response Timeout**: Large device lists can take time. Set appropriate timeouts in your `TargetSource`. + +### Reverse Proxy Considerations + +If NetBox is behind a reverse proxy: + +- **Base URL**: Ensure the reverse proxy correctly handles the `/api/dcim/devices/` path. +- **Authentication**: Some proxies may require additional headers; verify with your proxy and NetBox admin. +- **HTTPS**: If using HTTPS, ensure certificates are trusted by the operator or else use the `tls` setting. + +### Large Inventories + +For inventories with thousands of devices: + +- Consider using **Export Templates** (see [NetBox Export Templates]({{< relref "../Export Template" >}})) for better filtering and performance. +- Implement filtering in the REST API URL (e.g., `?site=us-west&status=active`). + +--- + +## Security Considerations + +### Token and Credentials + +- **Never** embed plaintext tokens or credentials in manifests or YAML files. +- Always store tokens in Kubernetes Secrets. +- Restrict RBAC permissions on the Secret to only necessary service accounts. + +### HTTPS and Certificates + +If connecting to NetBox via HTTPS: + +- Ensure cluster DNS resolves the hostname correctly. +- Mount CA certificates if using self-signed certificates. +- Verify the operator's HTTP client configuration for certificate validation. + +--- + +## Troubleshooting + +### Show TargetSource Errors + +```bash +kubectl describe targetsource netbox-rest-source -n gnmic-system +``` + +### Targets Not Appearing + +- Check that the `TargetProfile` exists and is correctly referenced. +- Verify labels and addresses are being extracted correctly from the NetBox response. +- Review operator logs for parsing errors: + ```bash + kubectl logs -l app=gnmic-operator -n gnmic-operator-system + ``` + +### Rate Limiting or Timeouts + +Increase the sync interval in your `TargetSource` or adjust timeouts: + +```yaml +spec: + provider: + http: + interval: 1h + timeout: 1m +``` diff --git a/docs/content/docs/examples/NetBox/_index.md b/docs/content/docs/examples/NetBox/_index.md new file mode 100644 index 00000000..4885d60e --- /dev/null +++ b/docs/content/docs/examples/NetBox/_index.md @@ -0,0 +1,8 @@ +--- +title: "NetBox" +linkTitle: "NetBox" +weight: 6 +# draft: true +description: > + Discover targets from NetBox using the HTTP provider +--- diff --git a/docs/content/docs/reference/api.md b/docs/content/docs/reference/api.md index 1cfe4fb0..d20cb3e5 100644 --- a/docs/content/docs/reference/api.md +++ b/docs/content/docs/reference/api.md @@ -153,30 +153,111 @@ description: > | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| -| `http` | HTTPConfig | No | - | HTTP endpoint for target discovery | -| `consul` | ConsulConfig | No | - | Consul service discovery config | -| `configMap` | string | No | - | ConfigMap name containing targets | -| `podSelector` | LabelSelector | No | - | Select Kubernetes Pods as targets | -| `serviceSelector` | LabelSelector | No | - | Select Kubernetes Services as targets | -| `labels` | map[string]string | No | - | Labels to apply to discovered targets | +| `provider` | ProviderSpec | Yes | - | Provider-specific discovery configuration | +| `targetPort` | int32 | No | - | Default port used when the discovered target does not provide a port | +| `targetProfile` | string | Yes | - | Reference to `TargetProfile` applied to discovered targets | +| `targetLabels` | map[string]string | No | - | Labels added to all discovered targets | + +### ProviderSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `http` | HTTPConfig | No | HTTP provider configuration | ### HTTPConfig +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `url` | string | No | - | HTTP endpoint used to pull targets. Required unless push is enabled | +| `method` | string | No | GET | HTTP request method | +| `headers` | map[string]string | No | - | HTTP headers to include in requests | +| `body` | string | No | - | Request body for POST requests | +| `authentication` | AuthenticationSpec | No | - | Authentication configuration for the HTTP endpoint | +| `interval` | duration | No | 6h | Polling interval used to refresh targets | +| `timeout` | duration | No | 10s | Timeout for HTTP requests | +| `tls` | ClientTLSConfig | No | - | Client TLS configuration for HTTPS endpoints | +| `pagination` | PaginationSpec | No | - | Pagination settings for parsing responses | +| `mapping` | ResponseMappingSpec | No | - | Response mapping configuration for JSON responses | +| `push` | PushSpec | No | - | Push-based update configuration | + +### ClientTLSConfig + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `insecureSkipVerify` | bool | No | false | Skip verification of the server certificate | +| `caBundleRef` | ConfigMapKeySelector | No | - | Reference to a ConfigMap containing a PEM CA bundle | + +### AuthenticationSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `basic` | BasicAuthSpec | No | Basic authentication configuration | +| `token` | TokenAuthSpec | No | Token authentication configuration | + +### BasicAuthSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `credentialsSecretRef` | SecretKeySelector | Yes | Reference to a Secret containing username/password keys | + +### TokenAuthSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `scheme` | string | Yes | Token scheme, e.g. Bearer | +| `tokenSecretRef` | SecretKeySelector | Yes | Reference to a Secret containing the token | + +### PaginationSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `nextField` | string | No | JSON field containing the next page reference or pagination token | + +### ResponseMappingSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `targetsField` | string | No | CEL expression selecting the list of targets from the response | +| `name` | string | No | CEL expression for the target name | +| `address` | string | No | CEL expression for the target address | +| `port` | string | No | CEL expression for the target port | +| `labels` | string | No | CEL expression returning a map of labels | +| `targetProfile` | string | No | CEL expression for the target profile | + +### PushSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `enabled` | bool | No | Enable push updates | +| `auth` | PushAuthSpec | No | Push authentication configuration | + +### PushAuthSpec + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `bearer` | PushBearerAuthSpec | No | Bearer token authentication configuration | +| `signature` | PushSignatureAuthSpec | No | Signature authentication configuration | + +### PushBearerAuthSpec + | Field | Type | Required | Description | |-------|------|----------|-------------| -| `url` | string | Yes | URL of the HTTP endpoint | +| `tokenSecretRef` | SecretKeySelector | Yes | Reference to a Secret containing the bearer token | -### ConsulConfig +### PushSignatureAuthSpec | Field | Type | Required | Description | |-------|------|----------|-------------| -| `url` | string | Yes | Consul server URL | +| `secretRef` | SecretKeySelector | Yes | Reference to a Secret used to verify request signatures | +| `header` | string | Yes | Header containing the signature | +| `algorithm` | string | No | Signature algorithm | ### TargetSourceStatus | Field | Type | Description | |-------|------|-------------| | `status` | string | Sync status (Synced, Error, Pending) | +| `observedGeneration` | int64 | Observed generation of the spec | | `targetsCount` | int32 | Number of discovered targets | | `lastSync` | Time | Last successful sync timestamp | diff --git a/docs/content/docs/user-guide/targetsource.md b/docs/content/docs/user-guide/targetsource.md deleted file mode 100644 index d1fee0c4..00000000 --- a/docs/content/docs/user-guide/targetsource.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -title: "TargetSource" -linkTitle: "TargetSource" -weight: 4 -description: > - Dynamic target discovery from external sources ---- - -The `TargetSource` resource enables dynamic discovery of network devices from external sources. The operator automatically creates, updates, and deletes `Target` resources based on discovered devices. - -## Discovery Sources - -TargetSource supports multiple discovery backends: - -| Source | Description | -|--------|-------------| -| `http` | Fetch targets from an HTTP endpoint | -| `consul` | Discover targets from Consul service registry | -| `configMap` | Read targets from a Kubernetes ConfigMap | -| `podSelector` | Create targets from Kubernetes Pods | -| `serviceSelector` | Create targets from Kubernetes Services | - -## HTTP Discovery - -Discover targets from an HTTP endpoint that returns a JSON list of targets: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: http-discovery -spec: - http: - url: http://inventory-service:8080/targets - labels: - source: inventory -``` - -The HTTP endpoint should return a JSON array of target objects. - -## Consul Discovery - -Discover targets from Consul service registry: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: consul-discovery -spec: - consul: - url: http://consul:8500 - labels: - source: consul - datacenter: dc1 -``` - -## ConfigMap Discovery - -Read targets from a Kubernetes ConfigMap: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: configmap-targets -spec: - configMap: network-devices - labels: - source: configmap -``` - -The ConfigMap should contain target definitions in a structured format. - -## Kubernetes Pod Discovery - -Create targets from Kubernetes Pods matching a label selector: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: pod-discovery -spec: - podSelector: - matchLabels: - app: network-simulator - gnmi: enabled - labels: - source: kubernetes - type: simulator -``` - -This is useful for: -- Containerized network simulators -- Virtual network functions (VNFs) -- Development/testing environments - -## Kubernetes Service Discovery - -Create targets from Kubernetes Services matching a label selector: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: service-discovery -spec: - serviceSelector: - matchLabels: - protocol: gnmi - labels: - source: kubernetes -``` - -## Label Inheritance - -Labels defined in the `TargetSource.spec.labels` field are applied to all discovered targets: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: datacenter-a -spec: - consul: - url: http://consul-dc-a:8500 - labels: - datacenter: dc-a - environment: production - source: consul -``` - -All targets discovered from this source will have: -- `datacenter: dc-a` -- `environment: production` -- `source: consul` - -This enables using label selectors in Pipelines to select targets by their discovery source. - -## Status - -The TargetSource status shows discovery state: - -```yaml -status: - status: Synced - targetsCount: 42 - lastSync: "2024-01-15T10:30:00Z" -``` - -| Field | Description | -|-------|-------------| -| `status` | Current sync status (Synced, Error, Pending) | -| `targetsCount` | Number of targets discovered | -| `lastSync` | Timestamp of last successful sync | - -## Example: Multi-Source Discovery - -Combine multiple TargetSources for different environments: - -```yaml -# Production devices from Consul -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: prod-consul -spec: - consul: - url: http://consul-prod:8500 - labels: - environment: production - source: consul ---- -# Lab devices from ConfigMap -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: lab-devices -spec: - configMap: lab-network-devices - labels: - environment: lab - source: configmap ---- -# Simulator pods -apiVersion: operator.gnmic.dev/v1alpha1 -kind: TargetSource -metadata: - name: simulators -spec: - podSelector: - matchLabels: - app: srlinux - labels: - environment: dev - source: kubernetes -``` - -Then use label selectors in your Pipeline: - -```yaml -apiVersion: operator.gnmic.dev/v1alpha1 -kind: Pipeline -metadata: - name: production-telemetry -spec: - clusterRef: prod-cluster - enabled: true - targetSelectors: - - matchLabels: - environment: production - # ... subscriptions, outputs -``` - -## Lifecycle - -### Target Creation - -When a TargetSource discovers a new device: -1. A new `Target` resource is created -2. Labels from `spec.labels` are applied -3. Owner reference is set to the TargetSource - -### Target Updates - -When a discovered device's properties change: -1. The corresponding `Target` is updated -2. Clusters using that target are reconciled - -### Target Deletion - -When a device is no longer discovered: -1. The `Target` resource is deleted -2. Clusters stop collecting from that target - -### TargetSource Deletion - -When a TargetSource is deleted: -1. All Targets owned by it are deleted (via owner references) -2. Clusters are reconciled to remove those targets - diff --git a/docs/content/docs/user-guide/targetsource/_index.md b/docs/content/docs/user-guide/targetsource/_index.md new file mode 100644 index 00000000..f9897a71 --- /dev/null +++ b/docs/content/docs/user-guide/targetsource/_index.md @@ -0,0 +1,214 @@ +--- +title: "TargetSource" +linkTitle: "TargetSource" +weight: 4 +description: > + Dynamic target discovery from external sources +--- + +The `TargetSource` resource enables dynamic discovery of network devices from external sources. The operator automatically creates, updates, and deletes `Target` resources based on discovered devices. + +## Basic Configuration + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: targetsource-1 +spec: + provider: + # Configure one of the supported providers + targetPort: 57400 + targetProfile: default + targetLabels: + source: inventory +``` + +The supported TargetSource providers are documented on the [TargetSource Provider](./providers/) page. + +## Spec Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `provider` | object | Yes | Provider-specific discovery configuration. Exactly one provider must be configured | +| `targetPort` | int32 | No | Default port used when the discovered target does not provide a port | +| `targetProfile` | string | No | Reference to default `TargetProfile` applied to all discovered targets if no profile was discovered | +| `targetLabels` | map[string]string | No | Labels added to all discovered targets | + + + + + + + +## Label Inheritance + +Each generated `Target` receives an ownership label identifying the originating `TargetSource`: +```yaml +operator.gnmic.dev/targetsource: targetsource-1 +``` + +This label is automatically managed by the operator and is used to: +- Identify targets owned by a specific `TargetSource` +- Determine which targets should be updated or deleted during reconciliation + +The `operator.gnmic.dev/targetsource` label is reserved and always takes precedence over any provider-supplied labels. + +### TargetSource Labels + +Additional labels can be applied to all generated targets using `spec.targetLabels`: + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: targetsource-1 +spec: + provider: + http: + url: http://targetsource-1:8080/targets + targetLabels: + datacenter: dc-a + environment: production +``` + +All targets discovered from this source will have: +- `datacenter: dc-a` +- `environment: production` + +This enables Pipelines to select targets using label selectors. + +### Labels from Discovery Providers + +Discovery providers may return additional labels for each target. These labels are applied directly to the generated `Target` resource. + +The `gnmic_operator_` label prefix is reserved for operator-specific behavior. Labels using this prefix are interpreted by the operator and are not applied directly to the generated `Target` resource. + +Supported operator labels: + +| Label | Description | +|--------|-------------| +| `gnmic_operator_target_profile` | Overrides the `TargetProfile` configured in the `TargetSource` | + +### Label Precedence + +If the same label key is defined in multiple places, labels are applied in the following order (highest precedence first): + +1. `TargetSource` ownership label (`operator.gnmic.dev/targetsource`) +2. Labels from `TargetSource.spec.targetLabels` +3. Labels returned by the discovery provider + +## Status + +The `TargetSource` status shows discovery state: + +```yaml +status: + status: Synced + observedGeneration: 1 + targetsCount: 42 + lastSync: "2024-01-15T10:30:00Z" +``` + +| Field | Description | +|-------|-------------| +| `status` | Current sync status (Synced, Error, Pending) | +| `observedGeneration` | Generation of the spec last processed by the controller | +| `targetsCount` | Number of targets discovered | +| `lastSync` | Timestamp of last successful sync | + +## Lifecycle + +### Target Creation + +When a `TargetSource` discovers a new device: + +1. A new `Target` resource is created +2. The `TargetProfile` referenced in `spec.targetProfile` is assigned +3. Labels from `spec.targetLabels` are applied +4. The `TargetSource` is set as the owner reference + +### Target Updates + +On each discovery cycle, existing `Target` resources are reconciled with the latest discovered state: + +1. The corresponding `Target` resource is updated and overwritten +2. Clusters consuming the target are reconciled automatically + +> Manual changes to `Target` resources managed by a `TargetSource` are overwritten on every reconciliation cycle. + +### Target Deletion + +When a device is no longer returned by the discovery provider: + +1. The corresponding `Target` resource is deleted +2. Clusters automatically stop using the target + +### TargetSource Deletion + +When a `TargetSource` is deleted: + +1. All `Target` resources owned by it are deleted via owner references +2. Clusters are reconciled and remove the deleted targets + diff --git a/docs/content/docs/user-guide/targetsource/providers/_index.md b/docs/content/docs/user-guide/targetsource/providers/_index.md new file mode 100644 index 00000000..29284d7f --- /dev/null +++ b/docs/content/docs/user-guide/targetsource/providers/_index.md @@ -0,0 +1,7 @@ +--- +title: "TargetSource Provider" +linkTitle: "TargetSourceProvider" +weight: 1 +description: > + Configuring TargetSource discovery providers +--- diff --git a/docs/content/docs/user-guide/targetsource/providers/http.md b/docs/content/docs/user-guide/targetsource/providers/http.md new file mode 100644 index 00000000..de0f3b6b --- /dev/null +++ b/docs/content/docs/user-guide/targetsource/providers/http.md @@ -0,0 +1,630 @@ +--- +title: "HTTP Provider" +linkTitle: "HTTP" +weight: 2 +description: > + The HTTP provider discovers targets from an HTTP endpoint returning JSON, or receives webhook-based updates when push mode is enabled. +--- + +The HTTP provider discovers targets from an HTTP endpoint returning JSON, or receives webhook-based updates when push mode is enabled. + +## Basic Configuration + +```yaml +apiVersion: operator.gnmic.dev/v1alpha1 +kind: TargetSource +metadata: + name: targetsource-1 +spec: + provider: + provider: + http: + url: http://inventory-service:8080/targets + authentication: + token: + scheme: Bearer + tokenSecretRef: + name: inventory-token + key: token + # Enable push mode + push: + enabled: true + targetPort: 57400 + targetProfile: default + targetLabels: + source: inventory +``` + +## HTTP Spec Fields + +| Field | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `url` | string | No | - | HTTP endpoint used to pull targets. Required unless `push.enabled` is enabled | +| `method` | string | No | GET | HTTP method used for requests | +| `headers` | map[string]string | No | - | HTTP headers to include in requests | +| `body` | string | No | - | Request body for POST requests | +| `authentication` | object | No | - | Authentication configuration for the HTTP endpoint | +| `interval` | duration | No | 30m | Polling interval used to refresh targets | +| `timeout` | duration | No | 30s | Timeout for HTTP requests | +| `tls` | object | No | - | Client TLS configuration for HTTPS endpoints | +| `pagination` | object | No | - | Pagination configuration for parsing HTTP responses | +| `mapping` | object | No | - | Response mapping configuration for JSON responses | +| `push` | object | No | - | Push-based update configuration | + +## Pull Mode + +The HTTP provider supports pull-based target discovery by periodically querying a remote HTTP endpoint that returns target data in JSON format. + +```yaml +spec: + provider: + http: + url: http://inventory-service:8080/targets +``` + +In pull mode, the operator sends HTTP requests to the configured url at a fixed interval and updates targets based on the response. The `push.enabled` field is optional when pull mode is enabled, but can still be used for accepting incoming webhook notifications. + +*How Pull Mode Works* +1. The operator sends an HTTP request to the configured url +2. The response is parsed (either directly or via mapping) +3. Targets are created, updated, or removed based on the returned data +4. This process repeats according to the configured interval + + +### Authentication + +The HTTP provider supports authenticated requests to the inventory endpoint. + +Exactly one authentication method can be configured. + +#### Basic Authentication + +Credentials are referenced from a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/). + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/targets + authentication: + basic: + credentialSecretRef: + name: inventory-credentials + key: username +``` + +#### Token Authentication + +Token authentication is configured using a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) reference. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/targets + authentication: + token: + scheme: Bearer + tokenSecretRef: + name: inventory-token + key: token +``` + +### TLS + +TLS settings can be configured for HTTPS endpoints. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/targets + tls: + insecureSkipVerify: false + caBundleRef: + name: inventory-ca + key: ca.crt +``` + +#### TLS Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `insecureSkipVerify` | bool | No | Skip verification of the server certificate. Defaults to `false` | +| `caBundleRef` | object | No | Reference to a [Kubernetes ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) containing a PEM-encoded CA bundle | + +### Pagination + +Pagination enables the operator to retrieve complete result sets from APIs that return data in multiple pages. The operator automatically follows pagination until no further pages are available. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/devices + pagination: + nextField: "self.next" +``` + +#### Pagination Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `nextField` | string | No | CEL expression used to extract the next page reference from the response | +| `requestParam` | string | No | Query parameter used when the extracted value is a token | + +The `nextField` value may either contain: +- A full URL for the next request +- A pagination token appended as a query parameter to the original URL + +#### How Pagination Works + +The operator handles the following pagination patterns: + +##### 1. Link Header Pagination +If the API provides a Link response header with `rel="next"`, the operator will automatically follow it. + +Example response header: +``` +Link: ; rel="next" +``` + +Behavior: +``` +Request 1: GET /devices?page=1 +Request 2: GET /devices?page=2 +Request 3: GET /devices?page=3 +... +``` + +##### 2. URL-Based Pagination +If the response contains a full URL in the body (e.g. `"next": "https://..."`), it will be used directly. + +Example response: +```json +{ + "devices": [...], + "next": "https://inventory.example.com/devices?offset=50" +} +``` + +##### 3. Token-Based Pagination +If the response contains a pagination token, the operator appends it as a query parameter. + +Example: +```yaml +pagination: + nextField: "self.next_token" + requestParam: "page_token" +``` + +Example: +``` +GET /devices +-> "next_token": "abc123" +GET /devices?page_token=abc123 +``` + +##### CEL-Based Extraction +The nextField is evaluated as a CEL expression using: +- `self` -> entire JSON response + +Example: +```yaml +pagination: + nextField: "self['@odata.nextLink']" +``` + +This allows extracting values from nested or special keys. + +### Response Processing + +The HTTP provider supports two response processing modes: + +- **Default response format**: The endpoint returns a JSON array of target objects. +- **Response mapping**: Custom JSON structures are mapped to target fields using CEL expressions. + +If `mapping` is configured, the custom mapping rules are used. Otherwise, the response itself must be a JSON array. + +#### Default Response Format + +If `mapping` is not configured, the endpoint must return a JSON array of objects with the following structure: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `name` | string | Yes | Name of the generated `Target` resource | +| `address` | string | Yes | Device address (FQDN or IP address) | +| `port` | int32 | No | Port used for gNMI connections. If omitted, `spec.targetPort` is used | +| `labels` | map[string]string | No | Labels added to the generated `Target` resource | +| `targetProfile` | string | No | Reference to a `TargetProfile`. If omitted, `spec.targetProfile` is used | + +Example response: + +```json +[ + { + "name": "spine1", + "address": "spine1.local", + "port": 57400, + "labels": { + "role": "spine" + }, + "targetProfile": "spine-profile" + }, + { + "name": "leaf1", + "address": "leaf1.local", + "port": 57400, + "labels": { + "role": "leaf" + } + }, + { + "name": "leaf2", + "address": "leaf2.local", + "port": 57400, + "labels": { + "role": "leaf" + } + } +] +``` + +#### Response Mapping via CEL + +When your inventory API's JSON structure differs from the default format, use CEL (Common Expression Language) mapping to extract target fields. + +```yaml +spec: + provider: + http: + url: https://inventory.example.com/devices + mapping: + targetsField: "self.results" + name: "item.hostname" + address: "item.management.ip" + port: "item.gnmi.port" + targetProfile: "item.profile" + labels: "{'role': item.metadata.role, 'site': item.metadata.site}" +``` + +##### Understanding `targetsField` + +The `targetsField` expression tells the operator where to find the list of target objects in your API response. It's particularly important when your API wraps the target list in a data structure. + +**When to use `targetsField`:** +- Your API returns `{"results": [...]}` -> use `"self.results"` +- Your API returns `{"data": {"devices": [...]}}` -> use `"self.data.devices"` +- Your API returns a plain array `[...]` -> omit `targetsField` (default behavior) + +**Example scenarios:** + +*Custom API response example 1:* +```json +{ + "count": 42, + "next": "https://...", + "results": [ + {"id": 1, "name": "device1", "primary_ip": "10.0.0.1"}, + {"id": 2, "name": "device2", "primary_ip": "10.0.0.2"} + ] +} +``` +Usage: `targetsField: "self.results"` + +*Custom API response example 2:* +```json +{ + "status": "success", + "data": { + "timestamp": "2024-01-01T00:00:00Z", + "devices": [ + {"name": "router1", "mgmt_ip": "192.168.1.1"}, + {"name": "router2", "mgmt_ip": "192.168.1.2"} + ] + } +} +``` +Usage: `targetsField: "self.data.devices"` + +##### Mapping Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `targetsField` | string | No | CEL expression selecting the target list from the response. If omitted, assumes response is a direct JSON array | +| `name` | string | No | CEL expression for the target name | +| `address` | string | No | CEL expression for the target address | +| `port` | string | No | CEL expression for the target port | +| `labels` | string | No | CEL expression returning a map of labels | +| `targetProfile` | string | No | CEL expression for the target profile | + +##### CEL Variables + +The mapping expressions support the following variables: +- `item`: the current target object being processed +- `self`: the complete unprocessed response from the HTTP endpoint + +#### Performance: CEL vs Direct Mapping + +Understanding the performance implications helps optimize your configurations: + +**Direct Mapping (No CEL)** - *Fastest* +- Used when your API response matches the default structure exactly +- No expression compilation or evaluation overhead +- Suitable for high-frequency polling (e.g., every minute) +- Example: API returns `[{"name": "...", "address": "..."}]` + +**CEL Mapping** - *Slight overhead* +- CEL expressions are compiled once at startup (not per request) +- Evaluation is performed per target object during each poll cycle +- At high scale (10,000+ targets), consider the `interval` between polls + +**Best practices:** +- Use direct mapping if your API already returns the correct structure +- For large result sets, increase the interval +- Combine CEL and direct mapping for efficiency (see hybrid mapping below) +- Use CEL extensions (see reference table below) to reduce complexity and improve readability + +#### CEL Extensions + +The operator includes a set of standard CEL extensions from the official [CEL Go library](https://github.com/google/cel-go) to enable more advanced expressions. + +These [extensions](https://pkg.go.dev/github.com/google/cel-go/ext) expand CEL with additional capabilities commonly needed when transforming API responses: + +| Extension | Purpose | +|----------|----------| +| **Strings** | String manipulation such as splitting values, case conversion, and extracting parts of text (e.g. parsing hostnames or IPs) | +| **Math** | Numeric operations and comparisons (e.g. calculations, min/max, type conversions) | +| **Lists** | Working with arrays (e.g. indexing, filtering, joining values) | +| **Sets** | Set-style operations such as membership checks and comparisons | +| **Regex** | Pattern matching and validation using regular expressions | +| **Bindings** | Defining intermediate variables to simplify complex expressions | + +**Examples:** + +```yaml +mapping: + # Extract site from hostname + labels: | + { + 'site': item.name.split('-')[0] + } + + # Conditional profile + targetProfile: "item.type == 'edge' ? 'edge' : 'core'" + + # Pattern-based classification + labels: | + { + 'role': item.name.matches('^spine') ? 'spine' : 'leaf' + } +``` + +#### Combining CEL and Direct Mapping (Hybrid Approach) + +You don't need to map all fields with CEL. The operator supports mixing CEL expressions and direct field lookups for maximum efficiency: + +| Scenario | Behavior | Use Case | +|----------|----------|----------| +| `name`, `address` use CEL; others omitted | Extracts mapped fields via CEL; looks for `port`, `labels`, `targetProfile` directly in item JSON | Simple API where only some fields need transformation | +| Only `labels` uses CEL | Other fields use direct mapping; labels constructed from CEL expression | API returns correct `name`, `address`, `port` but custom labels need extraction | +| Only `address` uses CEL | Direct mapping for other fields; only address requires transformation | Most fields match API exactly except address requires CIDR parsing or format conversion | +| All fields use CEL | Complete transformation via expressions | API structure completely different from expected format | + +This hybrid approach optimizes performance by only compiling and evaluating CEL where needed. + +**Example - Partial CEL mapping (only transform what needs transforming):** +```yaml +mapping: + # Use CEL only when you need to transform a field + name: "item.hostname" + address: "item.primary_ip4 != null ? item.primary_ip4.split('/')[0] : item.primary_ip6.split('/')[0]" # CEL: parse CIDR + + # Fields that already exist should be omitted + # Port already exists as "port" field in item + # port: item.port <- omit this + + # Use CEL for structured or derived values + labels: | + { + "site": item.site.name, + "role": item.device_role.name + } + + # targetProfile can also be omitted if already present or not needed +``` + +In this example, only `address` and `labels` use CEL expressions; `name`, `port`, and `targetProfile` use direct field lookups for efficiency. + +#### Using YAML `|` for Complex CEL Expressions + +When writing more complex CEL expressions, it is recommended to use YAML’s pipe (`|`) literal block instead of inline strings. + +This is especially useful for expressions that span multiple lines or contain nested logic. + +## Push Mode + +The HTTP provider supports webhook-based target updates via `spec.provider.http.push`. + +```yaml +spec: + provider: + http: + push: + enabled: true +``` + +When `push.enabled` is true, the operator accepts incoming webhook notifications and can update targets without polling a remote endpoint. The `url` field is optional when push mode is enabled, but can still be used for polling and fallback behavior. + +#### Recommended pattern (labels example) + +```yaml +mapping: + labels: | + { + "site": item.site.name, + "rack": item.rack != null ? item.rack.name : "", + "role": item.role != null ? item.role : "unknown", + "tags": item.tags.join(',') + } +``` + +**Why use `|` instead of quoted strings:** +- **Readability**: Multi-line expressions are easier to understand +- **Maintainability**: Complex CEL expressions don't require escaping +- **YAML best practice**: Literal blocks handle special characters naturally + +## Recommended Production Settings + +When deploying HTTP TargetSource providers in production networks, follow these guidelines to ensure reliable and efficient target discovery: + +### Polling Configuration +| Scenario | Setting | Rationale | +|----------|---------|-----------| +| **Small environment** (< 100 targets) | `interval: 5m` | Frequent updates without excessive load | +| **Medium environment** (100-500 targets) | `interval: 10m` | Balance between freshness and API load | +| **Large environment** (500-2000 targets) | `interval: 15m` | Reduce API polling overhead | +| **Very large environment** (2000+ targets) | `interval: 30m` | Minimize impact on inventory system | +| **High-frequency changes** | Use `push` mode with `interval` | Enables updates via push while periodic polling ensures completeness and consistency | + +**Timeout Configuration:** +```yaml +timeout: 30s # Allows for network latency +``` + +If timeouts consistently occur, increase `interval` instead of timeout (don't poll faster) + +### Authentication & Security + +**Always use TLS in production:** +```yaml +tls: + insecureSkipVerify: false # Never skip verification in production + caBundleRef: + name: inventory-ca-bundle + key: ca.crt +``` + +**For authenticated APIs:** +- Store credentials in [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) +- Rotate credentials periodically +- Use token-based auth when possible (simpler secret rotation) + +Example: +```yaml +authentication: + token: + scheme: Bearer + tokenSecretRef: + name: inventory-api-token + key: token +``` + +### Pagination & Large Result Sets + +**Configuration for APIs returning large result sets:** +```yaml +pagination: + nextField: next # Always configure pagination if your API supports it + +interval: 30m # Increase interval for large datasets (reduces cumulative API load) +timeout: 60s # Increase only if individual requests are slow or responses are large +``` + +Pagination splits large datasets into multiple smaller HTTP requests. This improves reliability and reduces the likelihood of timeouts compared to fetching a single large response. + +**Optimization strategies:** +- Request API filtering (if supported) to reduce result set size (e.g. ?limit=1000 or ?status=active) +- If the API does not support pagination or filtering increase the timeout +- Consider webhook push mode for frequently-changing inventories (if API supports it) + +### Mapping Optimization + +**Use hybrid CEL and direct mapping for performance:** +```yaml +# EFFICIENT - Only CEL-transform what needs it +mapping: + # + name: "item.hostname" # CEL expression + # port: (OMITTED) # Direct: exists as "port" in item + + # Only these need transformation -> use CEL + address: "item.primary_ip.split('/')[0]" # CEL: parse CIDR + labels: | # CEL: construct from nested fields + {'site': item.site.name} +``` + +**Avoid unnecessary CEL complexity:** +```yaml +# GOOD - Simple expressions +mapping: + address: "item.management_ip" + port: "int(item.gnmi_port)" + +# AVOID - Nested ternary logic (hard to debug) +mapping: + name: "item.has_override ? item.override_name : (item.hostname != '' ? item.hostname : 'default-' + string(item.id))" +``` + +**CEL expression best practices:** +- Compile expressions once at startup (not per request), so complexity is paid only once +- Use `ext.Bindings` for repeated expressions to avoid redundant evaluation +- Test CEL expressions thoroughly; they're compiled but errors only appear during evaluation +- Keep expressions under 200 characters for maintainability + +### Example Production Configuration + +```yaml +apiVersion: gnmic.openconfig.net/v1alpha1 +kind: TargetSource +metadata: + name: production-inventory +spec: + provider: + http: + # Security + url: https://inventory.prod.example.com/api/dcim/devices/?limit=100 + tls: + insecureSkipVerify: false + caBundleRef: + name: netbox-ca + key: ca.crt + + # Authentication + authentication: + token: + scheme: Bearer + tokenSecretRef: + name: api-token + key: token + + # Timing + interval: 15m # Balanced update frequency + timeout: 30s # Allow for network latency + + # Pagination + pagination: + nextField: next + + # Mapping for fields + mapping: + targetsField: "self.results" + #name: "item.name" -> already handled with fallback direct mapping + address: "item.primary_ip4 != null ? item.primary_ip4.split('/')[0] : item.primary_ip6.split('/')[0]" + port: "item.custom_fields.gnmi_port" + labels: "{\n 'site': item.site.name,\n 'role': item.device_role.name,\n 'status': item.status.value\n }" + targetProfile: "item.custom_fields.gnmi_profile" + + # Global settings + targetPort: 9339 + targetProfile: default-profile +``` + +This configuration ensures: + +- Secure HTTPS communication with certificate validation +- API authentication with token-based credentials +- Balanced polling interval for stable environments +- Proper pagination handling for large device inventories +- Rich label extraction from custom fields +- Fallback to defaults when fields are missing