From f923e00d5114aca4f2d6384f812d7805dcdbf71d Mon Sep 17 00:00:00 2001 From: Jiawei Huang Date: Wed, 1 Jul 2026 22:56:35 -0700 Subject: [PATCH] Update log collector docs for the fluentd to fluent-bit migration (EV-6164) Calico Enterprise removes the fluentd log collector in favour of Fluent Bit (calico-fluent-bit in calico-system; tigera/operator#4910). Update the next-version Calico Enterprise and Calico Cloud docs to match: - Rewrite "Filter flow logs" / "Filter DNS logs": examples move from fluentd syntax to Fluent Bit YAML filter lists, the ConfigMap is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note explains that old fluentd-syntax filters are not translated (the operator raises a tigera status warning naming the offending key). - Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020, /api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent plus a dropped-chunks alert on fluentbit_output_retries_failed_total. - BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP behind the allow-calico-fluent-bit policy); ServiceMonitor sample is fluent-bit-metrics-service-monitor.yaml. - Sweep of remaining pages: tigera-fluentd namespace -> calico-system, fluentd-node -> calico-fluent-bit (labels, secrets, packet capture retrieval via calico-node, diags output, resource override examples now use calicoFluentBitDaemonSet, architecture/overview descriptions, cc-arch-diagram ports, operator checklist). Not touched: versioned docs, releases.json (historical), and the generated installation API reference (regenerates from the operator release). Co-Authored-By: Claude Fable 5 --- calico-cloud/_includes/components/ReqsSys.js | 2 +- calico-cloud/get-started/cc-arch-diagram.mdx | 6 +- .../get-started/operator-checklist.mdx | 30 +++++----- .../elastic/dns/filtering-dns.mdx | 59 +++++++++++-------- .../observability/elastic/flow/filtering.mdx | 59 ++++++++----------- .../observability/elastic/overview.mdx | 16 ++--- calico-cloud/observability/kube-audit.mdx | 2 +- calico-cloud/observability/packetcapture.mdx | 10 ++-- calico-cloud/operations/comms/index.mdx | 24 ++++---- .../elasticsearch-and-fluentd-metrics.mdx | 36 +++++++---- .../monitor/prometheus/byo-prometheus.mdx | 56 +++++++----------- .../configure-resources.mdx | 16 ++--- .../_includes/components/ReqsSys.js | 2 +- .../_includes/content/_default-install.mdx | 2 +- .../elastic/dns/filtering-dns.mdx | 59 +++++++++++-------- .../observability/elastic/flow/filtering.mdx | 59 ++++++++----------- .../observability/elastic/overview.mdx | 16 ++--- .../observability/kube-audit.mdx | 2 +- .../observability/packetcapture.mdx | 10 ++-- .../review-unused-network-policies.mdx | 6 +- calico-enterprise/operations/comms/index.mdx | 42 ++++++------- .../operations/license-options.mdx | 2 +- .../elasticsearch-and-fluentd-metrics.mdx | 44 ++++++++------ .../monitor/prometheus/byo-prometheus.mdx | 58 ++++++++---------- .../reference/architecture/overview.mdx | 12 ++-- .../clis/calicoctl/cluster/diags.mdx | 25 ++++---- .../configure-resources.mdx | 28 ++++----- 27 files changed, 339 insertions(+), 344 deletions(-) diff --git a/calico-cloud/_includes/components/ReqsSys.js b/calico-cloud/_includes/components/ReqsSys.js index be8e5b7f0b..de88780dab 100644 --- a/calico-cloud/_includes/components/ReqsSys.js +++ b/calico-cloud/_includes/components/ReqsSys.js @@ -308,7 +308,7 @@ function NetworkRequirementsEnt(props) { Logs and storage - Elasticsearch with fluentd datastore + Elasticsearch log storage TCP 9200 (default) diff --git a/calico-cloud/get-started/cc-arch-diagram.mdx b/calico-cloud/get-started/cc-arch-diagram.mdx index b8986c4bba..5452ad2704 100644 --- a/calico-cloud/get-started/cc-arch-diagram.mdx +++ b/calico-cloud/get-started/cc-arch-diagram.mdx @@ -32,8 +32,8 @@ The following diagram shows the major components in a managed cluster, followed | $[prodname] tunnel server | Communicates with managed clusters by creating secure TLS tunnels. | Port 9000 from managed clusters | | calico-node | Bundles key components that are required for networking containers with $[prodname]:

• Felix
• BIRD
• confd | • TCP 5473 to Typha
• TCP 9900 and 9081 from Prometheus API service | | Container threat detection | A threat detection engine that analyzes observed file and process activity to detect known malicious and suspicious activity. Monitors the following types of suspicious activity within containers:

• Access to sensitive system files and directories
• Defense evasion
• Discovery
• Execution
• Persistence
• Privilege escalation

Includes these components:

**Runtime Security Operator**
An operator to manage and reconcile container threat defense components.

**Runtime Reporter Pods**
Pods running on each node in the cluster to perform the detection activity outlined above.They send activity reports to Elasticsearch for analysis by $[prodname]. | TCP to Kubernetes API server | -| Compliance | Generates compliance reports for the Kubernetes cluster. Reports are based on archived flow and audit logs for Calico Cloud resources, plus any audit logs you’ve configured for Kubernetes resources in the Kubernetes API server. Compliance reports provide the following high-level information:

• Endpoints explicitly protected using ingress or egress policy
• Policies and services
- Policies and services associated with endpoints
- Policy audit logs
• Traffic
- Allowed ingress/egress traffic to/from namespaces, and to/from the internet Compliance includes these components:

**compliance-snapshotter**
Handles listing of required Kubernetes and $[prodname]configuration and pushes snapshots to Elasticsearch. Snapshots give you visibility into configuration changes, and how the cluster-wide configuration has evolved within a reporting interval.

**compliance-reporter**
Handles report generation. Reads configuration history from Elasticsearch and determines time evolution of cluster-wide configuration, including relationships between policies, endpoints, services, and network sets. Data is then passed through a zero-trust aggregator to determine the “worst-case outliers” in the reporting interval.

**compliance-controller**
Reads report configuration and manages creation, deletion, and monitoring of report generation jobs.

**compliance-benchmarker**
A daemonset that runs checks in the CIS Kubernetes Benchmark on each node so you can see if Kubernetes is securely deployed.
| • TCP 8080 to Guardian
• TCP 6443 to Kubernetes API server | -| Fluentd | Open-source data collector for unified logging. Collects and forwards $[prodname] logs (flows, DNS, L7) to log storage. | • TCP 8080 to Guardian
• TCP 9080 from Prometheus API service | +| Compliance | Generates compliance reports for the Kubernetes cluster. Reports are based on archived flow and audit logs for Calico Cloud resources, plus any audit logs you’ve configured for Kubernetes resources in the Kubernetes API server. Compliance reports provide the following high-level information:

• Endpoints explicitly protected using ingress or egress policy
• Policies and services
- Policies and services associated with endpoints
- Policy audit logs
• Traffic
- Allowed ingress/egress traffic to/from namespaces, and to/from the internet Compliance includes these components:

**compliance-snapshotter**
Handles listing of required Kubernetes and $[prodname] configuration and pushes snapshots to Elasticsearch. Snapshots give you visibility into configuration changes, and how the cluster-wide configuration has evolved within a reporting interval.

**compliance-reporter**
Handles report generation. Reads configuration history from Elasticsearch and determines time evolution of cluster-wide configuration, including relationships between policies, endpoints, services, and network sets. Data is then passed through a zero-trust aggregator to determine the “worst-case outliers” in the reporting interval.

**compliance-controller**
Reads report configuration and manages creation, deletion, and monitoring of report generation jobs.

**compliance-benchmarker**
A DaemonSet that runs checks in the CIS Kubernetes Benchmark on each node so you can see if Kubernetes is securely deployed.
| • TCP 8080 to Guardian
• TCP 6443 to Kubernetes API server | +| Fluent Bit | Open-source, lightweight log processor and forwarder. Collects and forwards $[prodname] logs (flows, DNS, L7) to log storage. | • TCP 8080 to Guardian
• TCP 2020 from Prometheus API service | | Guardian | An agent running in each managed cluster that proxies communication between the $[prodname] tunnel server and your managed cluster. Secured using TLS tunnels.
| • Port 9000 to tunnel server
• TCP 6443 to Kubernetes API server
• TCP 6443 from $[prodname] components | | Installation endpoints | Endpoints at `*.calicocloud.io` and `*.projectcalico.org`. | TCP 443 for both | | Intrusion detection controller | Handles integrations with threat intelligence feeds and $[prodname] custom alerts. | • TCP 8080 to Guardian
• TCP 6443 to Kubernetes API server | @@ -42,7 +42,7 @@ The following diagram shows the major components in a managed cluster, followed | kube-controllers | Monitors the Kubernetes API and performs actions based on cluster state. $[prodname] kube-controllers container includes these controllers:

• Node
• Service
• Federated services
• Authorization
| • TCP 9094 from Prometheus API service
• TCP 6443 to Kubernetes API server | | Log storage | Storage for logs (flows, L7, DNS, audit). Data for each managed cluster is isolated and protected against unauthorized access. | n/a | | Packet capture API | Retrieves capture files (pcap format) generated by a packet capture for use with network protocol analysis tools like Wireshark. Packet capture data is visible in the web console and Service Graph. | • TCP 8449 Guardian to Packet Capture API
• TCP 6443 to Kubernetes API server | -| Prometheus API service | Collects metrics from $[prodname] components and makes the metrics available to the web console. | • TCP 6443 to Kubernetes API server
• TCP 9080 to Fluentd
• TCP 9900 and 9081 to Prometheus API service | +| Prometheus API service | Collects metrics from $[prodname] components and makes the metrics available to the web console. | • TCP 6443 to Kubernetes API server
• TCP 2020 to Fluent Bit
• TCP 9900 and 9081 to Prometheus API service | | Tigera API server | Allows users to manage $[prodname] resources such as policies and tiers through kubectl or the Kubernetes API server. | • TCP 9095 to Prometheus API service
• TCP 8080 from Kubernetes API server | | Typha | Increases scale by reducing each node’s impact on the datastore. | TCP 5473 from calico-node to Typha | | User access to the web console | Authenticated users can access the browser-based the web console, which provides network traffic visibility and troubleshooting, centralized multi-cluster management, threat-defense, container threat detection, policy lifecycle management, scan images for vulnerabilities, and compliance for multiple roles/stakeholders. | Port 443 to $[prodname] tunnel server | \ No newline at end of file diff --git a/calico-cloud/get-started/operator-checklist.mdx b/calico-cloud/get-started/operator-checklist.mdx index 4f10fc6a53..bf27847929 100644 --- a/calico-cloud/get-started/operator-checklist.mdx +++ b/calico-cloud/get-started/operator-checklist.mdx @@ -89,7 +89,7 @@ If you are using the **AWS or Azure CNI plugin**, a degraded state is likely bec **Sample output** -In the following example, the typha component has an issue because it is showing `AVAILABLE: FALSE`, and `DEGRADED: TRUE`. To understand details of $[prodname] components, see [Deep dive into custom resources](#deep-dive-into-custom-resources). +In the following example, the `typha` component has an issue because it is showing `AVAILABLE: FALSE`, and `DEGRADED: TRUE`. To understand details of $[prodname] components, see [Deep dive into custom resources](#deep-dive-into-custom-resources). ```yaml apiVersion: v1 @@ -400,7 +400,7 @@ kubectl get tigerastatus | | NAME | AVAILABLE | PROGRESSING | DEGRADED | SINCE | | --- | ----------------------------- | --------- | ----------- | -------- | ----- | -| 1 | apiserver | TRUE | FALSE | FALSE | 10m | +| 1 | `apiserver` | TRUE | FALSE | FALSE | 10m | | 2 | calico | TRUE | FALSE | FALSE | 11m | | 3 | cloud-core | TRUE | FALSE | FALSE | 11m | | 4 | compliance | TRUE | FALSE | FALSE | 9m39s | @@ -410,9 +410,9 @@ kubectl get tigerastatus | 8 | monitor | TRUE | FALSE | FALSE | 11m | | 9 | runtime-security | TRUE | FALSE | FALSE | 10m | -**1 - api server** +**1 - API server** -`apiserver` is a required component that is an aggregated api-server. It is required for things like applying the tigera license. If `tigerastatus` reports it as unavailable or degraded, check the pods and logs in the `calico-system`namespace. For example, +`apiserver` is a required component that is an aggregated API server. It is required for things like applying the Tigera license. If `tigerastatus` reports it as unavailable or degraded, check the pods and logs in the `calico-system` namespace. For example, ```bash kubectl get pods -n calico-system @@ -493,19 +493,19 @@ intrusion-detection-es-job-installer-xm22v 1/1 Running 0 6 **6 - log-collector** -`log-collector` collects flow and other logs and forwards them to $[prodname]. Check the pods and logs in the `tigera-fluentd` namespace. You should have one pod running on each of your nodes. +`log-collector` collects flow and other logs and forwards them to $[prodname]. Check the `calico-fluent-bit` pods and logs in the `calico-system` namespace. You should have one pod running on each of your nodes. ```bash -kubectl get pods -n tigera-fluentd +kubectl get pods -n calico-system -l k8s-app=calico-fluent-bit ``` ``` -NAME READY STATUS RESTARTS AGE -fluentd-node-5mzh6 1/1 Running 0 70m -fluentd-node-7vmxw 1/1 Running 0 70m -fluentd-node-bbc4p 1/1 Running 0 70m -fluentd-node-chfz4 1/1 Running 0 70m -fluentd-node-d6f56 1/1 Running 0 70m +NAME READY STATUS RESTARTS AGE +calico-fluent-bit-5mzh6 1/1 Running 0 70m +calico-fluent-bit-7vmxw 1/1 Running 0 70m +calico-fluent-bit-bbc4p 1/1 Running 0 70m +calico-fluent-bit-chfz4 1/1 Running 0 70m +calico-fluent-bit-d6f56 1/1 Running 0 70m ``` **7 - management-cluster-connection** @@ -557,7 +557,7 @@ cmEtdm9sdHJvbjAeFw0yMDEyMjExOTA1MzhaFw0yNTEyMjAxOTA1MzhaMBkxFzAV **8 - monitor** -`monitor` is responsible for configuring prometheus and associated custom resources. Check the pods and logs in the `tigera-prometheus` namespace. +`monitor` is responsible for configuring Prometheus and associated custom resources. Check the pods and logs in the `tigera-prometheus` namespace. ```bash $ kubectl get pods -n tigera-prometheus @@ -643,8 +643,8 @@ If cluster does not have enough capacity, it will not be able to deploy pods. Th The high-level components $[prodname] needs to run are: -- Per node: 1 fluentd, 1 compliance benchmarker -- On top of per node: 3 alertmanager (from statefulset), 1 prometheus, 1 prometheus operator, 1 kube-controllers, 2 compliance snapshotter and controller, 1 guardian, 1 ids controller, 1 apiserver +- Per node: 1 `fluent-bit`, 1 `compliance-benchmarker` +- On top of per node: 3 `alertmanager` (from StatefulSet), 1 `prometheus`, 1 `prometheus` operator, 1 kube-controllers, 2 compliance snapshotter and controller, 1 guardian, 1 ids controller, 1 `apiserver` Some clusters have limited pod-networked pod capacity. diff --git a/calico-cloud/observability/elastic/dns/filtering-dns.mdx b/calico-cloud/observability/elastic/dns/filtering-dns.mdx index 6705170172..52d6bfd0e7 100644 --- a/calico-cloud/observability/elastic/dns/filtering-dns.mdx +++ b/calico-cloud/observability/elastic/dns/filtering-dns.mdx @@ -1,5 +1,5 @@ --- -description: Suppress low-value Calico Cloud DNS log entries with Fluentd filters configured through a ConfigMap in the operator namespace of connected clusters. +description: Suppress low-value Calico Cloud DNS log entries with Fluent Bit filters configured through a ConfigMap in the operator namespace of connected clusters. --- # Filter DNS logs @@ -18,46 +18,53 @@ To enable DNS log filtering, follow these steps: your desired filter using [Filter configuration files](#filter-configuration-files). If you are also adding [flow filters](../flow/filtering.mdx) also add the `flow` file to the directory. -1. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace +1. Create the `fluent-bit-filters` ConfigMap in the `tigera-operator` namespace with the following command. ```bash - kubectl create configmap fluentd-filters -n tigera-operator --from-file=filters + kubectl create configmap fluent-bit-filters -n tigera-operator --from-file=filters ``` +The operator inserts the filters inline into the log collector configuration and rolls the +`calico-fluent-bit` daemonset automatically. + ## Filter configuration files -The filters defined by the ConfigMap are inserted into the fluentd configuration file. -The [upstream fluentd documentation](https://docs.fluentd.org/filter/grep) -describes how to write fluentd filters. The [DNS log schema](dns-logs.mdx) can be referred to -for the specification of the various fields you can filter based on. Remember to ensure -that the config file is properly indented in the ConfigMap. +Each file holds a YAML list of Fluent Bit filter entries. The +[upstream Fluent Bit documentation](https://docs.fluentbit.io/manual/data-pipeline/filters/grep) +describes how to write the `grep` filter used in the examples below; the +`calico-fluent-bit` log collector also ships the `record_modifier`, `parser`, and +`lua` filters. Filters in the `dns` file are applied to DNS logs automatically; +you do not need to set a `match` on each entry. The [DNS log schema](dns-logs.mdx) +can be referred to for the specification of the various fields you can filter based on. + +:::note Upgrading from a release that used Fluentd + +Earlier releases collected logs with Fluentd and read filters in Fluentd `` +syntax from a ConfigMap named `fluentd-filters`. That ConfigMap is no longer read, +and Fluentd filter syntax cannot be translated automatically. Recreate your filters +as Fluent Bit YAML filter lists under the new `fluent-bit-filters` name. If a filter +key does not parse as Fluent Bit YAML, the operator skips that filter, reports a +warning on the `tigera status` output naming the offending key, and continues to +ship unfiltered logs. + +::: ## Example 1: filter out cluster-internal lookups This example filters out lookups for domain names ending with ".cluster.local". More -logs could be filtered by adjusting the regular expression "pattern", or by adding -additional `exclude` blocks. +logs could be filtered by adjusting the regular expression, or by adding +additional `exclude` rules. -``` - - @type grep - - key qname - pattern /\.cluster\.local$/ - - +```yaml +- name: grep + exclude: qname \.cluster\.local$ ``` ## Example 2: keep logs only for particular domain names This example will filter out all logs _except_ those for domain names ending `.co.uk`. -``` - - @type grep - - key qname - pattern /\.co\.uk$/ - - +```yaml +- name: grep + regex: qname \.co\.uk$ ``` diff --git a/calico-cloud/observability/elastic/flow/filtering.mdx b/calico-cloud/observability/elastic/flow/filtering.mdx index 15738b32c8..9870aa7cbf 100644 --- a/calico-cloud/observability/elastic/flow/filtering.mdx +++ b/calico-cloud/observability/elastic/flow/filtering.mdx @@ -1,5 +1,5 @@ --- -description: Filter Calico Cloud flow logs through Fluentd to drop low-significance traffic and reduce managed Elasticsearch volume and cost. +description: Filter Calico Cloud flow logs through Fluent Bit to drop low-significance traffic and reduce managed Elasticsearch volume and cost. --- # Filter flow logs @@ -53,56 +53,49 @@ reporter packetsIn packetsOut bytesIn bytesOut action ### Create flow log filters -Create your [fluentd filters](https://docs.fluentd.org/filter/grep). +Filters are written as a YAML list of [Fluent Bit filter](https://docs.fluentbit.io/manual/data-pipeline/filters/grep) entries. The `calico-fluent-bit` log collector ships the `grep`, `record_modifier`, `parser`, and `lua` filters. Filters you add under the `flow` key are applied to flow logs automatically; you do not need to set a `match` on each entry. **Example: filter out a specific namespace** -This example filters out all flow logs whose source or destination namespace is "dev". Additional namespaces could be filtered by adjusting the regular expression "pattern"s, or by adding additional `exclude` blocks. +This example filters out all flow logs whose source or destination namespace is `dev`. A record is dropped when it matches any of the `exclude` rules; additional namespaces could be filtered by adjusting the regular expressions, or by adding more `exclude` rules. -``` - - @type grep - - key source_namespace - pattern dev - - - key dest_namespace - pattern dev - - +```yaml +- name: grep + exclude: + - source_namespace dev + - dest_namespace dev ``` **Example: filter out internet traffic to a specific deployment** -This example filters inbound internet traffic to the deployment with pods named, `nginx-internet-*`. Note the use of the `and` directive to filter out traffic that is both to the deployment, and from the internet (source `pub`). +This example filters inbound internet traffic to the deployment with pods named, `nginx-internet-*`. Note the use of `logical_op: and` to filter out only the traffic that is both to the deployment, and from the internet (source `pub`). -``` - - @type grep - - - key dest_name_aggr - pattern ^nginx-internet - - - key source_name_aggr - pattern pub - - - +```yaml +- name: grep + logical_op: and + exclude: + - dest_name_aggr ^nginx-internet + - source_name_aggr pub ``` ### Add filters to ConfigMap file -1. Create a `filters` directory with a file called `flow` with your desired filters. If you are also adding [dns filters](../dns/filtering-dns.mdx), add the `dns` file to the directory. +1. Create a `filters` directory with a file called `flow` with your desired filters. If you are also adding [DNS filters](../dns/filtering-dns.mdx), add the `dns` file to the directory. -1. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace with the following command. +1. Create the `fluent-bit-filters` ConfigMap in the `tigera-operator` namespace with the following command. ```bash - kubectl create configmap fluentd-filters -n tigera-operator --from-file=filters + kubectl create configmap fluent-bit-filters -n tigera-operator --from-file=filters ``` +The operator inserts the filters inline into the log collector configuration and rolls the `calico-fluent-bit` DaemonSet automatically. + +:::note Upgrading from a release that used Fluentd + +Earlier releases collected logs with Fluentd and read filters in Fluentd `` syntax from a ConfigMap named `fluentd-filters`. That ConfigMap is no longer read, and Fluentd filter syntax cannot be translated automatically. Recreate your filters as Fluent Bit YAML filter lists under the new `fluent-bit-filters` name. If a filter key does not parse as Fluent Bit YAML, the operator skips that filter, reports a warning on the `tigera status` output naming the offending key, and continues to ship unfiltered logs. + +::: + ## Additional resources - [Flow log aggregation](aggregation.mdx) diff --git a/calico-cloud/observability/elastic/overview.mdx b/calico-cloud/observability/elastic/overview.mdx index be619f8199..30adc18ec5 100644 --- a/calico-cloud/observability/elastic/overview.mdx +++ b/calico-cloud/observability/elastic/overview.mdx @@ -29,23 +29,23 @@ Elasticsearch logs provide the visibility and troubleshooting backend for $[prod | Log type | Description | Log source | RBAC | Index | | -------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------ | ----------------------------- | -| flow | Network flows for workloads: source and destination namespaces, pods, labels, and policies | $[prodname] cnx-node (Felix) | `flows` | `tigera_secure_ee_flows` | -| audit | Audit logs for $[prodname] resources | $[prodname] apiserver | `audit_ee` | `tigera_secure_ee_audit_ee` | -| | Audit logs for Kubernetes resources | Kubernetes apiserver | `audit_kube` | `tigera_secure_ee_audit_kube` | +| `flow` | Network flows for workloads: source and destination namespaces, pods, labels, and policies | $[prodname] cnx-node (Felix) | `flows` | `tigera_secure_ee_flows` | +| `audit` | Audit logs for $[prodname] resources | $[prodname] API server | `audit_ee` | `tigera_secure_ee_audit_ee` | +| | Audit logs for Kubernetes resources | Kubernetes API server | `audit_kube` | `tigera_secure_ee_audit_kube` | | | | Both audit logs above | `audit*` | `tigera_secure_ee_audit*` | -| bgp | $[prodname] networking BGP peering and route propagation | $[prodname] cnx-node (BIRD) | `ee_bgp` | `tigera_secure_ee_bgp.*` | -| dns | DNS lookups and responses from $[prodname] domain-based policy | $[prodname] cnx-node (Felix) | `ee_dns` | `tigera_secure_ee_dns` | -| ids | $[prodname] intrusion detection events: suspicious IPs, suspicious domains, and global alerts | $[prodname] intrusion-detection-controller | `ee_events` | `tigera_secure_ee_events` | +| `bgp` | $[prodname] networking BGP peering and route propagation | $[prodname] cnx-node (BIRD) | `ee_bgp` | `tigera_secure_ee_bgp.*` | +| `dns` | DNS lookups and responses from $[prodname] domain-based policy | $[prodname] cnx-node (Felix) | `ee_dns` | `tigera_secure_ee_dns` | +| `ids` | $[prodname] intrusion detection events: suspicious IPs, suspicious domains, and global alerts | $[prodname] intrusion-detection-controller | `ee_events` | `tigera_secure_ee_events` | :::note -Because of their high-volume, flow and dns logs support aggregation. +Because of their high-volume, flow and DNS logs support aggregation. ::: ### Default log configuration and security -$[prodname] automatically installs fluentd on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk. +$[prodname] automatically installs Fluent Bit on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk. $[prodname] enables user authentication in Elasticsearch, and secures access to Elasticsearch and Kibana instances using network policy. diff --git a/calico-cloud/observability/kube-audit.mdx b/calico-cloud/observability/kube-audit.mdx index 0c46e65921..b6e492d4a9 100644 --- a/calico-cloud/observability/kube-audit.mdx +++ b/calico-cloud/observability/kube-audit.mdx @@ -84,7 +84,7 @@ rules: The following updates require a restart to the Kubernetes API Server. -To enable Kubernetes resource audit logs to be read by $[prodname] in fluentd, follow these steps. +To enable Kubernetes resource audit logs to be read by the $[prodname] Fluent Bit log collector, follow these steps. On the Kubernetes API Server, update these flags. diff --git a/calico-cloud/observability/packetcapture.mdx b/calico-cloud/observability/packetcapture.mdx index e99031de8e..03b14dc9e3 100644 --- a/calico-cloud/observability/packetcapture.mdx +++ b/calico-cloud/observability/packetcapture.mdx @@ -12,7 +12,7 @@ Capture live traffic inside a Kubernetes cluster, and export to visualization to $[prodname] packet capture is implemented in a Kubernetes-native way so you can troubleshoot service/application connectivity issues and performance issues. You can start a packet capture in the web console Service Graph, or using the CLI. -Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, daemonset, statefulset, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. +Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, DaemonSet, StatefulSet, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. With $[prodname] packet capture you can: @@ -79,7 +79,7 @@ EOF ### Packet capture in Service Graph -1. Select an endpoint from the service graph (for example, namespace, service, replica set, daemonset, statefulset, or pod), right-click, and select **Initiate packet capture**. +1. Select an endpoint from the service graph (for example, namespace, service, replica set, DaemonSet, StatefulSet, or pod), right-click, and select **Initiate packet capture**. ![start-capture](/img/calico-enterprise/start-capture.png) @@ -226,19 +226,19 @@ status: Get the pod on the node with the packet capture that you want. ```bash -kubectl get pods -n tigera-fluentd --no-headers --field-selector spec.nodeName="" +kubectl get pods -n calico-system -l k8s-app=calico-node --no-headers --field-selector spec.nodeName="" ``` Copy the packet capture using the pod information. ```bash -kubectl cp tigera-fluentd/:var/log/calico/pcap/sample/sample-capture/ . +kubectl cp calico-system/:var/log/calico/pcap/sample/sample-capture/ . ``` **Delete packet capture files** ```bash -kubectl exec -it tigera-fluentd/ -- sh -c "rm -r /var/log/calico/pcap/sample/sample-capture/" +kubectl exec -it -n calico-system -- sh -c "rm -r /var/log/calico/pcap/sample/sample-capture/" ``` ### Store and rotate capture files diff --git a/calico-cloud/operations/comms/index.mdx b/calico-cloud/operations/comms/index.mdx index 5c0aeee98e..c7e8bda22b 100644 --- a/calico-cloud/operations/comms/index.mdx +++ b/calico-cloud/operations/comms/index.mdx @@ -47,18 +47,18 @@ The **Deployed to** column shows the namespace where the operator places the sec | Secret name | DNS name(s) | Deployed to | Owner resource | |---|---|---|---| -| `calico-apiserver-certs` | `calico-api` | `calico-system` | APIServer/tigera-secure | -| `calico-kube-controllers-metrics-tls` | `calico-kube-controllers-metrics` | `calico-system` | Installation/default | -| `calico-node-prometheus-client-tls` | `calico-node-prometheus-client-tls` | `tigera-prometheus` | Monitor/tigera-secure | -| `calico-node-prometheus-server-tls` | `calico-node-metrics` | `calico-system` | Installation/default | -| `calico-node-prometheus-tls` | `prometheus-http-api` | `tigera-prometheus` | Monitor/tigera-secure | -| `deep-packet-inspection-tls` | `intrusion-detection-tls` | `tigera-dpi` | IntrusionDetection/tigera-secure | -| `node-certs` | `typha-client` | `calico-system` | Installation/default | -| `node-certs` | `typha-client` | `tigera-dpi` | IntrusionDetection/tigera-secure | -| `tigera-ee-elasticsearch-metrics-tls` | `tigera-elasticsearch-metrics` | `tigera-elasticsearch` | LogStorage/tigera-secure | -| `tigera-fluentd-prometheus-tls` | `fluentd-http-input` | `tigera-fluentd` | LogCollector/tigera-secure | -| `typha-certs` | `typha-server` | `calico-system` | Installation/default | -| `typha-certs-noncluster-host` | `typha-server-noncluster-host` | `calico-system` | Installation/default | +| `calico-apiserver-certs` | `calico-api` | `calico-system` | `APIServer/tigera-secure` | +| `calico-kube-controllers-metrics-tls` | `calico-kube-controllers-metrics` | `calico-system` | `Installation/default` | +| `calico-node-prometheus-client-tls` | `calico-node-prometheus-client-tls` | `tigera-prometheus` | `Monitor/tigera-secure` | +| `calico-node-prometheus-server-tls` | `calico-node-metrics` | `calico-system` | `Installation/default` | +| `calico-node-prometheus-tls` | `prometheus-http-api` | `tigera-prometheus` | `Monitor/tigera-secure` | +| `deep-packet-inspection-tls` | `intrusion-detection-tls` | `tigera-dpi` | `IntrusionDetection/tigera-secure` | +| `node-certs` | `typha-client` | `calico-system` | `Installation/default` | +| `node-certs` | `typha-client` | `tigera-dpi` | `IntrusionDetection/tigera-secure` | +| `tigera-ee-elasticsearch-metrics-tls` | `tigera-elasticsearch-metrics` | `tigera-elasticsearch` | `LogStorage/tigera-secure` | +| `calico-fluent-bit-tls` | `calico-fluent-bit-http-input` | `calico-system` | `LogCollector/tigera-secure` | +| `typha-certs` | `typha-server` | `calico-system` | `Installation/default` | +| `typha-certs-noncluster-host` | `typha-server-noncluster-host` | `calico-system` | `Installation/default` | :::tip diff --git a/calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx b/calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx index 842f83e06b..5edc4b6222 100644 --- a/calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx +++ b/calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx @@ -1,12 +1,12 @@ --- -description: Track Calico Cloud Fluentd metrics in Prometheus to alert on flow, DNS, and audit log collection disruptions on connected clusters. +description: Track Calico Cloud Fluent Bit metrics in Prometheus to alert on flow, DNS, and audit log collection disruptions on connected clusters. --- -# Fluentd metrics +# Fluent Bit metrics ## Big picture -Use the Prometheus monitoring and alerting tool for Fluentd metrics to ensure continuous network visibility. +Use the Prometheus monitoring and alerting tool for Fluent Bit metrics to ensure continuous network visibility. ## Value @@ -16,14 +16,14 @@ Platform engineering teams rely on logs for visibility into their networks. If c | Component | Description | | ---------- | ------------------------------------------------------------ | -| Prometheus | Monitoring tool that scrapes metrics from instrumented jobs and displays time series data in a visualizer (such as Grafana). For $[prodname], the “jobs” that Prometheus can harvest metrics from the Fluentd component. | -| Fluentd | Sends $[prodname] logs to Elasticsearch for storage. | +| Prometheus | Monitoring tool that scrapes metrics from instrumented jobs and displays time series data in a visualizer (such as Grafana). For $[prodname], the “jobs” that Prometheus can harvest metrics from the Fluent Bit component. | +| Fluent Bit | Ships $[prodname] logs to the log storage backend. | ## How to -### Create Prometheus alerts for Fluentd +### Create Prometheus alerts for Fluent Bit -The following example creates a Prometheus rule to monitor some important Fluentd metrics, and alert when they have crossed certain thresholds: +The following example creates a Prometheus rule to monitor some important Fluent Bit metrics, and alert when they have crossed certain thresholds: ```yaml noValidation apiVersion: monitoring.coreos.com/v1 @@ -38,18 +38,28 @@ spec: groups: - name: tigera-log-collection.rules rules: - - alert: FluentdPodConsistentlyLowBufferSpace - expr: avg_over_time(fluentd_output_status_buffer_available_space_ratio[5m]) < 75 + - alert: FluentBitPodConsistentlyLowBufferSpace + expr: avg_over_time(fluentbit_output_chunk_available_capacity_percent[5m]) < 75 labels: severity: Warning annotations: - summary: "Fluentd pod {{$labels.pod}}'s buffer space is consistently below 75 percent capacity." - description: "Fluentd pod {{$labels.pod}} has very low buffer space. There may be connection issues between Elasticsearch -and Fluentd or there are too many logs to write out, check the logs for the Fluentd pod." + summary: "Fluent Bit pod {{$labels.pod}}'s output buffer space is consistently below 75 percent capacity." + description: "Fluent Bit pod {{$labels.pod}} has very low buffer space for output {{$labels.name}}. There may be +connection issues between Fluent Bit and the log destination, or there are too many logs to write out; check the logs +for the Fluent Bit pod." + - alert: FluentBitPodDroppingLogs + expr: rate(fluentbit_output_retries_failed_total[5m]) > 0 + labels: + severity: Warning + annotations: + summary: "Fluent Bit pod {{$labels.pod}} is dropping log chunks." + description: "Fluent Bit pod {{$labels.pod}} gave up retrying to deliver log chunks for output {{$labels.name}}. +Logs are being lost; check the logs for the Fluent Bit pod and the health of the log destination." ``` #### The alerts created in the example are described as follows: | Alert | Severity | Requires | Issue/reason | | ---------------------------------------- | --------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **FluentdPodConsistentlyLowBufferSpace** | Non-critical, warning | Immediate investigation to ensure logs are being gathered correctly. | A Fluentd pod’s available buffer size has averaged less than 75% over the last 5 minutes.

This could mean Fluentd is having trouble communicating with the Elasticsearch cluster, the Elasticsearch cluster is down, or there are simply too many logs to process. | +| **FluentBitPodConsistentlyLowBufferSpace** | Non-critical, warning | Immediate investigation to ensure logs are being gathered correctly. | A Fluent Bit pod’s available output buffer capacity has averaged less than 75% over the last 5 minutes.

This could mean Fluent Bit is having trouble communicating with a log destination, the destination is down, or there are simply too many logs to process. | +| **FluentBitPodDroppingLogs** | Non-critical, warning | Immediate investigation to ensure logs are not being lost. | A Fluent Bit pod exhausted the retries for one or more log chunks and dropped them. | diff --git a/calico-cloud/operations/monitor/prometheus/byo-prometheus.mdx b/calico-cloud/operations/monitor/prometheus/byo-prometheus.mdx index 80aaecdc4e..1d38f2b120 100644 --- a/calico-cloud/operations/monitor/prometheus/byo-prometheus.mdx +++ b/calico-cloud/operations/monitor/prometheus/byo-prometheus.mdx @@ -15,12 +15,12 @@ Scrape $[prodname] metrics for Bring Your Own (BYO) Prometheus. $[prodname] uses the Prometheus monitoring tool to scrape metrics from instrumented jobs, and displays time-series data in a visualizer such as Grafana. You can scrape the following time-series metrics for $[prodname] components to your own Prometheus: -- elasticsearch -- fluentd -- calico-node -- kube-controllers -- felix -- typha (not enabled by default) +- `elasticsearch` +- `fluent-bit` +- `calico-node` +- `kube-controllers` +- `felix` +- `typha` (not enabled by default) ## Before you begin @@ -41,7 +41,7 @@ For the supported version of Prometheus in this release, see the [Release Notes] In this section we create a service monitor that scrapes all enabled metrics. To enable metrics that are not enabled by default, please consult the [next section](#scrape-metrics-from-specific-components). -The following example shows a Prometheus server installed in namespace "external-prometheus" with a `serviceMonitorSelector` that selects all service monitors with the label `k8s-app=tigera-external-prometheus`. +The following example shows a Prometheus server installed in namespace `external-prometheus` with a `serviceMonitorSelector` that selects all service monitors with the label `k8s-app=tigera-external-prometheus`. 1. Save the following configuration in a file called `monitor.yaml`. @@ -123,34 +123,13 @@ kubectl apply -f $[filesUrl_CE]/manifests/prometheus/elasticsearch-metrics-servi The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. - - -**Configure TLS certificates** - -1. Copy the required secret and configmap to your namespace. -2. Save the manifest of the required TLS secret and CA configmap. - - ```bash - kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml - ``` - - ```bash - kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml - ``` - -3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` and change the namespace to the namespace where your prometheus instance is running. -4. Apply the manifests to your cluster. - - ```bash - kubectl apply -f calico-node-prometheus-client-tls.yaml - ``` - - ```bash - kubectl apply -f tigera-ca-bundle.yaml - ``` + **Create the service monitor** +Fluent Bit serves its metrics over plain HTTP (port 2020, path `/api/v2/metrics/prometheus`), so no TLS +configuration is required. + Apply the ServiceMonitor to the namespace where Prometheus is running. ```bash @@ -158,11 +137,20 @@ export NAMESPACE= ``` ```bash -kubectl apply -f $[filesUrl_CE]/manifests/prometheus/fluentd-metrics-service-monitor.yaml -n $NAMESPACE +kubectl apply -f $[filesUrl_CE]/manifests/prometheus/fluent-bit-metrics-service-monitor.yaml -n $NAMESPACE ``` The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. +:::note + +Access to the Fluent Bit metrics port is restricted by the `calico-system.allow-calico-fluent-bit` network policy, which +by default admits only the `tigera-prometheus` namespace. Create a policy that allows your BYO Prometheus +namespace to reach port 2020 on the `calico-fluent-bit` pods (see +[Create policy to secure traffic between pods](#create-policy-to-secure-traffic-between-pods)). + +::: + @@ -361,7 +349,7 @@ The .yamls have no namespace defined so when you apply `kubectl`, it is applied kubectl port-forward pod/byo-prometheus-pod 9090:9090 -n $NAMESPACE ``` -1. Browse to the Prometheus dashboard: http://localhost:9090. +1. Browse to the Prometheus dashboard: `http://localhost:9090`. 1. In the Expression text box, enter your metric name and click the **Execute** button. diff --git a/calico-cloud/reference/component-resources/configure-resources.mdx b/calico-cloud/reference/component-resources/configure-resources.mdx index 646748356a..f6ff0de77c 100644 --- a/calico-cloud/reference/component-resources/configure-resources.mdx +++ b/calico-cloud/reference/component-resources/configure-resources.mdx @@ -550,14 +550,14 @@ This command will output the configured resource requests and limits for the Int ## LogCollector custom resource -The [LogCollector](../../reference/installation/api.mdx#logcollector) CR provides a way to configure resources for FluentdDaemonSet, EKSLogForwarderDeployment. +The [LogCollector](../../reference/installation/api.mdx#logcollector) CR provides a way to configure resources for CalicoFluentBitDaemonSet, EKSLogForwarderDeployment. -### FluentdDaemonSet. +### CalicoFluentBitDaemonSet. -To configure resource specification for the [FluentdDaemonSet](../../reference/installation/api.mdx#fluentddaemonset), patch the LogCollector CR using the below command: +To configure resource specification for the [CalicoFluentBitDaemonSet](../../reference/installation/api.mdx#logcollector), patch the LogCollector CR using the below command: ```bash -kubectl patch logcollector tigera-secure --type=merge --patch='{"spec": {"fluentdDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"fluentd","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' +kubectl patch logcollector tigera-secure --type=merge --patch='{"spec": {"calicoFluentBitDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"calico-fluent-bit","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). @@ -566,14 +566,14 @@ This command sets the CPU request to 100 milliCPU (mCPU) and the memory request You can verify the configured resources using the following command: ```bash -kubectl get daemonset.apps/fluentd-node -n tigera-fluentd -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' +kubectl get daemonset.apps/calico-fluent-bit -n calico-system -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` -This command will output the configured resource requests and limits for the FluentdDaemonSet in JSON format. +This command will output the configured resource requests and limits for the CalicoFluentBitDaemonSet in JSON format. ```bash { - "name": "fluentd", + "name": "calico-fluent-bit", "resources": { "limits": { "cpu": "1", @@ -602,7 +602,7 @@ This command sets the CPU request to 100 milliCPU (mCPU) and the memory request You can verify the configured resources using the following command: ```bash -kubectl get deployment.apps/eks-log-forwarder -n tigera-fluentd -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' +kubectl get deployment.apps/eks-log-forwarder -n calico-system -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the EKSLogForwarderDeployment in JSON format. diff --git a/calico-enterprise/_includes/components/ReqsSys.js b/calico-enterprise/_includes/components/ReqsSys.js index 5637c945c3..7f5f068c49 100644 --- a/calico-enterprise/_includes/components/ReqsSys.js +++ b/calico-enterprise/_includes/components/ReqsSys.js @@ -293,7 +293,7 @@ function NetworkRequirementsEnt(props) { Logs and storage - Elasticsearch with fluentd datastore + Elasticsearch log storage TCP 9200 (default) diff --git a/calico-enterprise/_includes/content/_default-install.mdx b/calico-enterprise/_includes/content/_default-install.mdx index e458061512..b09c44e320 100644 --- a/calico-enterprise/_includes/content/_default-install.mdx +++ b/calico-enterprise/_includes/content/_default-install.mdx @@ -4,7 +4,7 @@ | Networking | Default networking implementation based on your provider and platform.
Default IP pool/CIDR range. | | Tiered network policy | Tiered network policy components.
Existing Kubernetes clusters are put in default tier.
$[prodname] components are secured with network policy. | | User interface | the $[prodname] web console user interface (with a default of “no access outside the cluster”). | -| Logs and data | All nodes configured for log data collection using fluentdLog storage for a single node.
One instance of Elasticsearch and Kibana. | +| Logs and data | All nodes configured for log data collection using Fluent Bit.
Log storage for a single node.
One instance of Elasticsearch and Kibana. | | Threat prevention and detection | Components and dashboards for tracing and blocking suspicious activity. | | Compliance reporting | Components and dashboards to assess Kubernetes workloads and environments for regulatory compliance to enforce controls, and generate audit and evidence data. | | Scaling | Built-in node scaling using Typha. | diff --git a/calico-enterprise/observability/elastic/dns/filtering-dns.mdx b/calico-enterprise/observability/elastic/dns/filtering-dns.mdx index 6d0b4d441b..93a52e6ade 100644 --- a/calico-enterprise/observability/elastic/dns/filtering-dns.mdx +++ b/calico-enterprise/observability/elastic/dns/filtering-dns.mdx @@ -1,5 +1,5 @@ --- -description: Suppress low-value Calico Enterprise DNS log entries with Fluentd filters configured through a ConfigMap in the operator namespace. +description: Suppress low-value Calico Enterprise DNS log entries with Fluent Bit filters configured through a ConfigMap in the operator namespace. --- # Filter DNS logs @@ -18,46 +18,53 @@ To enable DNS log filtering, follow these steps: your desired filter using [Filter configuration files](#filter-configuration-files). If you are also adding [flow filters](../flow/filtering.mdx) also add the `flow` file to the directory. -1. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace +1. Create the `fluent-bit-filters` ConfigMap in the `tigera-operator` namespace with the following command. ```bash - kubectl create configmap fluentd-filters -n tigera-operator --from-file=filters + kubectl create configmap fluent-bit-filters -n tigera-operator --from-file=filters ``` +The operator inserts the filters inline into the log collector configuration and rolls the +`calico-fluent-bit` daemonset automatically. + ## Filter configuration files -The filters defined by the ConfigMap are inserted into the fluentd configuration file. -The [upstream fluentd documentation](https://docs.fluentd.org/filter/grep) -describes how to write fluentd filters. The [DNS log schema](dns-logs.mdx) can be referred to -for the specification of the various fields you can filter based on. Remember to ensure -that the config file is properly indented in the ConfigMap. +Each file holds a YAML list of Fluent Bit filter entries. The +[upstream Fluent Bit documentation](https://docs.fluentbit.io/manual/data-pipeline/filters/grep) +describes how to write the `grep` filter used in the examples below; the +`calico-fluent-bit` log collector also ships the `record_modifier`, `parser`, and +`lua` filters. Filters in the `dns` file are applied to DNS logs automatically; +you do not need to set a `match` on each entry. The [DNS log schema](dns-logs.mdx) +can be referred to for the specification of the various fields you can filter based on. + +:::note Upgrading from a release that used Fluentd + +Earlier releases collected logs with Fluentd and read filters in Fluentd `` +syntax from a ConfigMap named `fluentd-filters`. That ConfigMap is no longer read, +and Fluentd filter syntax cannot be translated automatically. Recreate your filters +as Fluent Bit YAML filter lists under the new `fluent-bit-filters` name. If a filter +key does not parse as Fluent Bit YAML, the operator skips that filter, reports a +warning on the `tigera status` output naming the offending key, and continues to +ship unfiltered logs. + +::: ## Example 1: filter out cluster-internal lookups This example filters out lookups for domain names ending with ".cluster.local". More -logs could be filtered by adjusting the regular expression "pattern", or by adding -additional `exclude` blocks. +logs could be filtered by adjusting the regular expression, or by adding +additional `exclude` rules. -``` - - @type grep - - key qname - pattern /\.cluster\.local$/ - - +```yaml +- name: grep + exclude: qname \.cluster\.local$ ``` ## Example 2: keep logs only for particular domain names This example will filter out all logs _except_ those for domain names ending `.co.uk`. -``` - - @type grep - - key qname - pattern /\.co\.uk$/ - - +```yaml +- name: grep + regex: qname \.co\.uk$ ``` diff --git a/calico-enterprise/observability/elastic/flow/filtering.mdx b/calico-enterprise/observability/elastic/flow/filtering.mdx index bb10e35561..90e3601495 100644 --- a/calico-enterprise/observability/elastic/flow/filtering.mdx +++ b/calico-enterprise/observability/elastic/flow/filtering.mdx @@ -1,5 +1,5 @@ --- -description: Filter Calico Enterprise flow logs through Fluentd to drop low-significance traffic and reduce in-cluster Elasticsearch volume and cost. +description: Filter Calico Enterprise flow logs through Fluent Bit to drop low-significance traffic and reduce in-cluster Elasticsearch volume and cost. --- # Filter flow logs @@ -53,56 +53,49 @@ reporter packetsIn packetsOut bytesIn bytesOut action ### Create flow log filters -Create your [fluentd filters](https://docs.fluentd.org/filter/grep). +Filters are written as a YAML list of [Fluent Bit filter](https://docs.fluentbit.io/manual/data-pipeline/filters/grep) entries. The `calico-fluent-bit` log collector ships the `grep`, `record_modifier`, `parser`, and `lua` filters. Filters you add under the `flow` key are applied to flow logs automatically; you do not need to set a `match` on each entry. **Example: filter out a specific namespace** -This example filters out all flow logs whose source or destination namespace is "dev". Additional namespaces could be filtered by adjusting the regular expression "pattern"s, or by adding additional `exclude` blocks. +This example filters out all flow logs whose source or destination namespace is `dev`. A record is dropped when it matches any of the `exclude` rules; additional namespaces could be filtered by adjusting the regular expressions, or by adding more `exclude` rules. -``` - - @type grep - - key source_namespace - pattern dev - - - key dest_namespace - pattern dev - - +```yaml +- name: grep + exclude: + - source_namespace dev + - dest_namespace dev ``` **Example: filter out internet traffic to a specific deployment** -This example filters inbound internet traffic to the deployment with pods named, `nginx-internet-*`. Note the use of the `and` directive to filter out traffic that is both to the deployment, and from the internet (source `pub`). +This example filters inbound internet traffic to the deployment with pods named, `nginx-internet-*`. Note the use of `logical_op: and` to filter out only the traffic that is both to the deployment, and from the internet (source `pub`). -``` - - @type grep - - - key dest_name_aggr - pattern ^nginx-internet - - - key source_name_aggr - pattern pub - - - +```yaml +- name: grep + logical_op: and + exclude: + - dest_name_aggr ^nginx-internet + - source_name_aggr pub ``` ### Add filters to ConfigMap file -1. Create a `filters` directory with a file called `flow` with your desired filters. If you are also adding [dns filters](../dns/filtering-dns.mdx), add the `dns` file to the directory. +1. Create a `filters` directory with a file called `flow` with your desired filters. If you are also adding [DNS filters](../dns/filtering-dns.mdx), add the `dns` file to the directory. -1. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace with the following command. +1. Create the `fluent-bit-filters` ConfigMap in the `tigera-operator` namespace with the following command. ```bash - kubectl create configmap fluentd-filters -n tigera-operator --from-file=filters + kubectl create configmap fluent-bit-filters -n tigera-operator --from-file=filters ``` +The operator inserts the filters inline into the log collector configuration and rolls the `calico-fluent-bit` DaemonSet automatically. + +:::note Upgrading from a release that used Fluentd + +Earlier releases collected logs with Fluentd and read filters in Fluentd `` syntax from a ConfigMap named `fluentd-filters`. That ConfigMap is no longer read, and Fluentd filter syntax cannot be translated automatically. Recreate your filters as Fluent Bit YAML filter lists under the new `fluent-bit-filters` name. If a filter key does not parse as Fluent Bit YAML, the operator skips that filter, reports a warning on the `tigera status` output naming the offending key, and continues to ship unfiltered logs. + +::: + ## Additional resources - [Flow log aggregation](aggregation.mdx) diff --git a/calico-enterprise/observability/elastic/overview.mdx b/calico-enterprise/observability/elastic/overview.mdx index 2959a397b3..43b84a1463 100644 --- a/calico-enterprise/observability/elastic/overview.mdx +++ b/calico-enterprise/observability/elastic/overview.mdx @@ -29,23 +29,23 @@ Elasticsearch logs provide the visibility and troubleshooting backend for $[prod | Log type | Description | Log source | RBAC | Index | | -------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------ | ------------------------------ | -| flow | Network flows for workloads: source and destination namespaces, pods, labels, and policies | $[prodname] cnx-node (Felix) | `flows` | `tigera_secure_ee_flows.*` | -| audit | Audit logs for $[prodname] resources | $[prodname] apiserver | `audit_ee` | `tigera_secure_ee_audit_ee.*` | -| | Audit logs for Kubernetes resources | Kubernetes apiserver | `audit_kube` | `tigera_secure_ee_audit_kube.*`| +| `flow` | Network flows for workloads: source and destination namespaces, pods, labels, and policies | $[prodname] cnx-node (Felix) | `flows` | `tigera_secure_ee_flows.*` | +| `audit` | Audit logs for $[prodname] resources | $[prodname] API server | `audit_ee` | `tigera_secure_ee_audit_ee.*` | +| | Audit logs for Kubernetes resources | Kubernetes API server | `audit_kube` | `tigera_secure_ee_audit_kube.*`| | | | Both audit logs above | `audit*` | `tigera_secure_ee_audit*` | -| bgp | $[prodname] networking BGP peering and route propagation | $[prodname] cnx-node (BIRD) | `ee_bgp` | `tigera_secure_ee_bgp.*` | -| dns | DNS lookups and responses from $[prodname] domain-based policy | $[prodname] cnx-node (Felix) | `ee_dns` | `tigera_secure_ee_dns.*` | -| ids | $[prodname] intrusion detection events: suspicious IPs, suspicious domains, and global alerts | $[prodname] intrusion-detection-controller | `ee_events` | `tigera_secure_ee_events.*` | +| `bgp` | $[prodname] networking BGP peering and route propagation | $[prodname] cnx-node (BIRD) | `ee_bgp` | `tigera_secure_ee_bgp.*` | +| `dns` | DNS lookups and responses from $[prodname] domain-based policy | $[prodname] cnx-node (Felix) | `ee_dns` | `tigera_secure_ee_dns.*` | +| `ids` | $[prodname] intrusion detection events: suspicious IPs, suspicious domains, and global alerts | $[prodname] intrusion-detection-controller | `ee_events` | `tigera_secure_ee_events.*` | :::note -Because of their high-volume, flow and dns logs support aggregation. +Because of their high-volume, flow and DNS logs support aggregation. ::: ### Default log configuration and security -$[prodname] automatically installs fluentd on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk. +$[prodname] automatically installs Fluent Bit on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk. $[prodname] enables user authentication in Elasticsearch, and secures access to Elasticsearch and Kibana instances using network policy. diff --git a/calico-enterprise/observability/kube-audit.mdx b/calico-enterprise/observability/kube-audit.mdx index d960f5d801..82648e2772 100644 --- a/calico-enterprise/observability/kube-audit.mdx +++ b/calico-enterprise/observability/kube-audit.mdx @@ -84,7 +84,7 @@ rules: The following updates require a restart to the Kubernetes API Server. -To enable Kubernetes resource audit logs to be read by $[prodname] in fluentd, follow these steps. +To enable Kubernetes resource audit logs to be read by the $[prodname] Fluent Bit log collector, follow these steps. On the Kubernetes API Server, update these flags. diff --git a/calico-enterprise/observability/packetcapture.mdx b/calico-enterprise/observability/packetcapture.mdx index f350593b36..dcdc4bc2bd 100644 --- a/calico-enterprise/observability/packetcapture.mdx +++ b/calico-enterprise/observability/packetcapture.mdx @@ -12,7 +12,7 @@ Capture live traffic inside a Kubernetes cluster, and export to visualization to $[prodname] packet capture is implemented in a Kubernetes-native way so you can troubleshoot service/application connectivity issues and performance issues. You can start a packet capture in the web console Service Graph, or using the CLI. -Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, daemonset, statefulset, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. +Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, DaemonSet, StatefulSet, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like Wireshark. With $[prodname] packet capture you can: @@ -79,7 +79,7 @@ EOF ### Packet capture in Service Graph -1. Select an endpoint from the service graph (for example, namespace, service, replica set, daemonset, statefulset, or pod), right-click, and select **Initiate packet capture**. +1. Select an endpoint from the service graph (for example, namespace, service, replica set, DaemonSet, StatefulSet, or pod), right-click, and select **Initiate packet capture**. ![start-capture](/img/calico-enterprise/start-capture.png) @@ -226,19 +226,19 @@ status: Get the pod on the node with the packet capture that you want. ```bash -kubectl get pods -n tigera-fluentd --no-headers --field-selector spec.nodeName="" +kubectl get pods -n calico-system -l k8s-app=calico-node --no-headers --field-selector spec.nodeName="" ``` Copy the packet capture using the pod information. ```bash -kubectl cp tigera-fluentd/:var/log/calico/pcap/sample/sample-capture/ . +kubectl cp calico-system/:var/log/calico/pcap/sample/sample-capture/ . ``` **Delete packet capture files** ```bash -kubectl exec -it tigera-fluentd/ -- sh -c "rm -r /var/log/calico/pcap/sample/sample-capture/" +kubectl exec -it -n calico-system -- sh -c "rm -r /var/log/calico/pcap/sample/sample-capture/" ``` ### Store and rotate capture files diff --git a/calico-enterprise/observability/review-unused-network-policies.mdx b/calico-enterprise/observability/review-unused-network-policies.mdx index 69c22e2483..2836a95c80 100644 --- a/calico-enterprise/observability/review-unused-network-policies.mdx +++ b/calico-enterprise/observability/review-unused-network-policies.mdx @@ -81,7 +81,7 @@ StagedNetworkPolicy test-policies default.snp-v2 Ye Unused Rules since 2026-01-14T00:00:00Z (2) KIND NAMESPACE NAME DIRECTION INDEX -NetworkPolicy tigera-fluentd calico-system.allow-fluentd-node Egress 0 +NetworkPolicy calico-system calico-system.allow-calico-fluent-bit Egress 0 ... ``` @@ -131,8 +131,8 @@ $ calicoctl review unused-policies -o json "rules": [ { "policyKind": "NetworkPolicy", - "policyNamespace": "tigera-fluentd", - "policyName": "calico-system.allow-fluentd-node", + "policyNamespace": "calico-system", + "policyName": "calico-system.allow-calico-fluent-bit", "policyGeneration": 1, "creationTimestamp": "2026-04-21T16:26:00Z", "lastUpdate": "2026-04-21T16:26:00Z", diff --git a/calico-enterprise/operations/comms/index.mdx b/calico-enterprise/operations/comms/index.mdx index 5d21480708..c331ec751c 100644 --- a/calico-enterprise/operations/comms/index.mdx +++ b/calico-enterprise/operations/comms/index.mdx @@ -47,27 +47,27 @@ The **Deployed to** column shows the namespace where the operator places the sec | Secret name | DNS name(s) | Deployed to | Owner resource | |---|---|---|---| -| `calico-apiserver-certs` | `calico-api` | `calico-system` | APIServer/tigera-secure | -| `calico-kube-controllers-metrics-tls` | `calico-kube-controllers-metrics` | `calico-system` | Installation/default | -| `calico-node-prometheus-client-tls` | `calico-node-prometheus-client-tls` | `tigera-prometheus` | Monitor/tigera-secure | -| `calico-node-prometheus-server-tls` | `calico-node-metrics` | `calico-system` | Installation/default | -| `calico-node-prometheus-tls` | `prometheus-http-api` | `tigera-prometheus` | Monitor/tigera-secure | -| `deep-packet-inspection-tls` | `intrusion-detection-tls` | `tigera-dpi` | IntrusionDetection/tigera-secure | -| `internal-manager-tls` | `calico-manager` | `calico-system` | Manager/tigera-secure | -| `intrusion-detection-tls` | `intrusion-detection-tls` | `tigera-intrusion-detection` | IntrusionDetection/tigera-secure | -| `manager-tls` | `calico-manager` | `calico-system` | Manager/tigera-secure | -| `node-certs` | `typha-client` | `calico-system` | Installation/default | -| `node-certs` | `typha-client` | `tigera-dpi` | IntrusionDetection/tigera-secure | -| `policy-recommendation-tls` | `policy-recommendation-tls` | `calico-system` | PolicyRecommendation/tigera-secure | -| `tigera-ee-elasticsearch-metrics-tls` | `tigera-elasticsearch-metrics` | `tigera-elasticsearch` | LogStorage/tigera-secure | -| `tigera-fluentd-prometheus-tls` | `fluentd-http-input` | `tigera-fluentd` | LogCollector/tigera-secure | -| `tigera-operator-tls` | `tigera-operator-metrics` | `tigera-prometheus` | Monitor/tigera-secure | -| `tigera-secure-elasticsearch-cert` | `tigera-secure-es-gateway-http` | `tigera-elasticsearch` | LogStorage/tigera-secure | -| `tigera-secure-internal-elasticsearch-cert` | `tigera-secure-es-http` | `tigera-elasticsearch` | LogStorage/tigera-secure | -| `tigera-secure-kibana-cert` | `tigera-secure-kb-http` | `tigera-kibana` | LogStorage/tigera-secure | -| `tigera-secure-linseed-cert` | `tigera-linseed` | `tigera-elasticsearch` | LogStorage/tigera-secure | -| `typha-certs` | `typha-server` | `calico-system` | Installation/default | -| `typha-certs-noncluster-host` | `typha-server-noncluster-host` | `calico-system` | Installation/default | +| `calico-apiserver-certs` | `calico-api` | `calico-system` | `APIServer/tigera-secure` | +| `calico-kube-controllers-metrics-tls` | `calico-kube-controllers-metrics` | `calico-system` | `Installation/default` | +| `calico-node-prometheus-client-tls` | `calico-node-prometheus-client-tls` | `tigera-prometheus` | `Monitor/tigera-secure` | +| `calico-node-prometheus-server-tls` | `calico-node-metrics` | `calico-system` | `Installation/default` | +| `calico-node-prometheus-tls` | `prometheus-http-api` | `tigera-prometheus` | `Monitor/tigera-secure` | +| `deep-packet-inspection-tls` | `intrusion-detection-tls` | `tigera-dpi` | `IntrusionDetection/tigera-secure` | +| `internal-manager-tls` | `calico-manager` | `calico-system` | `Manager/tigera-secure` | +| `intrusion-detection-tls` | `intrusion-detection-tls` | `tigera-intrusion-detection` | `IntrusionDetection/tigera-secure` | +| `manager-tls` | `calico-manager` | `calico-system` | `Manager/tigera-secure` | +| `node-certs` | `typha-client` | `calico-system` | `Installation/default` | +| `node-certs` | `typha-client` | `tigera-dpi` | `IntrusionDetection/tigera-secure` | +| `policy-recommendation-tls` | `policy-recommendation-tls` | `calico-system` | `PolicyRecommendation/tigera-secure` | +| `tigera-ee-elasticsearch-metrics-tls` | `tigera-elasticsearch-metrics` | `tigera-elasticsearch` | `LogStorage/tigera-secure` | +| `calico-fluent-bit-tls` | `calico-fluent-bit-http-input` | `calico-system` | `LogCollector/tigera-secure` | +| `tigera-operator-tls` | `tigera-operator-metrics` | `tigera-prometheus` | `Monitor/tigera-secure` | +| `tigera-secure-elasticsearch-cert` | `tigera-secure-es-gateway-http` | `tigera-elasticsearch` | `LogStorage/tigera-secure` | +| `tigera-secure-internal-elasticsearch-cert` | `tigera-secure-es-http` | `tigera-elasticsearch` | `LogStorage/tigera-secure` | +| `tigera-secure-kibana-cert` | `tigera-secure-kb-http` | `tigera-kibana` | `LogStorage/tigera-secure` | +| `tigera-secure-linseed-cert` | `tigera-linseed` | `tigera-elasticsearch` | `LogStorage/tigera-secure` | +| `typha-certs` | `typha-server` | `calico-system` | `Installation/default` | +| `typha-certs-noncluster-host` | `typha-server-noncluster-host` | `calico-system` | `Installation/default` | :::tip diff --git a/calico-enterprise/operations/license-options.mdx b/calico-enterprise/operations/license-options.mdx index 45fd4155ef..bf22f53fa1 100644 --- a/calico-enterprise/operations/license-options.mdx +++ b/calico-enterprise/operations/license-options.mdx @@ -31,7 +31,7 @@ Data plane traffic continues to flow without interruption — policy enforcement | Area | Behavior after expiration | |---|---| | $[prodname] resources | Read-only. The Tigera API server rejects create and update requests; get, list, and delete on existing resources still succeed. Calico Open Source resources are not affected. | -| Flow, DNS, and audit logs | Stopped. Fluentd is scaled down and logs are no longer forwarded to Elasticsearch. | +| Flow, DNS, and audit logs | Stopped. The Fluent Bit log collector is scaled down and logs are no longer forwarded. | | Prometheus metrics for $[prodname] components | Stopped. The Operator removes the $[prodname] ServiceMonitor so metrics are no longer scraped. | | Web console | Service Graph, Kibana, Alerts, and Compliance remain visible in read-only mode; attempts to create or modify resources are not saved. | diff --git a/calico-enterprise/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx b/calico-enterprise/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx index b15bab97eb..5e1cf80ff9 100644 --- a/calico-enterprise/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx +++ b/calico-enterprise/operations/monitor/metrics/elasticsearch-and-fluentd-metrics.mdx @@ -1,12 +1,12 @@ --- -description: Track Calico Enterprise Elasticsearch and Fluentd metrics in Prometheus to alert on flow, DNS, audit, and compliance log collection or storage disruptions. +description: Track Calico Enterprise Elasticsearch and Fluent Bit metrics in Prometheus to alert on flow, DNS, audit, and compliance log collection or storage disruptions. --- -# Elasticsearch and Fluentd metrics +# Elasticsearch and Fluent Bit metrics ## Big picture -Use the Prometheus monitoring and alerting tool for Fluentd and Elasticsearch metrics to ensure continuous network visibility. +Use the Prometheus monitoring and alerting tool for Fluent Bit and Elasticsearch metrics to ensure continuous network visibility. ## Value @@ -16,16 +16,16 @@ Platform engineering teams rely on logs, such as flow logs and DNS logs, for vis | Component | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Prometheus | Monitoring tool that scrapes metrics from instrumented jobs and displays time series data in a visualizer (such as Grafana). For $[prodname], the “jobs” that Prometheus can harvest metrics from are the Elasticsearch and Fluentd components. | +| Prometheus | Monitoring tool that scrapes metrics from instrumented jobs and displays time series data in a visualizer (such as Grafana). For $[prodname], the “jobs” that Prometheus can harvest metrics from are the Elasticsearch and Fluent Bit components. | | Elasticsearch | Stores $[prodname] logs. | -| Fluentd | Sends $[prodname] logs to Elasticsearch for storage. | +| Fluent Bit | Ships $[prodname] logs to the log storage backend. | -**Multi-cluster management users**: Elasticsearch metrics are collected only from the management cluster. Because managed clusters do not have Elasticsearch clusters, do not monitor Elasticsearch for managed clusters. However, managed clusters do feed Fluentd logs to Elasticsearch, so you should monitor fluentd for managed clusters. +**Multi-cluster management users**: Elasticsearch metrics are collected only from the management cluster. Because managed clusters do not have Elasticsearch clusters, do not monitor Elasticsearch for managed clusters. However, managed clusters do ship logs to the management cluster, so you should monitor Fluent Bit for managed clusters. ## How to - [Create Prometheus alerts for Elasticsearch](#create-prometheus-alerts-for-elasticsearch) -- [Create Prometheus alerts for Fluentd](#create-prometheus-alerts-for-elasticsearch) +- [Create Prometheus alerts for Fluent Bit](#create-prometheus-alerts-for-fluent-bit) ### Create Prometheus alerts for Elasticsearch @@ -123,9 +123,9 @@ LogStorage resource limits." | ElasticsearchPodConsistentlyHighCPUUsage | Non-critical, warning | | An Elasticsearch pod is averaging above 90% of its CPU over the last 10 minutes. | | ElasticsearchPodConsistentlyHighMemoryUsage | Non-critical, warning | | An Elasticsearch pod is averaging above the set memory threshold over the last 10 minutes. | -### Create Prometheus alerts for Fluentd +### Create Prometheus alerts for Fluent Bit -The following example creates a Prometheus a rule to monitor some important Fluentd metrics, and alert when they +The following example creates a Prometheus rule to monitor some important Fluent Bit metrics, and alert when they have crossed certain thresholds: ```yaml noValidation @@ -141,18 +141,28 @@ spec: groups: - name: tigera-log-collection.rules rules: - - alert: FluentdPodConsistentlyLowBufferSpace - expr: avg_over_time(fluentd_output_status_buffer_available_space_ratio[5m]) < 75 + - alert: FluentBitPodConsistentlyLowBufferSpace + expr: avg_over_time(fluentbit_output_chunk_available_capacity_percent[5m]) < 75 labels: severity: Warning annotations: - summary: "Fluentd pod {{$labels.pod}}'s buffer space is consistently below 75 percent capacity." - description: "Fluentd pod {{$labels.pod}} has very low buffer space. There may be connection issues between Elasticsearch -and Fluentd or there are too many logs to write out, check the logs for the Fluentd pod." + summary: "Fluent Bit pod {{$labels.pod}}'s output buffer space is consistently below 75 percent capacity." + description: "Fluent Bit pod {{$labels.pod}} has very low buffer space for output {{$labels.name}}. There may be +connection issues between Fluent Bit and the log destination, or there are too many logs to write out; check the logs +for the Fluent Bit pod." + - alert: FluentBitPodDroppingLogs + expr: rate(fluentbit_output_retries_failed_total[5m]) > 0 + labels: + severity: Warning + annotations: + summary: "Fluent Bit pod {{$labels.pod}} is dropping log chunks." + description: "Fluent Bit pod {{$labels.pod}} gave up retrying to deliver log chunks for output {{$labels.name}}. +Logs are being lost; check the logs for the Fluent Bit pod and the health of the log destination." ``` #### The alerts created in the example are described as follows: -| Alert | Severity | Requires | Issue/reason | -| ---------------------------------------- | --------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **FluentdPodConsistentlyLowBufferSpace** | Non-critical, warning | Immediate investigation to ensure logs are being gathered correctly. | A Fluentd pod’s available buffer size has averaged less than 75% over the last 5 minutes.

This could mean Fluentd is having trouble communicating with the Elasticsearch cluster, the Elasticsearch cluster is down, or there are simply too many logs to process. | +| Alert | Severity | Requires | Issue/reason | +| ------------------------------------------ | --------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **FluentBitPodConsistentlyLowBufferSpace** | Non-critical, warning | Immediate investigation to ensure logs are being gathered correctly. | A Fluent Bit pod’s available output buffer capacity has averaged less than 75% over the last 5 minutes.

This could mean Fluent Bit is having trouble communicating with a log destination, the destination is down, or there are simply too many logs to process. | +| **FluentBitPodDroppingLogs** | Non-critical, warning | Immediate investigation to ensure logs are not being lost. | A Fluent Bit pod exhausted the retries for one or more log chunks and dropped them. | diff --git a/calico-enterprise/operations/monitor/prometheus/byo-prometheus.mdx b/calico-enterprise/operations/monitor/prometheus/byo-prometheus.mdx index 187e819bbd..6fd2f93b5b 100644 --- a/calico-enterprise/operations/monitor/prometheus/byo-prometheus.mdx +++ b/calico-enterprise/operations/monitor/prometheus/byo-prometheus.mdx @@ -15,13 +15,13 @@ Scrape $[prodname] metrics for Bring Your Own (BYO) Prometheus. $[prodname] uses the Prometheus monitoring tool to scrape metrics from instrumented jobs, and displays time-series data in a visualizer such as Grafana. You can scrape the following time-series metrics for $[prodname] components to your own Prometheus: -- elasticsearch -- fluentd -- calico-node -- kube-controllers -- felix -- typha (not enabled by default) -- operator +- `elasticsearch` +- `fluent-bit` +- `calico-node` +- `kube-controllers` +- `felix` +- `typha` (not enabled by default) +- `operator` ## Before you begin @@ -42,7 +42,7 @@ For the supported version of Prometheus in this release, see the [Release Notes] In this section we create a service monitor that scrapes all enabled metrics. To enable metrics that are not enabled by default, please consult the [next section](#scrape-metrics-from-specific-components). -The following example shows a Prometheus server installed in namespace "external-prometheus" with a `serviceMonitorSelector` that selects all service monitors with the label `k8s-app=tigera-external-prometheus`. +The following example shows a Prometheus server installed in namespace `external-prometheus` with a `serviceMonitorSelector` that selects all service monitors with the label `k8s-app=tigera-external-prometheus`. 1. Save the following configuration in a file called `monitor.yaml`. @@ -124,34 +124,13 @@ kubectl apply -f $[filesUrl]/manifests/prometheus/elasticsearch-metrics-service- The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. - - -**Configure TLS certificates** - -1. Copy the required secret and configmap to your namespace. -2. Save the manifest of the required TLS secret and CA configmap. - - ```bash - kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml - ``` - - ```bash - kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml - ``` - -3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` and change the namespace to the namespace where your prometheus instance is running. -4. Apply the manifests to your cluster. - - ```bash - kubectl apply -f calico-node-prometheus-client-tls.yaml - ``` - - ```bash - kubectl apply -f tigera-ca-bundle.yaml - ``` + **Create the service monitor** +Fluent Bit serves its metrics over plain HTTP (port 2020, path `/api/v2/metrics/prometheus`), so no TLS +configuration is required. + Apply the ServiceMonitor to the namespace where Prometheus is running. ```bash @@ -159,11 +138,20 @@ export NAMESPACE= ``` ```bash -kubectl apply -f $[filesUrl]/manifests/prometheus/fluentd-metrics-service-monitor.yaml -n $NAMESPACE +kubectl apply -f $[filesUrl]/manifests/prometheus/fluent-bit-metrics-service-monitor.yaml -n $NAMESPACE ``` The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. +:::note + +Access to the Fluent Bit metrics port is restricted by the `calico-system.allow-calico-fluent-bit` network policy, which +by default admits only the `tigera-prometheus` namespace. Create a policy that allows your BYO Prometheus +namespace to reach port 2020 on the `calico-fluent-bit` pods (see +[Create policy to secure traffic between pods](#create-policy-to-secure-traffic-between-pods)). + +::: + @@ -403,7 +391,7 @@ The .yamls have no namespace defined so when you apply `kubectl`, it is applied kubectl port-forward pod/byo-prometheus-pod 9090:9090 -n $NAMESPACE ``` -1. Browse to the Prometheus dashboard: http://localhost:9090. +1. Browse to the Prometheus dashboard: `http://localhost:9090`. 1. In the Expression text box, enter your metric name and click the **Execute** button. diff --git a/calico-enterprise/reference/architecture/overview.mdx b/calico-enterprise/reference/architecture/overview.mdx index 179ca2060a..a0864b717a 100644 --- a/calico-enterprise/reference/architecture/overview.mdx +++ b/calico-enterprise/reference/architecture/overview.mdx @@ -34,7 +34,7 @@ Calico open-source components are the foundation of $[prodname]. $[prodname] pro ## Bundled third-party components -- [fluentd](#fluentd) +- [Fluent Bit](#fluent-bit) - [Elasticsearch and Kibana](#elasticsearch-and-kibana) - [Prometheus](#prometheus) @@ -100,7 +100,7 @@ Provides the API for listing, downloading, and rendering reports, and RBAC by pe **compliance-benchmarker** -A daemonset that runs checks in the CIS Kubernetes Benchmark on each node so you can see if Kubernetes is securely deployed. +A DaemonSet that runs checks in the CIS Kubernetes Benchmark on each node so you can see if Kubernetes is securely deployed. ### Linseed API and ES gateway @@ -138,9 +138,9 @@ The Linseed API uses mTLS to connect to clients, and provides an API to access E **Main task**: Built-in third-party search-engine and visualization dashboard, which provide logs for visibility into workloads, to troubleshoot Kubernetes clusters. Installed and configured by default. [Elasticsearch](../../observability/index.mdx). -### fluentd +### Fluent Bit -**Main task**: Collects and forwards $[prodname] logs (flows, DNS, L7) to Elasticsearch. Open source data collector for unified logging. [fluentd open source](https://www.fluentd.org/). +**Main task**: Collects and forwards $[prodname] logs (flows, DNS, L7) to the log storage backend. Open source, lightweight log processor and forwarder. [Fluent Bit open source](https://fluentbit.io/). ### Prometheus @@ -182,7 +182,7 @@ The BGP client is responsible for: - BIRD - confd -The calico repository contains the Dockerfile for calico-node, along with various configuration files to configure and “glue” these components together. In addition, we use runit for logging (svlogd) and init (runsv) services. [calico-node](../component-resources/node/configuration.mdx). +The calico repository contains the `Dockerfile` for calico-node, along with various configuration files to configure and “glue” these components together. In addition, we use `runit` for logging (`svlogd`) and init (`runsv`) services. [calico-node](../component-resources/node/configuration.mdx). ### CNI plugin @@ -197,7 +197,7 @@ The Calico CNI plugin allows you to use Calico networking for any orchestrator t {/* vale Vale.Repetition = YES */} An open source, lightweight configuration management tool. -Confd dynamically generates BIRD configuration files based on the updates to data in the datastore. When the configuration file changes, confd triggers BIRD to load the new files. [Configure confd](../component-resources/node/configuration.mdx#content-main), and [confd project](https://github.com/kelseyhightower/confd). +`confd` dynamically generates BIRD configuration files based on the updates to data in the datastore. When the configuration file changes, confd triggers BIRD to load the new files. [Configure confd](../component-resources/node/configuration.mdx#content-main), and [confd project](https://github.com/kelseyhightower/confd). ### Datastore plugin diff --git a/calico-enterprise/reference/clis/calicoctl/cluster/diags.mdx b/calico-enterprise/reference/clis/calicoctl/cluster/diags.mdx index af93e03ba4..72b40ef395 100644 --- a/calico-enterprise/reference/clis/calicoctl/cluster/diags.mdx +++ b/calico-enterprise/reference/clis/calicoctl/cluster/diags.mdx @@ -284,6 +284,18 @@ Collect CNI log cni.log for the node calico-node-4vp2g Collect CNI log config for the node calico-node-4vp2g Collect CNI log current for the node calico-node-4vp2g Collect CNI log lock for the node calico-node-4vp2g +Collecting detailed diags for pod calico-fluent-bit-dncwx in namespace calico-system on node ip-172-16-101-171.us-west-2.compute.internal... +Collecting diags for pod: calico-fluent-bit-dncwx +Collect logs for pod calico-fluent-bit-dncwx +Collect describe for pod calico-fluent-bit-dncwx +Collecting detailed diags for pod calico-fluent-bit-cgtk9 in namespace calico-system on node ip-172-16-101-210.us-west-2.compute.internal... +Collecting diags for pod: calico-fluent-bit-cgtk9 +Collect logs for pod calico-fluent-bit-cgtk9 +Collect describe for pod calico-fluent-bit-cgtk9 +Collecting detailed diags for pod calico-fluent-bit-lht88 in namespace calico-system on node ip-172-16-101-83.us-west-2.compute.internal... +Collecting diags for pod: calico-fluent-bit-lht88 +Collect logs for pod calico-fluent-bit-lht88 +Collect describe for pod calico-fluent-bit-lht88 Collecting detailed diags for pod runtime-reporter-csztr in namespace calico-system on node ip-172-16-101-171.us-west-2.compute.internal... Collecting diags for pod: runtime-reporter-csztr Collect logs for pod runtime-reporter-csztr @@ -319,19 +331,6 @@ Collect describe for pod calico-apiserver-6f5ddf5697-x69r7 Collecting detailed diags for namespace tigera-access... Collecting detailed diags for namespace tigera-dex... Collecting detailed diags for namespace tigera-elasticsearch... -Collecting detailed diags for namespace tigera-fluentd... -Collecting detailed diags for pod fluentd-node-dncwx in namespace tigera-fluentd on node ip-172-16-101-171.us-west-2.compute.internal... -Collecting diags for pod: fluentd-node-dncwx -Collect logs for pod fluentd-node-dncwx -Collect describe for pod fluentd-node-dncwx -Collecting detailed diags for pod fluentd-node-cgtk9 in namespace tigera-fluentd on node ip-172-16-101-210.us-west-2.compute.internal... -Collecting diags for pod: fluentd-node-cgtk9 -Collect logs for pod fluentd-node-cgtk9 -Collect describe for pod fluentd-node-cgtk9 -Collecting detailed diags for pod fluentd-node-lht88 in namespace tigera-fluentd on node ip-172-16-101-83.us-west-2.compute.internal... -Collecting diags for pod: fluentd-node-lht88 -Collect logs for pod fluentd-node-lht88 -Collect describe for pod fluentd-node-lht88 Collecting detailed diags for namespace tigera-guardian... Collecting detailed diags for pod tigera-guardian-b97bf7d57-blqsn in namespace tigera-guardian on node ip-172-16-101-171.us-west-2.compute.internal... Collecting diags for pod: tigera-guardian-b97bf7d57-blqsn diff --git a/calico-enterprise/reference/component-resources/configure-resources.mdx b/calico-enterprise/reference/component-resources/configure-resources.mdx index 7281339d8f..09461f6e8b 100644 --- a/calico-enterprise/reference/component-resources/configure-resources.mdx +++ b/calico-enterprise/reference/component-resources/configure-resources.mdx @@ -583,14 +583,14 @@ This command will output the configured resource requests and limits for the Int ## LogCollector custom resource -The [LogCollector](../../reference/installation/api.mdx#logcollector) CR provides a way to configure resources for FluentdDaemonSet, EKSLogForwarderDeployment. +The [LogCollector](../../reference/installation/api.mdx#logcollector) CR provides a way to configure resources for CalicoFluentBitDaemonSet, EKSLogForwarderDeployment. -### FluentdDaemonSet. +### CalicoFluentBitDaemonSet. -To configure resource specification for the [FluentdDaemonSet](../../reference/installation/api.mdx#fluentddaemonset), patch the LogCollector CR using the below command: +To configure resource specification for the [CalicoFluentBitDaemonSet](../../reference/installation/api.mdx#logcollector), patch the LogCollector CR using the below command: ```bash -kubectl patch logcollector tigera-secure --type=merge --patch='{"spec": {"fluentdDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"fluentd","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' +kubectl patch logcollector tigera-secure --type=merge --patch='{"spec": {"calicoFluentBitDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"calico-fluent-bit","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). @@ -599,14 +599,14 @@ This command sets the CPU request to 100 milliCPU (mCPU) and the memory request You can verify the configured resources using the following command: ```bash -kubectl get daemonset.apps/fluentd-node -n tigera-fluentd -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' +kubectl get daemonset.apps/calico-fluent-bit -n calico-system -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` -This command will output the configured resource requests and limits for the FluentdDaemonSet in JSON format. +This command will output the configured resource requests and limits for the CalicoFluentBitDaemonSet in JSON format. ```bash { - "name": "fluentd", + "name": "calico-fluent-bit", "resources": { "limits": { "cpu": "1", @@ -634,7 +634,7 @@ This command sets the CPU request to 100 milliCPU (mCPU) and the memory request You can verify the configured resources using the following command: ```bash -kubectl get deployment.apps/eks-log-forwarder -n tigera-fluentd -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' +kubectl get deployment.apps/eks-log-forwarder -n calico-system -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the EKSLogForwarderDeployment in JSON format. @@ -910,7 +910,7 @@ The [Monitor](../../reference/installation/api.mdx#monitor) CR provides a way to ### Prometheus -To configure resource specification for the [Prometheus](../../reference/installation/api.mdx#prometheus), Resources for the default container "prometheus" can be configured using the "resources" field under "commonPrometheusFields". For all other injected containers, such as "authn-proxy", resource configuration can be set using the "containers" struct, as shown below in the patch command below. +To configure resource specification for the [Prometheus](../../reference/installation/api.mdx#prometheus), Resources for the default container `prometheus` can be configured using the `resources` field under "commonPrometheusFields". For all other injected containers, such as "authn-proxy", resource configuration can be set using the "containers" struct, as shown below in the patch command below. ```bash kubectl patch monitor tigera-secure --type=merge --patch='{"spec": {"prometheus": {"spec":{ "commonPrometheusFields": {"resources": {"limits": {"cpu":"500m","memory":"500Mi"}, "requests": {"cpu":"50m", "memory":"50Mi"}}, "containers":[{"name":"authn-proxy","resources":{"limits": {"cpu":"250m","memory":"500Mi"},"requests": {"cpu":"25m","memory":"50Mi"}}}]}}}}}' @@ -975,7 +975,7 @@ The "config-reloader" container has default resource values set based by the Pro ### Alertmanager -To configure resource specification for the [Alertmanager](../../reference/installation/api.mdx#alertmanager), you can set resources for the default container "prometheus" using the "resources" field under "commonPrometheusFields". For all other injected containers, like "authn-proxy", resource configuration can be set using the "containers" struct, as shown below in the patch command below. +To configure resource specification for the [Alertmanager](../../reference/installation/api.mdx#alertmanager), you can set resources for the default container `prometheus` using the `resources` field under "commonPrometheusFields". For all other injected containers, like "authn-proxy", resource configuration can be set using the "containers" struct, as shown below in the patch command below. ```bash kubectl patch monitor tigera-secure --type=merge --patch='{"spec": {"alertManager": {"spec": {"resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}}}}' @@ -1105,20 +1105,20 @@ This command will output the configured resource requests and limits for the Pol ## Update via Helm -To update configurations during installation via the [Helm chart](../../getting-started/install-on-clusters/kubernetes/helm#install-calico-enterprise), modify the values.yaml with the necessary resource values for the components prior to executing the Helm install. +To update configurations during installation via the [Helm chart](../../getting-started/install-on-clusters/kubernetes/helm#install-calico-enterprise), modify the `values.yaml` with the necessary resource values for the components prior to executing the Helm install. :::note -The provided example illustrates configuring the apiserver component. Follow a similar approach for other components to update resource requests and limits during installation using the Helm chart. +The provided example illustrates configuring the `apiserver` component. Follow a similar approach for other components to update resource requests and limits during installation using the Helm chart. ::: ### APIServer custom resource -The [APIServer](../../reference/installation/api.mdx#apiserver) CR provides a way to configure APIServerDeployment. The following sections provide example values.yaml for apiserver component. +The [APIServer](../../reference/installation/api.mdx#apiserver) CR provides a way to configure APIServerDeployment. The following sections provide example `values.yaml` for the `apiserver` component. #### APIServerDeployment -To configure resource specification for the [APIServerDeployment](../../reference/installation/api.mdx#apiserverdeployment), update values.yaml with the appropriate resource values. +To configure resource specification for the [APIServerDeployment](../../reference/installation/api.mdx#apiserverdeployment), update `values.yaml` with the appropriate resource values. ```bash apiServer: