Skip to content

Update log collector docs for the fluentd to fluent-bit migration (EV-6164) #2815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
hjiawei wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Update log collector docs for the fluentd to fluent-bit migration (EV-6164) #2815

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion calico-cloud/_includes/components/ReqsSys.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ function NetworkRequirementsEnt(props) {
<td>
<strong>Logs and storage</strong>
</td>
<td>Elasticsearch with fluentd datastore</td>
<td>Elasticsearch log storage</td>
<td>TCP 9200 (default)</td>
</tr>
<tr className='odd'>
Expand Down
6 changes: 3 additions & 3 deletions calico-cloud/get-started/cc-arch-diagram.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ The following diagram shows the major components in a managed cluster, followed
| $[prodname] tunnel server | Communicates with managed clusters by creating secure TLS tunnels. | Port 9000 from managed clusters |
| calico-node | Bundles key components that are required for networking containers with $[prodname]:<br/><br />&bull; Felix<br />&bull; BIRD<br />&bull; confd | &bull; TCP 5473 to Typha<br />&bull; TCP 9900 and 9081 from Prometheus API service |
| Container threat detection | A threat detection engine that analyzes observed file and process activity to detect known malicious and suspicious activity. Monitors the following types of suspicious activity within containers:<br/><br />&bull; Access to sensitive system files and directories<br />&bull; Defense evasion<br />&bull; Discovery<br />&bull; Execution<br />&bull; Persistence<br />&bull; Privilege escalation<br/><br />Includes these components:<br /><br />**Runtime Security Operator**<br />An operator to manage and reconcile container threat defense components.<br/><br />**Runtime Reporter Pods**<br />Pods running on each node in the cluster to perform the detection activity outlined above.They send activity reports to Elasticsearch for analysis by $[prodname]. | TCP to Kubernetes API server |
| Compliance | Generates compliance reports for the Kubernetes cluster. Reports are based on archived flow and audit logs for Calico Cloud resources, plus any audit logs you’ve configured for Kubernetes resources in the Kubernetes API server. Compliance reports provide the following high-level information:<br /><br />&bull; Endpoints explicitly protected using ingress or egress policy<br />&bull; Policies and services<br /> - Policies and services associated with endpoints<br /> - Policy audit logs <br />&bull; Traffic<br /> - Allowed ingress/egress traffic to/from namespaces, and to/from the internet Compliance includes these components: <br /><br />**compliance-snapshotter** <br />Handles listing of required Kubernetes and $[prodname]configuration and pushes snapshots to Elasticsearch. Snapshots give you visibility into configuration changes, and how the cluster-wide configuration has evolved within a reporting interval.<br /><br />**compliance-reporter**<br />Handles report generation. Reads configuration history from Elasticsearch and determines time evolution of cluster-wide configuration, including relationships between policies, endpoints, services, and network sets. Data is then passed through a zero-trust aggregator to determine the “worst-case outliers” in the reporting interval.<br /><br />**compliance-controller**<br />Reads report configuration and manages creation, deletion, and monitoring of report generation jobs.<br /><br />**compliance-benchmarker**<br />A daemonset that runs checks in the CIS Kubernetes Benchmark on each node so you can see if Kubernetes is securely deployed.<br /> | &bull; TCP 8080 to Guardian<br />&bull; TCP 6443 to Kubernetes API server |
| Fluentd | Open-source data collector for unified logging. Collects and forwards $[prodname] logs (flows, DNS, L7) to log storage. | &bull; TCP 8080 to Guardian<br />&bull; TCP 9080 from Prometheus API service |
| Compliance | Generates compliance reports for the Kubernetes cluster. Reports are based on archived flow and audit logs for Calico Cloud resources, plus any audit logs you’ve configured for Kubernetes resources in the Kubernetes API server. Compliance reports provide the following high-level information:<br /><br />&bull; Endpoints explicitly protected using ingress or egress policy<br />&bull; Policies and services<br /> - Policies and services associated with endpoints<br /> - Policy audit logs <br />&bull; Traffic<br /> - Allowed ingress/egress traffic to/from namespaces, and to/from the internet Compliance includes these components: <br /><br />**compliance-snapshotter** <br />Handles listing of required Kubernetes and $[prodname] configuration and pushes snapshots to Elasticsearch. Snapshots give you visibility into configuration changes, and how the cluster-wide configuration has evolved within a reporting interval.<br /><br />**compliance-reporter**<br />Handles report generation. Reads configuration history from Elasticsearch and determines time evolution of cluster-wide configuration, including relationships between policies, endpoints, services, and network sets. Data is then passed through a zero-trust aggregator to determine the “worst-case outliers” in the reporting interval.<br /><br />**compliance-controller**<br />Reads report configuration and manages creation, deletion, and monitoring of report generation jobs.<br /><br />**compliance-benchmarker**<br />A DaemonSet that runs checks in the CIS Kubernetes Benchmark on each node so you can see if Kubernetes is securely deployed.<br /> | &bull; TCP 8080 to Guardian<br />&bull; TCP 6443 to Kubernetes API server |
| Fluent Bit | Open-source, lightweight log processor and forwarder. Collects and forwards $[prodname] logs (flows, DNS, L7) to log storage. | &bull; TCP 8080 to Guardian<br />&bull; TCP 2020 from Prometheus API service |
| Guardian | An agent running in each managed cluster that proxies communication between the $[prodname] tunnel server and your managed cluster. Secured using TLS tunnels.<br /> | &bull; Port 9000 to tunnel server<br />&bull; TCP 6443 to Kubernetes API server<br />&bull; TCP 6443 from $[prodname] components |
| Installation endpoints | Endpoints at `*.calicocloud.io` and `*.projectcalico.org`. | TCP 443 for both |
| Intrusion detection controller | Handles integrations with threat intelligence feeds and $[prodname] custom alerts. | &bull; TCP 8080 to Guardian<br />&bull; TCP 6443 to Kubernetes API server |
Expand All @@ -42,7 +42,7 @@ The following diagram shows the major components in a managed cluster, followed
| kube-controllers | Monitors the Kubernetes API and performs actions based on cluster state. $[prodname] kube-controllers container includes these controllers:<br/><br />&bull; Node<br />&bull; Service<br />&bull; Federated services<br />&bull; Authorization<br /> | &bull; TCP 9094 from Prometheus API service<br />&bull; TCP 6443 to Kubernetes API server |
| Log storage | Storage for logs (flows, L7, DNS, audit). Data for each managed cluster is isolated and protected against unauthorized access. | n/a |
| Packet capture API | Retrieves capture files (pcap format) generated by a packet capture for use with network protocol analysis tools like Wireshark. Packet capture data is visible in the web console and Service Graph. | &bull; TCP 8449 Guardian to Packet Capture API<br />&bull; TCP 6443 to Kubernetes API server |
| Prometheus API service | Collects metrics from $[prodname] components and makes the metrics available to the web console. | &bull; TCP 6443 to Kubernetes API server<br />&bull; TCP 9080 to Fluentd<br />&bull; TCP 9900 and 9081 to Prometheus API service |
| Prometheus API service | Collects metrics from $[prodname] components and makes the metrics available to the web console. | &bull; TCP 6443 to Kubernetes API server<br />&bull; TCP 2020 to Fluent Bit<br />&bull; TCP 9900 and 9081 to Prometheus API service |
| Tigera API server | Allows users to manage $[prodname] resources such as policies and tiers through kubectl or the Kubernetes API server. | &bull; TCP 9095 to Prometheus API service<br />&bull; TCP 8080 from Kubernetes API server |
| Typha | Increases scale by reducing each node’s impact on the datastore. | TCP 5473 from calico-node to Typha |
| User access to the web console | Authenticated users can access the browser-based the web console, which provides network traffic visibility and troubleshooting, centralized multi-cluster management, threat-defense, container threat detection, policy lifecycle management, scan images for vulnerabilities, and compliance for multiple roles/stakeholders. | Port 443 to $[prodname] tunnel server |
30 changes: 15 additions & 15 deletions calico-cloud/get-started/operator-checklist.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ If you are using the **AWS or Azure CNI plugin**, a degraded state is likely bec

**Sample output**

In the following example, the typha component has an issue because it is showing `AVAILABLE: FALSE`, and `DEGRADED: TRUE`. To understand details of $[prodname] components, see [Deep dive into custom resources](#deep-dive-into-custom-resources).
In the following example, the `typha` component has an issue because it is showing `AVAILABLE: FALSE`, and `DEGRADED: TRUE`. To understand details of $[prodname] components, see [Deep dive into custom resources](#deep-dive-into-custom-resources).

```yaml
apiVersion: v1
Expand Down Expand Up @@ -400,7 +400,7 @@ kubectl get tigerastatus

| | NAME | AVAILABLE | PROGRESSING | DEGRADED | SINCE |
| --- | ----------------------------- | --------- | ----------- | -------- | ----- |
| 1 | apiserver | TRUE | FALSE | FALSE | 10m |
| 1 | `apiserver` | TRUE | FALSE | FALSE | 10m |
| 2 | calico | TRUE | FALSE | FALSE | 11m |
| 3 | cloud-core | TRUE | FALSE | FALSE | 11m |
| 4 | compliance | TRUE | FALSE | FALSE | 9m39s |
Expand All @@ -410,9 +410,9 @@ kubectl get tigerastatus
| 8 | monitor | TRUE | FALSE | FALSE | 11m |
| 9 | runtime-security | TRUE | FALSE | FALSE | 10m |

**1 - api server**
**1 - API server**

`apiserver` is a required component that is an aggregated api-server. It is required for things like applying the tigera license. If `tigerastatus` reports it as unavailable or degraded, check the pods and logs in the `calico-system`namespace. For example,
`apiserver` is a required component that is an aggregated API server. It is required for things like applying the Tigera license. If `tigerastatus` reports it as unavailable or degraded, check the pods and logs in the `calico-system` namespace. For example,

```bash
kubectl get pods -n calico-system
Expand Down Expand Up @@ -493,19 +493,19 @@ intrusion-detection-es-job-installer-xm22v 1/1 Running 0 6

**6 - log-collector**

`log-collector` collects flow and other logs and forwards them to $[prodname]. Check the pods and logs in the `tigera-fluentd` namespace. You should have one pod running on each of your nodes.
`log-collector` collects flow and other logs and forwards them to $[prodname]. Check the `calico-fluent-bit` pods and logs in the `calico-system` namespace. You should have one pod running on each of your nodes.

```bash
kubectl get pods -n tigera-fluentd
kubectl get pods -n calico-system -l k8s-app=calico-fluent-bit
```

```
NAME READY STATUS RESTARTS AGE
fluentd-node-5mzh6 1/1 Running 0 70m
fluentd-node-7vmxw 1/1 Running 0 70m
fluentd-node-bbc4p 1/1 Running 0 70m
fluentd-node-chfz4 1/1 Running 0 70m
fluentd-node-d6f56 1/1 Running 0 70m
NAME READY STATUS RESTARTS AGE
calico-fluent-bit-5mzh6 1/1 Running 0 70m
calico-fluent-bit-7vmxw 1/1 Running 0 70m
calico-fluent-bit-bbc4p 1/1 Running 0 70m
calico-fluent-bit-chfz4 1/1 Running 0 70m
calico-fluent-bit-d6f56 1/1 Running 0 70m
```

**7 - management-cluster-connection**
Expand Down Expand Up @@ -557,7 +557,7 @@ cmEtdm9sdHJvbjAeFw0yMDEyMjExOTA1MzhaFw0yNTEyMjAxOTA1MzhaMBkxFzAV

**8 - monitor**

`monitor` is responsible for configuring prometheus and associated custom resources. Check the pods and logs in the `tigera-prometheus` namespace.
`monitor` is responsible for configuring Prometheus and associated custom resources. Check the pods and logs in the `tigera-prometheus` namespace.

```bash
$ kubectl get pods -n tigera-prometheus
Expand Down Expand Up @@ -643,8 +643,8 @@ If cluster does not have enough capacity, it will not be able to deploy pods. Th

The high-level components $[prodname] needs to run are:

- Per node: 1 fluentd, 1 compliance benchmarker
- On top of per node: 3 alertmanager (from statefulset), 1 prometheus, 1 prometheus operator, 1 kube-controllers, 2 compliance snapshotter and controller, 1 guardian, 1 ids controller, 1 apiserver
- Per node: 1 `fluent-bit`, 1 `compliance-benchmarker`
- On top of per node: 3 `alertmanager` (from StatefulSet), 1 `prometheus`, 1 `prometheus` operator, 1 kube-controllers, 2 compliance snapshotter and controller, 1 guardian, 1 ids controller, 1 `apiserver`

Some clusters have limited pod-networked pod capacity.

Expand Down
59 changes: 33 additions & 26 deletions calico-cloud/observability/elastic/dns/filtering-dns.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Suppress low-value Calico Cloud DNS log entries with Fluentd filters configured through a ConfigMap in the operator namespace of connected clusters.
description: Suppress low-value Calico Cloud DNS log entries with Fluent Bit filters configured through a ConfigMap in the operator namespace of connected clusters.
---

# Filter DNS logs
Expand All @@ -18,46 +18,53 @@
your desired filter using [Filter configuration files](#filter-configuration-files).
If you are also adding [flow filters](../flow/filtering.mdx) also add the `flow` file
to the directory.
1. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace
1. Create the `fluent-bit-filters` ConfigMap in the `tigera-operator` namespace
with the following command.
```bash
kubectl create configmap fluentd-filters -n tigera-operator --from-file=filters
kubectl create configmap fluent-bit-filters -n tigera-operator --from-file=filters
```

The operator inserts the filters inline into the log collector configuration and rolls the
`calico-fluent-bit` daemonset automatically.

Check warning on line 28 in calico-cloud/observability/elastic/dns/filtering-dns.mdx

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Terms] Use 'DaemonSets?' instead of 'daemonset'.

Raw Output:
{"message": "[Vale.Terms] Use 'DaemonSets?' instead of 'daemonset'.", "location": {"path": "calico-cloud/observability/elastic/dns/filtering-dns.mdx", "range": {"start": {"line": 28, "column": 21}}}, "severity": "WARNING"}

## Filter configuration files

The filters defined by the ConfigMap are inserted into the fluentd configuration file.
The [upstream fluentd documentation](https://docs.fluentd.org/filter/grep)
describes how to write fluentd filters. The [DNS log schema](dns-logs.mdx) can be referred to
for the specification of the various fields you can filter based on. Remember to ensure
that the config file is properly indented in the ConfigMap.
Each file holds a YAML list of Fluent Bit filter entries. The
[upstream Fluent Bit documentation](https://docs.fluentbit.io/manual/data-pipeline/filters/grep)
describes how to write the `grep` filter used in the examples below; the
`calico-fluent-bit` log collector also ships the `record_modifier`, `parser`, and
`lua` filters. Filters in the `dns` file are applied to DNS logs automatically;
you do not need to set a `match` on each entry. The [DNS log schema](dns-logs.mdx)
can be referred to for the specification of the various fields you can filter based on.

:::note Upgrading from a release that used Fluentd

Earlier releases collected logs with Fluentd and read filters in Fluentd `<filter>`
syntax from a ConfigMap named `fluentd-filters`. That ConfigMap is no longer read,
and Fluentd filter syntax cannot be translated automatically. Recreate your filters
as Fluent Bit YAML filter lists under the new `fluent-bit-filters` name. If a filter
key does not parse as Fluent Bit YAML, the operator skips that filter, reports a
warning on the `tigera status` output naming the offending key, and continues to
ship unfiltered logs.

:::

## Example 1: filter out cluster-internal lookups

This example filters out lookups for domain names ending with ".cluster.local". More
logs could be filtered by adjusting the regular expression "pattern", or by adding
additional `exclude` blocks.
logs could be filtered by adjusting the regular expression, or by adding
additional `exclude` rules.

```
<filter dns>
@type grep
<exclude>
key qname
pattern /\.cluster\.local$/
</exclude>
</filter>
```yaml
- name: grep
exclude: qname \.cluster\.local$
```

## Example 2: keep logs only for particular domain names

This example will filter out all logs _except_ those for domain names ending `.co.uk`.

```
<filter dns>
@type grep
<regexp>
key qname
pattern /\.co\.uk$/
</regexp>
</filter>
```yaml
- name: grep
regex: qname \.co\.uk$
```
Loading