Conntrack resolution in DataDog
Example:
- The Postgres client (10.42.0.10) contacts the Postgres server (10.42.0.9). There is a clusterIp in the middle (10.43.168.100)
- The client sends the packet to the clusterIP src(10.42.0.10:35926) -> dst(10.43.168.100:5432).
- The conntrack saves the entry. We can see it
tcp 6 86399 ESTABLISHED src=10.42.0.10 dst=10.43.168.100 sport=35926 dport=5432 src=10.42.0.9 dst=10.42.0.10 sport=5432 dport=35926 [ASSURED] mark=0 use=1
the key is the packet before the NAT, the value is the expected reply tuple. In this way from the serverIP we can reply to the client.
# The expected one
key: src(10.42.0.10:35926) -> dst(10.43.168.100:5432) ----------- value: src(10.42.0.9:5432) -> dst(10.42.0.10:35926)
The opposite one
key: src(10.42.0.9:5432) -> dst(10.42.0.10:35926) ----------- value: src(10.42.0.10:35926) -> dst(10.43.168.100:5432)
Conntrack expiration
Actual mitigation
If the ebpf conntrack entry expires, we query the root ns conntrack to find the resolution. This solution is not ideal because querying the conntrack each time we want to resolve a clusterIP to a pod brings a penalty in performance. Today, we try to store the resolution in an LRU cache in userspace, but balancing this LRU is not easy at all
Ideal solution
Instead of keeping the conntrack EBPF side, we should send the resolution to userspace and keep a hash map (not LRU) there. In this way, we can let this map grow if the system is experiencing more connections. To clean up this, we can keep the strategies already in place in userspace. As a limit in memory, we can use the conntrack size of the system (maybe we should multiply it by 2 since we are storing the conntrack resolution in both directions)
BOUNS POINT: store just one resolution and not both directions to avoid extra memory consumption.
STAC-23356
Conntrack resolution in DataDog
Example:
the key is the packet before the NAT, the value is the expected reply tuple. In this way from the serverIP we can reply to the client.
Conntrack expiration
Actual mitigation
If the ebpf conntrack entry expires, we query the root ns conntrack to find the resolution. This solution is not ideal because querying the conntrack each time we want to resolve a clusterIP to a pod brings a penalty in performance. Today, we try to store the resolution in an LRU cache in userspace, but balancing this LRU is not easy at all
Ideal solution
Instead of keeping the conntrack EBPF side, we should send the resolution to userspace and keep a hash map (not LRU) there. In this way, we can let this map grow if the system is experiencing more connections. To clean up this, we can keep the strategies already in place in userspace. As a limit in memory, we can use the conntrack size of the system (maybe we should multiply it by 2 since we are storing the conntrack resolution in both directions)
BOUNS POINT: store just one resolution and not both directions to avoid extra memory consumption.