You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e452d19
Browse filesBrowse the repository at this point in the historyBrowse files
Can you add a detailed description to the Diagram on line xxxx for the mermaid diagram right above it? The intention is to provide a detailed description for users with visual impairments so they can understand the diagram.
Copy file name to clipboardExpand all lines: src/content/articles/kubernetes-dns-debugging-ndots-coredns-troubleshooting/index.mdx
+13-8Lines changed: 13 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,16 +14,22 @@ featured: true
14
14
15
15
An application works perfectly in development. You deploy it to Kubernetes, and suddenly external API calls take 5+ seconds. The API itself responds in 100ms. You spend hours debugging network policies, service meshes, and egress configurations. Finally, you discover the culprit: `ndots:5`.
16
16
17
-
That single configuration option—the default in every Kubernetes cluster—causes every lookup like `api.stripe.com` to first try `api.stripe.com.default.svc.cluster.local`, then `api.stripe.com.svc.cluster.local`, then `api.stripe.com.cluster.local`, before finally trying the actual hostname. Three failed lookups, each potentially timing out, equals mysterious latency.
17
+
That single configuration option — the default in every Kubernetes cluster — causes every lookup like `api.stripe.com` to first try `api.stripe.com.default.svc.cluster.local`, then `api.stripe.com.svc.cluster.local`, then `api.stripe.com.cluster.local`, before finally trying the actual hostname. Three failed lookups, each potentially timing out, equals mysterious latency.
18
18
19
19
I've watched teams lose days to this problem. The symptoms look like network issues, timeout misconfigurations, or flaky external services. But most Kubernetes DNS latency traces back to one misunderstood setting. The good news: once you know what to look for, you can diagnose the problem in 30 seconds and fix it in another 30.
20
20
21
21
## Understanding the ndots Problem
22
22
23
-
The `ndots` option controls when the resolver tries search domains versus treating a hostname as absolute. The rule is simple: if a hostname has fewer dots than the `ndots` value, the resolver appends search domains first. With the default `ndots:5`, any hostname with 4 or fewer dots—which includes nearly every external hostname—gets the search domain treatment.
23
+
The `ndots` option controls when the resolver tries search domains versus treating a hostname as absolute. The rule is simple: if a hostname has fewer dots than the `ndots` value, the resolver appends search domains first. With the default `ndots:5`, any hostname with 4 or fewer dots — which includes nearly every external hostname — gets the search domain treatment.
24
24
25
25
Here's what happens when your application looks up `api.stripe.com`. The resolver counts two dots. Two is less than five, so it appends each search domain in order before trying the hostname as-is. The diagram below shows this slow path:
26
26
27
+
<Diagram
28
+
title="External DNS lookup showing failed search domain queries before success."
29
+
description={
30
+
`Sequence diagram showing how Kubernetes DNS resolution slows an external lookup when ndots is set to 5. An Application Pod sends a lookup for api.stripe.com to /etc/resolv.conf. A note over the resolver explains that the hostname has 2 dots and that ndots:5 causes search domains to be tried first. The resolver then sends three queries to CoreDNS in order: api.stripe.com.default.svc.cluster.local, api.stripe.com.svc.cluster.local, and api.stripe.com.cluster.local. CoreDNS returns NXDOMAIN for each of those internal-style names. After those failures, the resolver finally sends api.stripe.com as a fully qualified domain name to CoreDNS. CoreDNS forwards that final query to an upstream DNS server, receives the A record 93.184.216.34, and returns the successful answer back to the application. The diagram emphasizes that the delay comes from the three failed search-domain lookups that happen before the real external hostname is attempted.`
31
+
}
32
+
>
27
33
```mermaid
28
34
sequenceDiagram
29
35
participant App as Application Pod
@@ -45,8 +51,7 @@ sequenceDiagram
45
51
Upstream-->>CoreDNS: A record: 93.184.216.34
46
52
CoreDNS-->>App: A record: 93.184.216.34
47
53
```
48
-
49
-
Figure: External DNS lookup showing failed search domain queries before success.
54
+
</Diagram>
50
55
51
56
Each of those NXDOMAIN responses takes time. The math is painful: with the default 5-second timeout, three failed queries could mean 15+ seconds of latency before your application gets a response. Even with a tuned 1-second timeout, you're looking at 3+ seconds versus the 100ms the API actually needs.
52
57
@@ -82,7 +87,7 @@ time nslookup api.stripe.com
82
87
time nslookup api.stripe.com.
83
88
```
84
89
85
-
Code: The key diagnostic—compare resolution with and without trailing dot.
90
+
Code: The key diagnostic — compare resolution with and without trailing dot.
86
91
87
92
What counts as "significantly faster"? If the FQDN version (with the trailing dot) resolves in 50-100ms while the non-FQDN version takes 2-5 seconds, you've found your problem. The difference tells you exactly how much overhead the search domains add. In severe cases, I've seen the non-FQDN version take 15+ seconds while the FQDN version returns in under 100ms.
88
93
@@ -130,9 +135,9 @@ This approach requires no cluster changes and works immediately. The trailing do
130
135
131
136
### Which Fix Should You Choose?
132
137
133
-
Both approaches work, but they fit different situations. Use **pod-level dnsConfig** when you control the deployment manifests and want a systematic fix that applies to all external lookups from that pod. This is the cleaner solution for new applications or when you're refactoring existing ones.
138
+
Both approaches work, but they fit different situations. Use __pod-level dnsConfig__ when you control the deployment manifests and want a systematic fix that applies to all external lookups from that pod. This is the cleaner solution for new applications or when you're refactoring existing ones.
134
139
135
-
Use **trailing dots in configuration** when you need a quick fix without changing deployment manifests, or when you're dealing with a third-party application where you can only modify environment variables. It's also useful for targeted fixes—if only one external service is causing problems, you can add the trailing dot to just that hostname.
140
+
Use __trailing dots in configuration__ when you need a quick fix without changing deployment manifests, or when you're dealing with a third-party application where you can only modify environment variables. It's also useful for targeted fixes — if only one external service is causing problems, you can add the trailing dot to just that hostname.
136
141
137
142
For most teams, I recommend starting with trailing dots for immediate relief, then migrating to pod-level dnsConfig as part of your next deployment update.
138
143
@@ -210,7 +215,7 @@ When connectivity issues arise and you suspect DNS, here's the fastest path to c
210
215
]}
211
216
/>
212
217
213
-
If the ndots test doesn't show a significant difference, the problem lies elsewhere—check CoreDNS health with `kubectl get pods -n kube-system -l k8s-app=kube-dns` and examine its logs for errors.
218
+
If the ndots test doesn't show a significant difference, the problem lies elsewhere — check CoreDNS health with `kubectl get pods -n kube-system -l k8s-app=kube-dns` and examine its logs for errors.
214
219
215
220
<Callout type="warning" title="Always Test From Inside a Pod">
216
221
Node-level DNS and pod-level DNS are configured differently. A successful `nslookup` from the node doesn't prove DNS works for your application pods. Always test from inside a pod to get accurate results.
0 commit comments