Troubleshoot NodeLocalDNS¶
Running NodeLocalDNS presents issues for certain Linux distributions, such as RHEL, centOS, and Rocky Linux.
Pods stuck in crash loopbackFailure¶
After enabling NodelocalDNS in MKE, NodelocalDNS pods may become stuck in the crash loopback.
kubectl get po -o wide -n kube-system -l k8s-app=node-local-dns
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-local-dns-cg49w 0/1 CrashLoopBackOff 5 (79s ago) 3m15s 172.31.32.61 centos7-centos-0 <none> <none>
node-local-dns-ldjk4 0/1 CrashLoopBackOff 5 (7s ago) 3m15s 172.31.45.15 centos7-centos-1 <none> <none>
kubectl logs -f -n kube-system -l k8s-app=node-local-dns
2024/05/14 17:34:05 [ERROR] Failed to add non-existent interface nodelocaldns: operation not supported
2024/05/14 17:34:05 [INFO] Added interface - nodelocaldns
2024/05/14 17:34:05 [ERROR] Error checking dummy device nodelocaldns - operation not supported
listen tcp 169.254.0.10:8080: bind: cannot assign requested address
The reason this happens is that the NodeLocalDNS DaemonSet creates a dummy interface during network setup, and this dummy kernel module is not loaded in RHEL or CENTOS by default. To fix the issue, load the dummy kernel module and run the following command on every node in the cluster:
sudo modprobe dummy
NodeLocalDNS containers are unable to add iptables rules¶
Although the NodeLocalDNS Pods switch to running
state after the dummy
kernel module is loaded, the Pods still fail to add iptables rules.
The error presents in the NodeLocalDNS Pods logs.
kubectl get po -o wide -n kube-system -l k8s-app=node-local-dns
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-local-dns-khfh7 1/1 Running 0 4m11s 172.31.32.61 centos7-centos-0 <none> <none>
node-local-dns-shtqn 1/1 Running 3 (3m48s ago) 4m11s 172.31.45.15
centos7-centos-1 <none> <none>
kubectl logs -f -n kube-system -l k8s-app=node-local-dns
Notice: The NOTRACK target is converted into CT target in rule listing and saving.
Fatal: can't open lock file /run/xtables.lock: Permission denied
[ERROR] Error checking/adding iptables rule {raw OUTPUT [-p tcp -d 10.96.0.10 --dport 8080 -j NOTRACK -m comment --comment NodeLocal DNS Cache: skip conntrack]}, error - error checking rule: exit status 4: Ignoring deprecated --wait-interval option.
Warning: Extension CT revision 0 not supported, missing kernel module?
Notice: The NOTRACK target is converted into CT target in rule listing and saving.
Fatal: can't open lock file /run/xtables.lock: Permission denied
You can fix this problem in two different ways:
Use
audit2allow
to generate SELinux policy rules for the denied operations:module localdnsthird 1.0; require { type kernel_t; type spc_t; type rpm_script_t; type firewalld_t; type container_t; type iptables_var_run_t; class process transition; class capability { sys_admin sys_resource }; class system module_request; class file { lock open read }; } #============= container_t ============== allow container_t iptables_var_run_t:file lock; #!!!! This avc is allowed in the current policy allow container_t iptables_var_run_t:file { open read }; #!!!! This avc is allowed in the current policy allow container_t kernel_t:system module_request; #============= firewalld_t ============== #!!!! This avc is allowed in the current policy allow firewalld_t self:capability { sys_admin sys_resource }; #============= spc_t ============== #!!!! This avc is allowed in the current policy allow spc_t rpm_script_t:process transition;
Change the SELinux mode to permissive:
sudo setenforce 0