The policy that allows the API but blocks its names | Labs | DevOps in Practice

inventory-api started failing after an egress policy rollout. Pods are Running, but the application cannot resolve redis.cache.svc.cluster.local.

Separate name resolution from TCP reachability using a test inside the affected Pod.

Try: kubectl exec deployment/inventory-api -- nslookup redis.cache.svc.cluster.local

Terminal command

Service discovery works without opening egress broadly.

inventory-api can resolve and reach Redis while the default-deny boundary remains in place.

Lesson

Network policies fail at dependency boundaries. Isolate DNS from direct IP reachability, then add the narrowest rule that permits the missing dependency—both UDP and TCP 53 to the actual cluster DNS Pods.

Edit networkpolicy/inventory-egress

Preserve default-deny and the Redis rule. Add DNS egress only to kube-dns on UDP and TCP 53.

Add kube-system + k8s-app: kube-dns selectors and allow both UDP 53 and TCP 53.

Key concepts

DNS is an application dependency

Allowing a destination does not help if the client cannot resolve its name.

What it is

Cluster DNS translates Service names into virtual IPs before most application connections begin.

In this incident

Redis IP traffic is allowed, but queries to kube-dns time out.

Why it matters

Default-deny egress policies must account for foundational dependencies such as DNS.

Test one network layer at a time

A direct IP test distinguishes DNS from destination reachability.

What it is

Layered diagnosis tests name resolution, routing, transport, TLS, and application behavior separately.

In this incident

TCP to 10.96.77.12:6379 succeeds while nslookup fails.

Why it matters

This prevents changing Redis or the application when the failure is DNS egress.

Allow the dependency, not the world

DNS access can remain namespace, Pod, protocol, and port scoped.

What it is

NetworkPolicy combines selectors and ports to describe allowed traffic under default-deny.

In this incident

The new rule targets kube-dns Pods in kube-system on UDP and TCP 53 only.

Why it matters

Allowing 0.0.0.0/0 on every port would restore service by discarding the security boundary.