The private hostname pointing to yesterday | Labs | DevOps in Practice

Internal clients time out after api.corp.internal moved to a new private load balancer. The application itself is healthy.

Resolve the hostname through the VPC resolver and record the answer and TTL.

Try: dig api.corp.internal @10.0.0.2

Terminal command

Internal clients resolve the new ingress.

The private resolver returns a routable address and the HTTPS probe succeeds.

Lesson

DNS troubleshooting is a chain: query the client's resolver, inspect the answer and TTL, verify the route, and only then change the authoritative record. A syntactically valid DNS answer can still be operationally wrong.

Edit private-dns.tf

Point the private A record to the new ingress and use TTL 60 during the migration window.

The A record needs 10.52.3.27 with ttl = 60.

Key concepts

Private DNS is a separate view

The same DNS namespace can return different answers inside a VPC.

What it is

Split-horizon DNS serves internal records to private resolvers while public resolvers may serve another view or none.

In this incident

The VPC private zone answers api.corp.internal with a stale private address.

Why it matters

Always query the resolver used by affected clients, not only a public resolver.

Resolution is not reachability

A valid DNS answer can still point outside a usable route.

What it is

Routing decides which interface and gateway carry packets to the resolved address.

In this incident

10.40.8.15 follows the default NAT path instead of the connected private ingress subnet.

Why it matters

Checking the route prevents misdiagnosing a network-path failure as an application failure.

TTL controls migration lag

Resolvers may cache an old answer until its TTL expires.

What it is

DNS TTL tells caches how long they may reuse a record before querying again.

In this incident

TTL 60 limits the transition window while clients move to 10.52.3.27.

Why it matters

Lower TTL before a planned migration, then raise it after the new path is stable.