The ALB with healthy instances and no traffic | Labs | DevOps in Practice

A release completed successfully, but the public API returns 503 because every ALB target is marked unhealthy.

Inspect target health before changing the application or instances.

Try: aws elbv2 describe-target-health --target-group-arn tg/payments-api

Terminal command

The ALB is routing again.

Both targets pass the exact readiness check and are eligible for traffic.

Lesson

When an ALB reports response-code mismatches, inspect the configured probe and test it directly. Keep health checks specific: fixing the path is safer than accepting every response as healthy.

Edit payments-api-target-group.tf

Change only the health-check path; keep the success matcher strict.

The health check needs path = "/health/ready" and matcher = "200".

Key concepts

Target health is the routing gate

An ALB routes only to targets that pass its checks.

What it is

A target group periodically probes every registered backend and records its health.

In this incident

The instances run, but HTTP 404 makes the ALB reject them.

Why it matters

Instance health and application readiness are different signals.

Readiness endpoints are contracts

Infrastructure depends on a stable application path and status code.

What it is

A readiness endpoint reports whether a process can safely receive traffic.

In this incident

The application moved readiness from /health to /health/ready.

Why it matters

Changing the contract requires coordinated application and infrastructure changes.

Fix the failed contract, not the symptoms

A narrow health-check change is safer than replacing healthy instances.

What it is

The smallest safe change corrects the known mismatch without widening accepted responses.

In this incident

The matcher remains 200 while only the path changes.

Why it matters

Broad matchers can hide redirects, authentication pages, and real failures.