The service with no endpoints | Labs | DevOps in Practice

After a tidy-up of application labels, checkout-web is running but traffic through its Kubernetes Service returns 503.

Start by checking whether the workload is actually running.

Try: kubectl get pods

Terminal command

The Service has endpoints again.

checkout-web now selects the running Pods, so traffic has concrete backends instead of an empty endpoint list.

Lesson

When Pods are healthy but a Service returns 503, compare the Service selector with Pod labels before changing the workload. Fixing the selector is smaller and safer than restarting healthy Pods.

Edit service/checkout-web

Change the selector so it matches the labels on the running checkout-web Pods.

The Service selector still needs app: checkout-web while keeping tier: frontend.

Key concepts

What is Kubernetes?

A system for running and connecting containers.

What it is

Kubernetes is an open-source platform that runs containerized applications and gives them stable networking primitives.

In this incident

The cluster is running checkout-web Pods, but traffic through the Service does not reach them.

Why it matters

Kubernetes separates running workloads from the stable names and virtual IPs that other workloads use.

What is a Pod?

The smallest workload Kubernetes runs.

What it is

A Pod is the smallest workload Kubernetes runs. It wraps one or more containers that share network and storage.

In this incident

The checkout-web Pods are healthy, so the application process itself is not the first suspect.

Why it matters

A healthy Pod can still be unreachable if the routing object in front of it cannot select it.

What is a Service?

A stable network entry point for matching Pods.

What it is

A Service gives clients a stable virtual IP and DNS name, then forwards traffic to Pods selected by labels.

In this incident

service/checkout-web exists, but its selector does not match the Pods it should route to.

Why it matters

Services decouple callers from individual Pod names and IPs, which change during deploys and restarts.

What is a selector?

The label query a Kubernetes object uses.

What it is

A selector is a label query. A Service selector identifies which Pods should receive traffic.

In this incident

The Service asks for app=checkout, but the Pods now advertise app=checkout-web.

Why it matters

Selectors are small strings with large impact. One stale label value can remove every backend from a Service.

What are labels?

Key-value metadata used for grouping resources.

What it is

Labels are key-value pairs attached to Kubernetes objects so other objects and commands can group or select them.

In this incident

The checkout-web Pods have app=checkout-web and tier=frontend labels.

Why it matters

Label changes are coordination changes. Deployments, Services, NetworkPolicies, and monitors may all depend on them.

What are endpoints?

The concrete backend addresses behind a Service.

What it is

Endpoints are the Pod IPs and ports currently selected by a Service.

In this incident

The empty endpoint list explains the 503s: there are no selected backends to receive traffic.

Why it matters

Checking endpoints is a direct way to confirm whether Service discovery is connected to real Pods.