The pods that won't stay up | Labs | DevOps in Practice

A new revision of the Kubernetes Deployment receipt-api is restarting its Pods and degrading the service.

Start by checking the workload state.

Try: kubectl get pods

Terminal command

The rollout is healthy.

The application can write to /app/runtime while the rest of the container filesystem stays read-only.

Lesson

A read-only root filesystem is a useful security boundary. Mount a writable volume only where the process actually needs one; emptyDir is appropriate here because the report is temporary and does not need to survive the pod.

Edit deployment/receipt-api

Use the pattern above to complete the writable volume for /app/runtime.

The Pod template still needs matching volumes and volumeMounts, an emptyDir, and mountPath: /app/runtime.

Key concepts

What is Kubernetes?

A system for running and coordinating containers.

What it is

Kubernetes is an open-source platform that runs containerized applications across one or more machines.

In this incident

The cluster is trying to keep the receipt-api workload running, but its containers keep exiting.

Why it matters

Kubernetes continuously compares the desired state with reality and acts when they differ.

What is a Pod?

The smallest workload Kubernetes runs.

What it is

A Pod is the smallest workload Kubernetes runs. It wraps one or more containers that share network and storage.

In this incident

The receipt-api Deployment creates Pods that run the application container.

Why it matters

Pods are disposable. Kubernetes usually replaces an unhealthy Pod instead of repairing it in place.

What is CrashLoopBackOff?

A container is repeatedly starting and exiting.

What it is

CrashLoopBackOff is a Kubernetes status shown when a container keeps failing and Kubernetes delays each restart attempt.

In this incident

Both receipt-api Pods enter CrashLoopBackOff because the application process exits during startup.

Why it matters

It confirms a restart cycle, but not its cause. Logs are the next useful source of evidence.

Read-only does not mean no storage

The container image can stay immutable.

What it is

readOnlyRootFilesystem prevents a process from changing the container's root filesystem.

In this incident

The decisive log line shows the process exiting when it tries to write to /app/runtime.

Why it matters

The security boundary can stay in place. A volume can make only the required path writable.

What is a Deployment?

A declaration of the workload's desired state.

What it is

A Deployment declares the desired state of a workload and manages its ReplicaSets and Pods.

In this incident

Its Pod template enables a read-only root filesystem but does not provide a writable mount.

Why it matters

Changing the template creates new Pods with the corrected configuration instead of mutating the existing ones.

Why use a volume?

It creates a controlled writable boundary.

What it is

A volume is storage attached to a Pod and exposed to a container through a volumeMount.

In this incident

An emptyDir mounted at /app/runtime gives the process the writable path it needs.

Why it matters

emptyDir lasts for the lifetime of the Pod. It fits temporary files, but not data that must persist.

What does rollout status prove?

That the desired replacement became available.

What it is

A rollout is the process of replacing the old Pods after a Deployment's Pod template changes.

In this incident

Kubernetes created the replacement Pods and they reached the available state.

Why it matters

It confirms deployment progress. A functional application check would still be the next production verification.