Application troubleshooting on OpenShift: 10 commands before blaming the cluster

The problem that the first wrong command amplifies

The API Route stops responding at 2 PM on a Wednesday. The incident reaches the team channel and the first reflex is oc get nodes. The nodes are Ready. oc get clusterversion — cluster Available. Meanwhile, the API Deployment has zero Ready replicas in the namespace, because the rollout pulled an image whose registry Secret disappeared in the last cleanup.

The cluster was never unavailable. The application was.

This pattern repeats because the boundary between application and platform is not intuitive. A CrashLoop from an image with USER root on restricted-v2 looks like "OpenShift blocked my application," but the cause is in the Dockerfile. A Route with 503 looks like "the router is down," but the Service Endpoints are empty because the selector does not match the pod after a Helm refactor. Part 1 mapped the seven application Day-2 mistakes. This article is the operational runbook — ten commands, by layer, in the project.

If a ClusterOperator or a node is NotReady, the scope changes: the incident belongs to the platform. The commands below assume the opposite.

---

What a layered runbook in the namespace is

A useful runbook is not a list of oc get. It is an order that narrows the scope. Each layer that answers "healthy" rules out the next one as root cause. If the Deployment is Available and the pods are Ready, the next suspect is the Route or the Secret — not the kubelet.

The six layers, from the application into the namespace:

1. Rollout — did the Deployment (or StatefulSet) progress?

2. Pods — Pending, CrashLoop (phase stays Running), ImagePullBackOff

3. Usage — real CPU and memory (oc adm top pod)

4. Pod logs and debug — --previous, oc debug — in the container, not on the node

5. Exposure — Endpoints, Service, Route (503 vs 502)

6. Events, TLS, Secret, and collection — what the API server recorded; inspect of the project

Skipping a layer is the classic mistake. oc logs on a pod whose Deployment has replicas: 0 wastes the first half hour — there is no container to read. The inverse also applies: oc get clusterversion while the Service selector does not match the pod. The right layer is almost always the one closest to the application.

Output below is illustrative (service api in project <project>, OCP 4.16). Placeholders: <project>, <app>, <digest>.

---

10 commands, from the application into the namespace

Each command: what it answers, how to read the output, the common trap, and the next step.

Layer 1 — Rollout

1. oc get deploy

Always first in the application scope. It answers: does the object that GitOps applies exist? How many replicas does the ReplicaSet want, and how many are Ready?

$ oc get deploy -n <project>
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
api       0/2     2            0           40d
consumer  1/1     1            1           40d

READY 0/2 with UP-TO-DATE 2 is a rollout that never became Ready — not a cluster down. AVAILABLE 0 on the API with consumer 1/1 isolates the problem in the API Deployment, not in the node. The UP-TO-DATE column shows how many replicas are already in the latest ReplicaSet; if that number diverges from READY, the rollout stalled before completing.

To confirm numerically:

oc get deploy api -n <project> -o jsonpath='{.spec.replicas} {.status.unavailableReplicas}{"\n"}'

2. oc rollout status

Blocks until the Deployment meets progressDeadlineSeconds (default 600 s) or fails. If the terminal hangs for several minutes, the rollout is waiting for pods that never become Ready — the cause is in layer 2.

$ oc rollout status deploy/api -n <project>
error: deployment "api" exceeded its progress deadline

oc rollout history deploy/api -n <project> lists the ReplicaSets. Rollback is oc rollout undo — only safe if the previous image is pinned by digest (Part 1, mistake 2). With a mutable tag, the undo may pull a different binary from the one that was tested.

Layer 2 — Pods

3. oc get pods

Pending and ImagePullBackOff appear in the STATUS column. The trap is CrashLoopBackOff: CrashLoop is not a Pod phase. kubectl/oc displays CrashLoopBackOff in the STATUS column, but the pod's .status.phase stays Running. This means the filter --field-selector status.phase!=Running does not catch CrashLoop pods — it is the most common trap with field-selectors.

oc get pods -n <project>
oc get pods -n <project> --field-selector status.phase!=Running,status.phase!=Succeeded
oc get pods -n <project> | grep CrashLoop

grep works, but for automation the robust path is jq on containerStatuses:

oc get pods -n <project> -o json | \
  jq -r '.items[] | select(.status.containerStatuses[]?.state.waiting.reason == "CrashLoopBackOff")
    | .metadata.name'

Three immediate states and the next step for each:

• CrashLoopBackOff — the container exited with exit ≠ 0 and the kubelet is restarting with exponential backoff (10 s → 20 s → 40 s → … up to 5 min). On OpenShift the UID is random; an image with USER root breaks on restricted-v2 with FailedCreate. Next step: oc logs --previous.

• ImagePullBackOff — registry unreachable, imagePullSecrets missing, nonexistent tag, or a digest the registry does not have. Verify with oc describe pod → Events.

• Pending — FailedScheduling (insufficient resources, taint without toleration), pending PVC, nodeSelector without a match.

Layer 3 — Usage

4. oc adm top pod

Shows real CPU and memory usage in the namespace — not the request declared in the manifest. Without a Metrics Server / monitoring stack enabled, the command fails with an error; that is not proof the pod is healthy.

$ oc adm top pod -n <project>
NAME           CPU(cores)   MEMORY(bytes)
api-6b8f-x4k   12m          180Mi
api-6b8f-n2p   890m         500Mi

A replica at 500Mi with a 512Mi limit is on the edge of OOMKill. The kernel selects the container for termination based on oom_score_adj — BestEffort pods receive the highest score (1000) and are the first to die under memory pressure.

The OOM signal on OpenShift 4.18 does not appear as a dedicated event in oc get events. OOMKilled (exit 137) appears only in the Last State field of oc describe pod, inside containerStatuses. Relying on oc get events to detect OOM is the trap: the event does not exist.

Layer 4 — Pod logs and debug

5. oc logs --previous

After identifying CrashLoop pods in layer 2, the next step is the container — not the node.

oc logs -n <project> deploy/api --tail=80 --previous

--previous reads the log of the container that died. Without that flag, the output is the current restart — the container just came up, has not produced useful output yet, and the log looks empty. This is why "there is no log" is the most frequent complaint during CrashLoop: the log is in the previous container, not the current one.

oc logs deploy/api follows the pod the Deployment points at — convenient, but if there is more than one replica, it picks only one.

6. oc debug

oc debug -n <project> pod/api-6b8f-x4k -- /bin/sh

Creates a copy of the pod with the same spec, without probes and with a TTY. It allows verifying environment variables, network connectivity, and filesystem without affecting the original pod.

oc debug node/… and chroot /host are platform scope — the RHCOS filesystem, not the application container. Outside this runbook, unless the application's node is NotReady.

Layer 5 — Service, Endpoints, Route

7. oc get endpoints and oc get route

The OpenShift router is HAProxy. The difference between 503 and 502 is operationally critical:

• 503 — HAProxy has no backend to send the request to. The router log shows <NOSRV>. The most frequent cause: a Service with empty Endpoints (label selector mismatch, or no Ready pod). The request never reached the application.

• 502 — HAProxy reached the backend, but the connection failed: TCP RST, application down on the port, reencrypt with an invalid certificate. The request reached the backend and was refused.

oc get svc,endpoints,route -n <project>
oc describe route api -n <project>

Empty Endpoints and READY 0/2 confirm layers 1–2. Populated Endpoints and Route 502: TLS or port. A certificate inline on the Route (spec.tls.certificate) is Part 1, mistake 1.

The TLS termination mode determines where to investigate. edge terminates TLS on the router — the router certificate is the suspect. reencrypt requires a valid certificate on the pod — check the application Secret. passthrough delivers TCP straight to the application — use openssl s_client against the pod, not the router.

Selector mismatch is the most frequent cause of 503 with pods Running. oc get svc api -n <project> -o yaml shows the selector; compare it with pod.metadata.labels. app.kubernetes.io/name on the Deployment and app: on the Service is the classic mismatch after a Helm refactor.

Layer 6 — Events, Secret, collection

8. oc get events

Only Normal and Warning. Default TTL on OpenShift: 3 hours (180 minutes, configurable via eventTTLMinutes in KubeAPIServer; upstream Kubernetes: 1 hour). Yesterday's event is already gone from etcd — and if the incident started on the previous shift, the Warning may have vanished before the investigation.

oc get events -n <project> --field-selector type=Warning --sort-by='.lastTimestamp'

Signals in the namespace: FailedScheduling, BackOff, Unhealthy (probe), FailedCreate (quota, SCC), FailedMount (Secret/volume). If the event is FailedCreate with an SCC message, the cause is restricted-v2 refusing UID 0 or a missing capability — application scope, not platform.

9. Route TLS and the mounted Secret

oc extract secret/<name> -n <project> --keys=tls.crt --to=- 2>/dev/null | \
  openssl x509 -noout -enddate
oc get secret -n <project>
oc describe pod -n <project> -l app=api | grep -A2 'SecretName:'

A Secret not found on the mount explains a CrashLoop right after oc delete secret — the kubelet cannot mount the volume and the container does not start. An expired Route certificate explains 502 only on public HTTPS, with pods Ready.

10. oc adm inspect ns/<project>

Runs on the local client — does not schedule a platform must-gather pod. It collects resources, logs, and events from the namespace into a compressed archive. It is the right artifact for a case that belongs to the application: send it to the team or attach it to a ticket.

oc adm inspect ns/<project> --dest-dir=/tmp/inspect-<project>

Cluster-wide oc adm must-gather is a different scope — reserved for when layer 0 (platform) has failed and the infrastructure team needs etcd logs, MCO state, or ClusterOperator details.

---

When the scope changes: application vs platform

Three quick checks separate the scopes:

• Deployment and pods in the namespace. oc get deploy,pods -n <project> answers in seconds. API 0/2 and consumer 1/1: the problem is the API Deployment, not the node. If both are 0/n, layer 6 (events) usually explains it.

• Platform ClusterOperators. oc get clusteroperators | grep -v 'True.*False.*False' — if every operator is healthy, oc get nodes is not the next command. A Warning in openshift-machine-config-operator is a different ticket.

• Signals from describe pod. Readiness probe failed with the process up: the Service already dropped the pod from Endpoints — 503 on the Route with oc get pods showing Running. Liveness probe failed: restart, then CrashLoop. A probe on / that depends on the database turns a momentary middleware instability into pod death (Part 1, mistake 7). FailedScheduling with Insufficient memory in the API namespace, nodes Ready: it is pod request or quota — not cluster-wide capacity.

---

Complete example: the runbook on one page

The artifact that ties the ten commands together is runbooks/api.md in the application Git — the same repository as Part 1. During an incident, run in this order and stop when the layer answers:

oc get deploy -n <project>              →  replicas Ready?
oc rollout status deploy/api -n <project>  →  deadline exceeded?
oc get pods -n <project>                →  Pending / CrashLoop / ImagePull?
oc adm top pod -n <project>             →  memory at the limit ceiling?
oc get pods -n <project> -o json | jq   →  CrashLoop names?
oc logs deploy/api --previous -n <project>  →  cause of death?
oc debug pod/… -n <project>             →  env / network / filesystem?
oc get svc,endpoints,route -n <project> →  503 (no backend) or 502 (backend refused)?
oc get events -n <project> --field-selector type=Warning  →  TTL 3h — event gone?
oc extract secret/… ; openssl enddate   →  TLS expired / Secret missing?
oc adm inspect ns/<project>             →  local collection, not must-gather

Do not start with oc get nodes because the API Route failed. Layers 1 and 2 of the project first.

In Part 3, the guardrails in the application Git — so this runbook is used less.

---

Closing thoughts

In the first hours of an application incident, order in the namespace matters more than the number of commands run cluster-wide.

This runbook does not replace the platform team when the node is NotReady. It stops the application team from treating an image CrashLoop as an OpenShift outage. Three quick signals: 503 with empty Endpoints is selector or readiness; 502 with populated Endpoints is backend or TLS; oc logs --previous only exists if the container already restarted.

Versioned in the service Git, rehearsed on a game day — otherwise it is fiction (Part 1, mistake 7).

---

Resources

• Part 1 of this series

• Deployments (OCP 4.18)

• Gathering cluster data — inspect and must-gather (OCP 4.18)

• Routes — Ingress and load balancing (OCP 4.18)

• Working with pods — logs and debug (OCP 4.18)

• Troubleshooting (OCP 4.18)

• Kubernetes — debug application pods