# Application troubleshooting on OpenShift: 10 commands before blaming the cluster ## The problem that the first wrong command amplifies The API Route stops responding at 2 PM on a Wednesday. The incident reaches the team channel and the first reflex is `oc get nodes`. The nodes are `Ready`. `oc get clusterversion` — cluster `Available`. Meanwhile, the API Deployment has zero Ready replicas in the **namespace**, because the rollout pulled an image whose registry Secret disappeared in the last cleanup. The cluster was never unavailable. The application was. This pattern repeats because the boundary between application and platform is not intuitive. A CrashLoop from an image with USER root on `restricted-v2` looks like "OpenShift blocked my application," but the cause is in the Dockerfile. A Route with 503 looks like "the router is down," but the Service Endpoints are empty because the selector does not match the pod after a Helm refactor. Part 1 mapped the seven application Day-2 mistakes. This article is the operational runbook — ten commands, by layer, in the project. If a ClusterOperator or a node is `NotReady`, the scope changes: the incident belongs to the platform. The commands below assume the opposite. * * * ## What a layered runbook in the namespace is A useful runbook is not a list of `oc get`. It is an **order that narrows the scope**. Each layer that answers "healthy" rules out the next one as root cause. If the Deployment is `Available` and the pods are `Ready`, the next suspect is the Route or the Secret — not the kubelet. The six layers, from the application into the namespace: 1. **Rollout** — did the Deployment (or StatefulSet) progress? 2. **Pods** — `Pending`, CrashLoop (phase stays `Running`), ImagePullBackOff 3. **Usage** — real CPU and memory (`oc adm top pod`) 4. **Pod logs and debug** — `--previous`, `oc debug` — in the container, not on the node 5. **Exposure** — Endpoints, Service, Route (503 vs 502) 6. **Events, TLS, Secret, and collection** — what the API server recorded; `inspect` of the project Skipping a layer is the classic mistake. `oc logs` on a pod whose Deployment has `replicas: 0` wastes the first half hour — there is no container to read. The inverse also applies: `oc get clusterversion` while the Service `selector` does not match the pod. The right layer is almost always the one closest to the application. Output below is **illustrative** (service `api` in project ``, OCP 4.16). Placeholders: ``, ``, ``. * * * ## 10 commands, from the application into the namespace Each command: what it answers, how to read the output, the common trap, and the next step. ### Layer 1 — Rollout #### 1\. `oc get deploy` Always first in the application scope. It answers: does the object that GitOps applies exist? How many replicas does the ReplicaSet want, and how many are Ready? ```bash $ oc get deploy -n NAME READY UP-TO-DATE AVAILABLE AGE api 0/2 2 0 40d consumer 1/1 1 1 40d ``` `READY 0/2` with `UP-TO-DATE 2` is a rollout that never became Ready — not a cluster down. `AVAILABLE 0` on the API with consumer `1/1` isolates the problem in the API Deployment, not in the node. The `UP-TO-DATE` column shows how many replicas are already in the latest ReplicaSet; if that number diverges from `READY`, the rollout stalled before completing. To confirm numerically: ```bash oc get deploy api -n -o jsonpath='{.spec.replicas} {.status.unavailableReplicas}{"\n"}' ``` #### 2\. `oc rollout status` Blocks until the Deployment meets `progressDeadlineSeconds` (default 600 s) or fails. If the terminal hangs for several minutes, the rollout is waiting for pods that never become Ready — the cause is in layer 2. ```bash $ oc rollout status deploy/api -n error: deployment "api" exceeded its progress deadline ``` `oc rollout history deploy/api -n ` lists the ReplicaSets. Rollback is `oc rollout undo` — only safe if the previous image is pinned by digest (Part 1, mistake 2). With a mutable tag, the undo may pull a different binary from the one that was tested. ### Layer 2 — Pods #### 3\. `oc get pods` `Pending` and `ImagePullBackOff` appear in the STATUS column. The trap is CrashLoopBackOff: **CrashLoop is not a Pod phase**. `kubectl`/`oc` displays `CrashLoopBackOff` in the STATUS column, but the pod's `.status.phase` stays `Running`. This means the filter `--field-selector status.phase!=Running` **does not catch** CrashLoop pods — it is the most common trap with field-selectors. ```bash oc get pods -n oc get pods -n --field-selector status.phase!=Running,status.phase!=Succeeded oc get pods -n | grep CrashLoop ``` `grep` works, but for automation the robust path is `jq` on `containerStatuses`: ```bash oc get pods -n -o json | \ jq -r '.items[] | select(.status.containerStatuses[]?.state.waiting.reason == "CrashLoopBackOff") | .metadata.name' ``` Three immediate states and the next step for each: * **CrashLoopBackOff** — the container exited with exit ≠ 0 and the kubelet is restarting with exponential backoff (10 s → 20 s → 40 s → … up to 5 min). On OpenShift the UID is random; an image with USER root breaks on `restricted-v2` with `FailedCreate`. Next step: `oc logs --previous`. * **ImagePullBackOff** — registry unreachable, `imagePullSecrets` missing, nonexistent tag, or a digest the registry does not have. Verify with `oc describe pod` → `Events`. * **Pending** — `FailedScheduling` (insufficient resources, taint without toleration), pending PVC, nodeSelector without a match. ### Layer 3 — Usage #### 4\. `oc adm top pod` Shows **real** CPU and memory usage in the namespace — not the request declared in the manifest. Without a Metrics Server / monitoring stack enabled, the command fails with an error; that is not proof the pod is healthy. ```bash $ oc adm top pod -n NAME CPU(cores) MEMORY(bytes) api-6b8f-x4k 12m 180Mi api-6b8f-n2p 890m 500Mi ``` A replica at 500Mi with a 512Mi limit is on the edge of OOMKill. The kernel selects the container for termination based on `oom_score_adj` — BestEffort pods receive the highest score (`1000`) and are the first to die under memory pressure. The OOM signal on OpenShift 4.18 **does not** appear as a dedicated event in `oc get events`. `OOMKilled` (exit 137) appears only in the `Last State` field of `oc describe pod`, inside `containerStatuses`. Relying on `oc get events` to detect OOM is the trap: the event does not exist. ### Layer 4 — Pod logs and debug #### 5\. `oc logs --previous` After identifying CrashLoop pods in layer 2, the next step is the container — not the node. ```bash oc logs -n deploy/api --tail=80 --previous ``` `--previous` reads the log of the container **that died**. Without that flag, the output is the current restart — the container just came up, has not produced useful output yet, and the log looks empty. This is why "there is no log" is the most frequent complaint during CrashLoop: the log is in the previous container, not the current one. `oc logs deploy/api` follows the pod the Deployment points at — convenient, but if there is more than one replica, it picks only one. #### 6\. `oc debug` ```bash oc debug -n pod/api-6b8f-x4k -- /bin/sh ``` Creates a **copy** of the pod with the same spec, without probes and with a TTY. It allows verifying environment variables, network connectivity, and filesystem without affecting the original pod. `oc debug node/…` and `chroot /host` are **platform** scope — the RHCOS filesystem, not the application container. Outside this runbook, unless the application's node is `NotReady`. ### Layer 5 — Service, Endpoints, Route #### 7\. `oc get endpoints` and `oc get route` The OpenShift router is HAProxy. The difference between 503 and 502 is operationally critical: * **503** — HAProxy has **no** backend to send the request to. The router log shows ``. The most frequent cause: a Service with empty Endpoints (label selector mismatch, or no Ready pod). The request never reached the application. * **502** — HAProxy reached the backend, but the connection failed: TCP RST, application down on the port, reencrypt with an invalid certificate. The request reached the backend and was refused. ```bash oc get svc,endpoints,route -n oc describe route api -n ``` Empty `Endpoints` and `READY 0/2` confirm layers 1–2. Populated Endpoints and Route 502: TLS or port. A certificate **inline** on the Route (`spec.tls.certificate`) is Part 1, mistake 1. The TLS termination mode determines where to investigate. `edge` terminates TLS on the router — the router certificate is the suspect. `reencrypt` requires a valid certificate **on the pod** — check the application Secret. `passthrough` delivers TCP straight to the application — use `openssl s_client` against the pod, not the router. Selector mismatch is the most frequent cause of 503 with pods Running. `oc get svc api -n -o yaml` shows the `selector`; compare it with `pod.metadata.labels`. `app.kubernetes.io/name` on the Deployment and `app:` on the Service is the classic mismatch after a Helm refactor. ### Layer 6 — Events, Secret, collection #### 8\. `oc get events` Only `Normal` and `Warning`. Default TTL on OpenShift: **3 hours** (180 minutes, configurable via `eventTTLMinutes` in KubeAPIServer; upstream Kubernetes: 1 hour). Yesterday's event is already gone from etcd — and if the incident started on the previous shift, the Warning may have vanished before the investigation. ```bash oc get events -n --field-selector type=Warning --sort-by='.lastTimestamp' ``` Signals in the namespace: `FailedScheduling`, `BackOff`, `Unhealthy` (probe), `FailedCreate` (quota, SCC), `FailedMount` (Secret/volume). If the event is `FailedCreate` with an SCC message, the cause is `restricted-v2` refusing UID 0 or a missing capability — application scope, not platform. #### 9\. Route TLS and the mounted Secret ```bash oc extract secret/ -n --keys=tls.crt --to=- 2>/dev/null | \ openssl x509 -noout -enddate oc get secret -n oc describe pod -n -l app=api | grep -A2 'SecretName:' ``` A Secret `not found` on the mount explains a CrashLoop right after `oc delete secret` — the kubelet cannot mount the volume and the container does not start. An expired Route certificate explains 502 only on public HTTPS, with pods Ready. #### 10\. `oc adm inspect ns/` Runs on the **local client** — does not schedule a platform must-gather pod. It collects resources, logs, and events from the namespace into a compressed archive. It is the right artifact for a case that belongs to the **application**: send it to the team or attach it to a ticket. ```bash oc adm inspect ns/ --dest-dir=/tmp/inspect- ``` Cluster-wide `oc adm must-gather` is a different scope — reserved for when layer 0 (platform) has failed and the infrastructure team needs etcd logs, MCO state, or ClusterOperator details. * * * ## When the scope changes: application vs platform Three quick checks separate the scopes: * **Deployment and pods in the namespace.** `oc get deploy,pods -n ` answers in seconds. API `0/2` and consumer `1/1`: the problem is the API Deployment, not the node. If both are `0/n`, layer 6 (events) usually explains it. * **Platform ClusterOperators.** `oc get clusteroperators | grep -v 'True.*False.*False'` — if every operator is healthy, `oc get nodes` is not the next command. A Warning in `openshift-machine-config-operator` is a different ticket. * **Signals from** `describe pod`**.** `Readiness probe failed` with the process up: the Service already dropped the pod from Endpoints — 503 on the Route with `oc get pods` showing `Running`. `Liveness probe failed`: restart, then CrashLoop. A probe on `/` that depends on the database turns a momentary middleware instability into pod death (Part 1, mistake 7). `FailedScheduling` with `Insufficient memory` in the API namespace, nodes `Ready`: it is pod request or quota — not cluster-wide capacity. * * * ## Complete example: the runbook on one page The artifact that ties the ten commands together is `runbooks/api.md` in the application Git — the same repository as Part 1. During an incident, run **in this order** and stop when the layer answers: ```plaintext oc get deploy -n → replicas Ready? oc rollout status deploy/api -n → deadline exceeded? oc get pods -n → Pending / CrashLoop / ImagePull? oc adm top pod -n → memory at the limit ceiling? oc get pods -n -o json | jq → CrashLoop names? oc logs deploy/api --previous -n → cause of death? oc debug pod/… -n → env / network / filesystem? oc get svc,endpoints,route -n → 503 (no backend) or 502 (backend refused)? oc get events -n --field-selector type=Warning → TTL 3h — event gone? oc extract secret/… ; openssl enddate → TLS expired / Secret missing? oc adm inspect ns/ → local collection, not must-gather ``` Do not start with `oc get nodes` because the API Route failed. Layers 1 and 2 of the **project** first. In Part 3, the guardrails in the application Git — so this runbook is used less. * * * ## Closing thoughts **In the first hours of an application incident, order in the namespace matters more than the number of commands run cluster-wide.** This runbook does not replace the platform team when the node is `NotReady`. It stops the application team from treating an image CrashLoop as an OpenShift outage. Three quick signals: 503 with empty Endpoints is selector or readiness; 502 with populated Endpoints is backend or TLS; `oc logs --previous` only exists if the container already restarted. Versioned in the service Git, rehearsed on a game day — otherwise it is fiction (Part 1, mistake 7). * * * ## Resources * [Part 1 of this series](https://gferreir.hashnode.dev/openshift-day2-mistakes-that-take-down-clusters) * [Deployments (OCP 4.18)](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/building_applications/deployments) * [Gathering cluster data — inspect and must-gather (OCP 4.18)](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/support/gathering-cluster-data) * [Routes — Ingress and load balancing (OCP 4.18)](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/ingress_and_load_balancing/routes) * [Working with pods — logs and debug (OCP 4.18)](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/nodes/working-with-pods) * [Troubleshooting (OCP 4.18)](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/support/troubleshooting) * [Kubernetes — debug application pods](https://kubernetes.io/docs/tasks/debug/)