openshift-deployment

---
name: openshift-deployment
description: BC Gov Private Cloud OpenShift workload-manifest guidance: picking the right controller (Deployment, StatefulSet, DaemonSet, Job, CronJob) and shaping it for the platform — container resources and QoS, namespace LimitRange defaults, pod and container securityContext hardening (runAsNonRoot, drop ALL capabilities, no privilege escalation, read-only root filesystem) for the restricted-v2 SCC, liveness/readiness/startup probes, terminationGracePeriod and SIGTERM handling, PID-1 reaping with tini or dumb-init, PodDisruptionBudgets, horizontal scaling (HPA, VPA), and CronJob admission rules. Use when authoring or reviewing a workload manifest bound for a license-plate runtime namespace. Out of scope: NetworkPolicies, Vault/ESO, Argo CD/GitOps, Artifactory image pulls, Porter TCP exposure, storage class choice, Sysdig, ACS runtime policies, security assessment.
license: Apache-2.0
---

# OpenShift Deployment — BC Gov Private Cloud Workloads

## Use When
- Authoring or reviewing a new workload manifest (`Deployment`, `StatefulSet`, `DaemonSet`, `Job`, or `CronJob` — raw, Helm chart, or Kustomize base) bound for a license-plate `-dev` / `-test` / `-prod` namespace on Silver, Gold, GoldDR, or Emerald.
- Picking which controller type fits a given workload (stateless API → Deployment; ordered identity + per-pod PVC → StatefulSet; one pod per node → DaemonSet; one-shot / parallel work → Job; scheduled → CronJob).
- Hardening an existing workload to meet the BC Gov platform's mandatory pod hygiene — resource requests / QoS class, the pod and container `securityContext` (`runAsNonRoot`, `allowPrivilegeEscalation: false`, drop `ALL` capabilities, `readOnlyRootFilesystem`, `seccompProfile`), probes, SIGTERM and `terminationGracePeriodSeconds`, PID-1 reaping, PDB, HPA (with explicit rampup / rampdown), and pod anti-affinity / `nodeAffinity` / `topologySpreadConstraints`.
- Setting the `securityContext` so a pod is admitted by the namespace's default `restricted-v2` Security Context Constraint, and adding the writable `emptyDir` volumes (e.g. for `/tmp`) that a `readOnlyRootFilesystem: true` container needs.
- Adding `VerticalPodAutoscaler` recommendations (`updateMode: "Off"`) on a workload with a drifting resource footprint.
- Getting past a Kyverno workload-admission rejection — `CronJob` cadence (≥ 5 min, ≥ 1 h with a PVC), `CRON_TZ=` in `schedule`, or the Emerald-only `DataClass` label rule.
- Diagnosing pod-level failures on a workload you own — `CrashLoopBackOff`, `OOMKilled`, `Evicted`, `TerminationGracePeriodExceeded`, or a defunct-PIDs platform alert.

## Don't Use When

The following concerns are each their own scope and belong in a sibling BC Gov OpenShift skill. Those siblings are **planned** — until they land in this repo, use the linked upstream Kubernetes / Red Hat docs or the BC Gov Private Cloud TechDocs:

- Writing or debugging `NetworkPolicy` objects (default-deny, allow-from-router, allow-from-same-namespace, SDN-vs-NSX-T egress) — `openshift-networking` (planned).
- Sourcing secrets from Vault via External Secrets Operator, or rotating Vault data — `openshift-secrets` (planned).
- Choosing or configuring an image registry pull path, creating an `ArtifactoryServiceAccount`, or rotating Artifactory credentials — `openshift-images` (planned).
- Onboarding to Argo CD, authoring a `GitOpsTeam` CR, or designing a `bcgov-c/tenant-gitops-<licenseplate>` repo layout — `openshift-gitops` (planned).
- Picking a storage class, requesting a quota tier, or designing PVC backups — `openshift-storage` (planned).
- Onboarding to Sysdig Monitor, building dashboards, or producing the evidence for a quota-increase request — `openshift-sysdig` (planned).
- Configuring Red Hat Advanced Cluster Security (ACS) runtime policies, image scanning, or producing a security-assessment artifact — `openshift-acs` (planned). The pod and container `securityContext` *manifest field* itself is in scope here; ACS is the cluster-side enforcement / scanning layer on top of it.
- Exposing a non-HTTPS TCP port via the Porter Operator (`TransportServerClaim`) — `openshift-networking` (planned).
- Designing CI workflows (GitHub Actions chassis, OIDC, fork-gate) — sibling [`github-actions`](../github-actions/SKILL.md) skill.
- Anything on AKS, EKS, or vanilla Kubernetes outside the BC Gov Private Cloud.

## Workflow
1. **Pick the controller from the workload's identity model, not personal habit.** Stateless, fungible replicas behind a Service → `Deployment`. Ordered identity, stable network ID, per-pod PVC, controlled rolling restarts (databases, brokers, leader-elected services) → `StatefulSet`. Exactly one pod per node (node-agent, log shipper, CSI side-car) → `DaemonSet`. Run-to-completion batch work → `Job` (`completions` / `parallelism` for fan-out). Scheduled batch → `CronJob` (subject to the cadence rules in step 8). Avoid `DeploymentConfig` for new work — it's an OpenShift-only legacy resource and the platform's automation treats it identically to `Deployment` anyway.
2. **Set `requests` on every container and pick a deliberate QoS class.** `Guaranteed` (requests == limits, both set for CPU and memory) is what ideal production workloads should run — the scheduler evicts these last under node pressure. `Burstable` (requests < limits, or only request is set) is reasonable for most workloads in BCGov. If you omit a value, the namespace `LimitRange` injects `cpu: 50m` / `memory: 256Mi` requests and a memory limit equal to the namespace memory quota (capped at 16 GiB). Those defaults are conservative cluster-wide guardrails, so override them with values that fit your workload.
3. **Right-size requests using observed data.** Build the manifest with a starting guess, deploy to `-dev`, let it run a representative load (or steady idle for a week), then read CPU/memory P95 and P99 from Sysdig and adjust. Add `VerticalPodAutoscaler` in `updateMode: "Off"` (recommend-only) for long-lived workloads that drift over time; never run VPA in `Auto` mode alongside HPA on the same metric.
4. **Configure liveness, readiness, and (if needed) startup probes; keep them cheap and downstream-independent.** Readiness probe gates Service endpoints: it must go `not ready` when the pod can't serve traffic (warming caches, lost downstream dep, draining) and `ready` when it can. Liveness probe restarts the container: only fail it when the process is genuinely wedged (deadlocked, stuck event loop) — *not* when a downstream is down. Startup probe is for slow-booting apps (legacy JVMs, large model loads): set generous `failureThreshold * periodSeconds` so the liveness probe doesn't restart the pod mid-boot. HTTP probes should be a dedicated `/livez` / `/readyz` endpoint that's cheap and doesn't touch downstreams; `exec` probes work but cost a process exec each interval, so keep them simple.
5. **Make sure PID 1 reaps zombie children.** If your container `ENTRYPOINT` is your app process directly (`["/app/server"]`), it's PID 1 — and most app runtimes don't reap. Defunct children accumulate, the platform sends a defunct-PIDs alert, and the quick fix is `oc delete pod` (the root cause returns next deploy). The proper fix is to set the entrypoint to an init that reaps and forwards signals: `tini` (`ENTRYPOINT ["tini", "--", "/app/server"]`), `dumb-init`, or `s6-overlay`. Distroless and most OCP base images do not include one — add it in your Dockerfile.
6. **Trap SIGTERM, set a sane `terminationGracePeriodSeconds`, and never exceed 600 s.** When a pod is being evicted (rollout, drain, scale-down), Kubernetes sends SIGTERM, waits up to `terminationGracePeriodSeconds`, then sends SIGKILL. Apps must use that window to stop accepting new work (close listeners, deregister from the LB), finish in-flight requests, flush buffers, and exit. Use ≤ 60 s for stateless services, up to 600 s for stateful (the Platform Operations team caps it at 600 s on node drains and force-kills anything still running). Add a `preStop` lifecycle hook if you need a "drain" pause between the LB deregister and the actual shutdown (`["/bin/sleep", "10"]` is a common pattern to let the Service's endpoint propagate).
7. **Always ship a `PodDisruptionBudget` for any multi-replica workload.** Use `maxUnavailable` for stateless (e.g. `maxUnavailable: 1` on a 3-replica Deployment means up to 1 can be down during a drain). Use `minAvailable` for stateful where you need a quorum (e.g. `minAvailable: 2` on a 3-pod Postgres). **Never set `maxUnavailable: 0`** — the Platform Operations team overrides it during node drains, so it only masks the misconfiguration until the next maintenance window.
8. **Use `CronJob` only when it fits the platform's admission rules; otherwise run an in-pod scheduler.** Kyverno admission policies reject: `schedule` more frequent than every 5 minutes; `schedule` more frequent than every 1 hour if the pod mounts a PVC; any `CRON_TZ=` prefix in `schedule` (use `spec.timeZone` instead). For high-frequency cron with a PVC, convert to a long-running `Deployment` with [`go-crond`](https://github.com/webdevops/go-crond) or any in-pod scheduler so you pay the pod-startup cost once. Always set `successfulJobsHistoryLimit` and `failedJobsHistoryLimit` (defaults of 3 and 1 are usually fine — set to 0 only if you have external observability for the runs), `concurrencyPolicy` (`Forbid` is the safe default for tasks that hold a lock), `startingDeadlineSeconds`, and a sensible `activeDeadlineSeconds` on the Job spec to bound stuck runs.
9. **Always ship a `HorizontalPodAutoscaler` with explicit rampup and rampdown windows.** HPA on CPU or memory is the baseline; set `minReplicas ≥ 2` for any production workload (one replica + a node drain = an outage); set `maxReplicas` from your quota (one runaway HPA can exhaust `compute-long-running-quota` and starve siblings). Always set `spec.behavior.scaleUp` (rampup) and `spec.behavior.scaleDown` (rampdown) — never leave them defaulted — so traffic spikes don't get throttled by slow scale-up and so scale-down doesn't thrash mid-rollout (typical baseline: `scaleUp.stabilizationWindowSeconds: 30` with a `Percent: 100 / periodSeconds: 60` policy; `scaleDown.stabilizationWindowSeconds: 300` with a `Percent: 50 / periodSeconds: 60` policy — tune from observed traffic, but never ship without them). Pair every HPA with the PDB from step 7.
10. **Always set pod anti-affinity for any multi-replica workload, and add node affinity when the workload must land on (or avoid) a specific node pool / zone.** Use `podAntiAffinity` with `topologyKey: kubernetes.io/hostname` so replicas don't co-locate on the same node (`preferredDuringSchedulingIgnoredDuringExecution` for soft spread on small clusters; `requiredDuringSchedulingIgnoredDuringExecution` for hard spread when you have node headroom). On Gold/GoldDR add `topologySpreadConstraints` with `topologyKey: topology.kubernetes.io/zone` so replicas survive a zone loss. Use `nodeAffinity` (not hard-coded node names — they rotate) when the workload needs GPU, storage-class-local, or other node-pool-specific scheduling.
11. **Set an explicit `securityContext` on every container so the pod is admitted by the namespace's default `restricted-v2` SCC and stays hardened.** At the container level set `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, and `seccompProfile.type: RuntimeDefault`. `restricted-v2` already enforces most of these, but stating them in the manifest makes intent explicit, survives an SCC change, and is portable. **Do not hard-code `runAsUser`, `runAsGroup`, `fsGroup`, or `supplementalGroups`** — OpenShift assigns a UID/GID from the namespace's allocated range, and a hard-coded value outside that range is rejected by `restricted-v2` (`unable to validate against any security context constraint`). When you set `readOnlyRootFilesystem: true`, mount a writable `emptyDir` for every path the process writes to (commonly `/tmp`, plus any cache/run dirs); use `emptyDir.medium: Memory` with a `sizeLimit` for small scratch space so it counts against the pod's memory, not node disk.

## Rules
- Always set `requests` on every container, and pick a deliberate QoS class. Prefer `Guaranteed` (`requests == limits` for CPU and memory) for workloads where eviction is costly — databases, leader-elected services, anything stateful or otherwise critical. `Burstable` (requests set, CPU limit omitted or memory limit ≥ request) is the practical norm for typical BC Gov application workloads. Avoid `BestEffort` (no requests, no limits) outside opportunistic batch. (Why: a pod without `requests` is `BestEffort`, gives the scheduler nothing to place against, and is the first evicted under node pressure. The namespace `LimitRange` defaults of `cpu: 50m` / `memory: 256Mi` are conservative cluster-wide guardrails, not workload-tuned values. A missing memory limit risks OOM-thrashing siblings; omitting a CPU limit is the BC Gov norm so apps can burst into spare capacity, but the pod becomes throttle-prone whenever a node neighbour is busy.)
- Never run a database — or anything else with persistent state — as a `Deployment` with a single `ReadWriteOnce` PVC. (Why: nodes are not guaranteed up for 24 h, and a Deployment+RWO+rolling-update will deadlock the second pod waiting for the volume the first pod hasn't released. Use a `StatefulSet` with one PVC per replica, or an HA pattern like CrunchyDB / Patroni / Mongo ReplicaSet, three pods minimum.)
- Always set `terminationGracePeriodSeconds` deliberately — ≤ 60 s for stateless services, ≤ 600 s for stateful — and ensure the app traps `SIGTERM` to stop accepting new work, drain in-flight, and exit. (Why: the Platform Operations team caps `terminationGracePeriodSeconds` at 600 s during node drains and force-kills anything still running; ignoring SIGTERM is data-loss waiting to happen on the next drain.)
- Never set `maxUnavailable: 0` on a `PodDisruptionBudget`, and never ship a multi-replica long-running workload without one. (Why: the Platform Operations team overrides `maxUnavailable: 0` during node drains, so it only masks the misconfiguration until the next maintenance window; a missing PDB lets a node drain take all replicas down simultaneously. Set a real number — e.g. `maxUnavailable: 1` on a 3-replica Deployment, or `minAvailable: 2` on a 3-pod StatefulSet — so drains are gracefully orchestrated.)
- Always make readiness and liveness probes cheap and downstream-independent. (Why: a readiness probe that checks a downstream DB will cascade an outage when the DB blips; a liveness probe that fails on transient downstream errors will restart the pod loop and accelerate the failure. Dedicate `/livez` and `/readyz` endpoints that only check in-process health.)
- Always set PID 1 to an init that reaps children (`tini`, `dumb-init`, or `s6-overlay`) unless the application runtime is documented to reap. (Why: defunct `<defunct>` children accumulate when PID 1 doesn't reap, the platform sends an automated alert to the Product Owner and Technical Lead, and `oc delete pod` is only a band-aid — the bug returns until PID 1 is fixed.)
- Never schedule a `CronJob` more often than every 5 minutes (every 1 hour if it mounts a PVC), and never put `CRON_TZ=` in the `schedule` string. (Why: a Kyverno cluster policy denies the `CronJob` create on all three counts. Use `spec.timeZone` for the timezone, reduce the cadence, or switch to a long-running `Deployment` driven by an in-pod scheduler like `go-crond`.)
- Always set `concurrencyPolicy: Forbid` (or `Replace`) on a `CronJob` whose work cannot safely overlap. (Why: the default `Allow` lets a slow run pile up parallel pods on the next tick — for tasks that hold a row lock, mutate shared state, or push to an external system that doesn't dedupe, this is a data-corruption incident, not a performance issue.)
- Always set `minReplicas ≥ 2` for any production workload — Deployment or HPA-driven. (Why: a single-replica workload becomes unavailable the moment its node is drained for patching, which happens routinely. Two replicas + a PDB is the minimum for "stays available across voluntary disruptions".)
- Always ship a `HorizontalPodAutoscaler` for any long-running workload, and always set both `spec.behavior.scaleUp` (rampup) and `spec.behavior.scaleDown` (rampdown) explicitly — never rely on defaults. (Why: HPA defaults are conservative — scale-up can lag a traffic spike by minutes and scale-down can thrash during a rollout, both of which cause user-visible incidents on this platform. Baseline: `scaleUp.stabilizationWindowSeconds: 30` with a `Percent: 100 / periodSeconds: 60` policy; `scaleDown.stabilizationWindowSeconds: 300` with a `Percent: 50 / periodSeconds: 60` policy. Tune from observed traffic, but the keys must be present.)
- Always set pod anti-affinity on every multi-replica workload, and use `nodeAffinity` (never hard-coded node names) when scheduling must target a specific node pool or zone. (Why: without `podAntiAffinity` the scheduler is free to stack all replicas on one node — one node drain or hardware failure then takes the whole workload down, regardless of the PDB. Use `topologyKey: kubernetes.io/hostname` for per-node spread, and on Gold/GoldDR add `topologySpreadConstraints` with `topologyKey: topology.kubernetes.io/zone` so a zone outage doesn't take all replicas. Hard-coded node names rotate and break the manifest the next time the cluster is reconciled.)
- Always scale horizontally (more replicas) rather than vertically (bigger pods), and pair HPA with a PDB. (Why: large single pods exhaust schedulability — a 32 GiB pod requires a node with 32 GiB of unallocated memory, which becomes rare as the cluster fills; horizontal scaling spreads load and tolerates node loss. Without a PDB, HPA's scale-down can take the workload below the budget during a drain.)
- Always set an explicit container `securityContext` with `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, and `seccompProfile.type: RuntimeDefault`. (Why: these are the hardening defaults the platform expects under the `restricted-v2` SCC; declaring them in the manifest makes the posture explicit, portable, and resilient to an SCC change instead of relying on cluster-side defaulting. `readOnlyRootFilesystem: true` also blocks a whole class of tamper-the-binary attacks — pair it with writable `emptyDir` mounts for the paths the app actually writes to.)
- Never hard-code `runAsUser`, `runAsGroup`, `fsGroup`, or `supplementalGroups` in a workload bound for a license-plate namespace. (Why: OpenShift's `restricted-v2` SCC assigns a UID/GID from the namespace's pre-allocated range; a hard-coded value outside that range fails admission with `unable to validate against any security context constraint`, and even a value inside the range will break the next time the range is reassigned. Set `runAsNonRoot: true` and let the platform pick the UID. Build images so they run as an arbitrary non-root UID — group-writable, group `0` — rather than a fixed one.)
- Never set `privileged: true`, add Linux capabilities, or use `hostNetwork` / `hostPID` / `hostPath` on a workload bound for a license-plate namespace. (Why: the `restricted-v2` SCC rejects all of them. If you genuinely believe you need one, that is a platform-team conversation via a Public Cloud Service Request — not a manifest change.)
- Never run `VerticalPodAutoscaler` in `updateMode: "Auto"` on a workload that already has an `HorizontalPodAutoscaler` on the same metric (CPU or memory). (Why: VPA changes requests, HPA reads utilization-vs-requests; the two fight and both can oscillate. Use VPA in `updateMode: "Off"` for recommendations only when an HPA is in play.)
- Always set `successfulJobsHistoryLimit`, `failedJobsHistoryLimit`, `startingDeadlineSeconds`, and an `activeDeadlineSeconds` on the Job template of any `CronJob`. (Why: default unbounded history fills `compute-time-bound-quota` with completed pods and blocks new runs; missing `activeDeadlineSeconds` lets a stuck run consume a worker slot indefinitely.)

## Examples
- **"Standard production HTTP API"** → `Deployment`, `replicas: 3`, container with explicit `requests == limits` for CPU and memory (`Guaranteed` QoS), a hardened container `securityContext` (`allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, `runAsNonRoot: true`, `readOnlyRootFilesystem: true`, `seccompProfile.type: RuntimeDefault`) with a writable `emptyDir` mounted at `/tmp`, `livenessProbe` and `readinessProbe` on dedicated `/livez` and `/readyz` endpoints, `terminationGracePeriodSeconds: 30`, `preStop: { exec: { command: ["/bin/sleep", "10"] } }`, `podAntiAffinity` on `kubernetes.io/hostname`, a mandatory `PodDisruptionBudget` with `maxUnavailable: 1`, and a mandatory `HorizontalPodAutoscaler` with `minReplicas: 3` / `maxReplicas: 10` on CPU utilization 70% **and** explicit `behavior.scaleUp` (`stabilizationWindowSeconds: 30`, `Percent: 100 / periodSeconds: 60`) + `behavior.scaleDown` (`stabilizationWindowSeconds: 300`, `Percent: 50 / periodSeconds: 60`).
- **"Pod rejected with `unable to validate against any security context constraint`"** → the manifest is asking for something `restricted-v2` won't grant. Most often a hard-coded `runAsUser` / `fsGroup` outside the namespace's allocated range, a `privileged` container, an added capability, or `hostNetwork`/`hostPath`. Remove the offending field; set `runAsNonRoot: true` and let OpenShift assign the UID. Confirm the range your namespace allows from its annotations — `oc get project <ns> -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.uid-range}'` (reading the cluster `SecurityContextConstraints` object itself with `oc get scc` is cluster-scoped and normally Forbidden for project users — you don't need it).
- **"HA Postgres for a production service"** → `StatefulSet` of 3 pods (CrunchyDB Operator or Patroni image from `bcgov-docker-local`), `volumeClaimTemplates` for one PVC per pod on `netapp-block-standard`, `PodDisruptionBudget` with `minAvailable: 2`, paired with a [`backup-container`](https://github.com/BCDevOps/backup-container) `CronJob` writing to a `netapp-file-backup` PVC. Not a `Deployment`.
- **"Daily 02:00 cleanup job that mounts the PVC the app uses"** → `CronJob` with `schedule: "0 2 * * *"`, `spec.timeZone: "America/Vancouver"`, `concurrencyPolicy: Forbid`, `successfulJobsHistoryLimit: 3`, `failedJobsHistoryLimit: 1`, `startingDeadlineSeconds: 300`, and `activeDeadlineSeconds: 3600` on the Job template. ≥ 1 h cadence satisfies the Kyverno PVC rule.
- **"Process queue messages every 30 seconds"** → `CronJob` is rejected at admission (< 5 min). Convert to a long-running `Deployment` with `replicas: 2` that polls (or subscribes to) the queue in a loop and shuts down cleanly on `SIGTERM`; size replica count and per-replica concurrency for steady-state throughput. Still ship the mandatory `PodDisruptionBudget` (`maxUnavailable: 1`), `podAntiAffinity` on `kubernetes.io/hostname`, and a CPU-based `HorizontalPodAutoscaler` with `minReplicas: 2` / `maxReplicas: 6` and the standard `behavior.scaleUp` (`stabilizationWindowSeconds: 30`, `Percent: 100 / periodSeconds: 60`) + `behavior.scaleDown` (`stabilizationWindowSeconds: 300`, `Percent: 50 / periodSeconds: 60`) — queue-depth autoscaling is not in scope for this skill, but the CPU-based HPA keeps the mandatory-HPA rule satisfied and protects the workload when poll cost spikes.
- **"Slow-booting Java app keeps getting killed by liveness probe during startup"** → add a `startupProbe` with `failureThreshold: 30, periodSeconds: 10` (5-minute boot budget). Liveness probe only kicks in after startup probe passes once, so the boot is no longer a restart loop.
- **"Defunct PIDs alert email"** → container `ENTRYPOINT` is the app process and the runtime doesn't reap. Switch the entrypoint to `tini` (`ENTRYPOINT ["tini", "--", "/app/server"]`) or `dumb-init`; quick band-aid is `oc delete pod` but the bug returns until PID 1 is fixed.
- **"Pod stuck `OOMKilled` on every restart"** → memory request is below working set. Read Sysdig memory P99 and set `requests.memory` and `limits.memory` to that + headroom (~25%). If working set legitimately spikes, profile for leaks (`pprof`, JVM heap dump, etc.) before throwing more memory at it.
- **"Pod stuck `Pending` with `FailedScheduling` insufficient memory"** → the requested memory is larger than any node's free allocatable. Either scale horizontally (more, smaller pods) or check if the namespace is sitting on a stuck terminating pod holding the request. Use `oc get pods --field-selector=status.phase=Pending` to list the pending pods (the API has no `Terminating` phase — `Pending`, `Running`, `Succeeded`, `Failed`, and `Unknown` are the only valid values), and `oc get pods -o json | jq '.items[] | select(.metadata.deletionTimestamp) | .metadata.name'` to list pods that are mid-terminate (the API signals that with `metadata.deletionTimestamp`, not a phase).

## Edge Cases
- **`Job` completes but the pod sticks around.** That's intentional — `Job` keeps the terminated pod for log access. Tune with `ttlSecondsAfterFinished` on the Job spec (e.g. `86400` to garbage-collect after a day). `CronJob` history limits do this for you.
- **`StatefulSet` rolling update hangs at one pod.** Ordered rollout means the next pod won't start until the current one is `Ready`. Check the failing pod's events / logs; if it's a probe failure, your readiness probe may be too strict or the new image is genuinely broken — rolling back via `oc rollout undo statefulset/<name>` is safe.
- **`DaemonSet` won't schedule on a node.** Likely a taint without a matching toleration, or a node selector / affinity rule. `oc describe pod <ds-pod>` will surface the scheduling failure; add the `toleration` or relax the selector.
- **HPA reports `unknown` for a metric.** The metric server can't read the source — for `Resource` metrics (CPU/memory) the pods must have requests set on that resource (no requests means no utilization-vs-requests math).
- **`Eviction` event with reason `TerminationGracePeriodExceeded`.** Your app didn't exit within `terminationGracePeriodSeconds` after SIGTERM. Either bump the grace period (to the cap of 600 s) or — more often the right fix — fix the app to handle SIGTERM and drain.
- **A `CronJob` accidentally set to `schedule: "*/1 * * * *"` is rejected at admission.** Kyverno denies it. Reduce the cadence (≥ 5 min, ≥ 1 h with a PVC), or convert to a long-running scheduler if you genuinely need higher frequency.
- **Need to pin a workload to a specific node pool / zone.** Use `nodeSelector` or `affinity.nodeAffinity` with the labels published on the cluster's nodes (`oc get nodes --show-labels`). Avoid hard-coding node names — they rotate. For DR / topology spread on Gold/GoldDR use `topologySpreadConstraints` with `topologyKey: topology.kubernetes.io/zone`.
- **Pod stuck `CrashLoopBackOff` for more than a day.** Read the prior run's stdout/stderr with `oc logs <pod> --previous`, push the fix, and roll out (`oc rollout restart deployment/<name>`); if you can't fix it immediately, scale the workload to 0 (`oc scale deployment/<name> --replicas=0`) so it stops consuming a worker slot while you investigate.
- **`StatefulSet` PVC needs to grow.** All BC Gov NetApp storage classes are resize-up-only: edit `.spec.volumeClaimTemplates[*].spec.resources.requests.storage`, then restart the pods (rolling update doesn't reshape existing PVCs — you may need to delete-and-recreate or use the platform's documented expansion procedure). Resize-down is not supported.
- **`readOnlyRootFilesystem: true` makes the app crash with `read-only file system` / `EROFS`.** The process is writing somewhere on the root filesystem. Find the path from the error (commonly `/tmp`, `/var/run`, a framework cache, or a PID/lock file) and mount a writable `emptyDir` there. For small scratch space prefer `emptyDir.medium: Memory` with a `sizeLimit` so it's bounded and counts against the pod's memory rather than node disk. Keep `readOnlyRootFilesystem: true` — adding the writable mount is the fix, turning the flag off is not.
- **App can't bind a privileged port (`< 1024`) under `runAsNonRoot`.** A non-root UID can't bind `:80` / `:443`. Configure the app to listen on a high port (e.g. `:8080`) and let the `Service` map `port: 80 → targetPort: 8080`. Do not reach for `NET_BIND_SERVICE` or root — `restricted-v2` won't grant either.

## References
**Scope of this skill**: container-level and controller-level workload manifests inside an existing license-plate `-dev` / `-test` / `-prod` namespace. Cluster onboarding, network policy, secrets, GitOps, image-registry, storage class choice, monitoring, and security-assessment concerns live in their own sibling skills (see below).

See [references/REFERENCE.md](./references/REFERENCE.md) for: the namespace `LimitRange` and quota objects that shape what your manifests are allowed to request; a worked Deployment / StatefulSet / DaemonSet / CronJob template set; the probe, lifecycle, and `preStop` recipe; the `restricted-v2` SCC `securityContext` recipe (what to set, what never to hard-code, and the writable-`emptyDir` pattern for `readOnlyRootFilesystem`); PDB sizing tables; an HPA and VPA decision matrix; the Kyverno workload-admission policy list (`no-fast-cronjob`, `no-fast-cronjob-with-pvc`, `no-unsupported-timezone`, plus the Emerald-only `DataClass` label rule that affects pod templates); the defunct-PID + PID-1 init container recipe; and a manifest-level error cheat sheet (`OOMKilled`, `Evicted`, `CreateContainerConfigError`, `ImagePullBackOff`, `TerminationGracePeriodExceeded`).

Sibling BC Gov OpenShift skills (each tightly scoped to one concern — **planned**, not yet shipped in this repo; use upstream docs in the interim):
- `openshift-networking` — `NetworkPolicy` starter kit, default-deny, allow-from-router, SDN-vs-NSX-T egress, Porter `TransportServerClaim` for direct TCP exposure.
- `openshift-secrets` — Vault + External Secrets Operator, `SecretStore` and `ExternalSecret` patterns, hyphen-free key gotcha, rotation.
- `openshift-images` — Artifactory remote/local repos, prebuilt platform images, `ArtifactoryServiceAccount` + paired `artifacts-*` Secrets, `npm login --auth-type=legacy`.
- `openshift-gitops` — Argo CD shared-per-cluster, `GitOpsTeam` onboarding, `bcgov-c/tenant-gitops-<licenseplate>` repo layout, sync waves, Helm OCI through Artifactory.
- `openshift-storage` — `netapp-file-standard` / `netapp-file-backup` / `netapp-block-standard`, quota tiers, backup-container pattern, OCIO Object Storage Service.
- `openshift-sysdig` — Sysdig Monitor onboarding, dashboards, alerts, evidence for quota-increase requests.
- `openshift-acs` — Red Hat Advanced Cluster Security policies, image scanning, runtime alerts.

External (use upstream docs, not duplicates here):
- [Kubernetes Workloads](https://kubernetes.io/docs/concepts/workloads/) — controller reference for Deployment / StatefulSet / DaemonSet / Job / CronJob.
- [Kubernetes Configure Quality of Service](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/).
- [Kubernetes Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) — probes, termination, init/sidecar containers.
- [Kubernetes Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) — `securityContext` fields (`runAsNonRoot`, `allowPrivilegeEscalation`, `capabilities`, `readOnlyRootFilesystem`, `seccompProfile`).
- [OpenShift Managing Security Context Constraints](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html) — how `restricted-v2` assigns UID/GID ranges and what it admits.
- [Kubernetes Specifying a Disruption Budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/).
- [Kubernetes HorizontalPodAutoscaler Walkthrough](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) and [VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler).
- [tini](https://github.com/krallin/tini) and [dumb-init](https://github.com/Yelp/dumb-init) — PID-1 init container choices.
- [Red Hat OpenShift Container Platform docs](https://docs.openshift.com/) — for OCP-specific behaviour where it diverges from upstream Kubernetes.