gke-reliability

---
name: gke-reliability
description: >-
license: Apache-2.0
---

# GKE Reliability

This reference covers high availability and reliability configuration for GKE
clusters and workloads.

> **MCP Tools:** `get_cluster`, `get_k8s_resource`, `describe_k8s_resource`,
> `apply_k8s_manifest`, `list_k8s_events`

## Golden Path Reliability Defaults

| Setting          | Golden Path Value     | Notes                            |
| ---------------- | --------------------- | -------------------------------- |
| Cluster type     | Regional (4 zones:    | Control plane replicated across  |
:                  : us-central1-a/b/c/f)  : zones                            :
| Upgrade strategy | SURGE (`maxSurge: 1`) | Rolling upgrades with extra      |
:                  :                       : capacity                         :
| Auto-repair      | `true`                | Unhealthy nodes replaced         |
:                  :                       : automatically                    :
| Auto-upgrade     | `true`                | Nodes follow control plane       |
:                  :                       : version                          :
| Release channel  | REGULAR               | Balanced freshness and stability |
| Stateful HA      | Enabled               | Leader election for stateful     |
:                  :                       : workloads                        :

## Workflows

### 1. Verify Cluster High Availability

```
# MCP (preferred)
get_cluster(name="projects/<PROJECT>/locations/<REGION>/clusters/<CLUSTER>",
  readMask="location,locations,nodePools.locations")

# gcloud fallback
gcloud container clusters describe <CLUSTER> --region <REGION> \
  --format="json(location, locations)" \
  --quiet
```

-   If `location` is a region (e.g., `us-central1`), the control plane is
    regional
-   If `locations` has multiple entries, nodes span multiple zones

### 2. Pod Disruption Budgets (PDBs)

PDBs ensure minimum pod availability during voluntary disruptions (node
upgrades, autoscaler scale-down).

**Check existing PDBs:**

```
# MCP (preferred)
get_k8s_resource(parent="...", resourceType="poddisruptionbudget")

# kubectl fallback
kubectl get pdb --all-namespaces
```

**Create PDB:**

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  minAvailable: 2       # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app
```

> Every production Deployment with 2+ replicas should have a PDB.

### 3. Health Probes

Every production container should have liveness and readiness probes. Startup
probes are recommended for slow-starting apps.

**Check existing probes:**

```
# MCP (preferred)
describe_k8s_resource(parent="...", resourceType="deployment", name="<APP>", namespace="<NS>")

# kubectl fallback
kubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"
```

**Recommended probe configuration:**

```yaml
spec:
  containers:
  - name: app
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    startupProbe:             # For slow-starting apps
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 30    # 30 * 5s = 150s max startup time
```

-   **Readiness**: Determines when a pod can accept traffic
-   **Liveness**: Determines when to restart a container
-   **Startup**: Disables liveness/readiness until the app is ready (prevents
    premature restarts)

### 4. Graceful Shutdown

Ensure applications handle `SIGTERM` and drain in-flight requests:

```yaml
spec:
  terminationGracePeriodSeconds: 30    # Default; increase for long-running requests
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]  # Allow LB to deregister
```

### 5. Topology Spread Constraints

Distribute pods across zones and nodes to survive failures:

```yaml
spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app
```

-   **Zone spread** (`DoNotSchedule`): Hard requirement -- pods must be balanced
    across zones
-   **Node spread** (`ScheduleAnyway`): Best-effort -- prefer distribution but
    don't block scheduling

### 6. Replicas

| Workload Type        | Minimum Replicas     | Reason                         |
| -------------------- | -------------------- | ------------------------------ |
| Stateless web/API    | 2                    | Survive single pod/node        |
:                      :                      : failure                        :
| Critical services    | 3                    | Survive zone failure with zone |
:                      :                      : spread                         :
| Stateful (databases) | 3 (with replication) | Application-level quorum       |
| Batch/jobs           | 1                    | Ephemeral by nature            |

## Best Practices & Production Guidelines

1.  **Regional clusters for production**: Always use regional clusters to
    survive zone failures.
2.  **PDBs for everything**: Every production workload with 2+ replicas needs a
    PodDisruptionBudget (PDB) to protect against voluntary disruptions.
3.  **Probes with Explicit Timeouts**: Every production container must have both
    liveness and readiness probes defined. **Always explicitly define
    `initialDelaySeconds`, `periodSeconds`, and `timeoutSeconds`** for all
    probes. Never rely on the Kubernetes default timeout of 1 second if your
    application requires more, but always set a strict limit to prevent hanging
    connections.
4.  **Zone spreading**: Use topology spread constraints to distribute pods
    across failure domains (zones and nodes).
5.  **Graceful shutdown**: Handle `SIGTERM` and set appropriate
    `terminationGracePeriodSeconds` with a `preStop` sleep hook to allow load
    balancer deregistration.
6.  **Maintenance windows**: Schedule upgrades during low-traffic periods (see
    the `gke-upgrades` skill).