gke-inference

---
name: gke-inference
description: >-
license: Apache-2.0
---

# GKE AI/ML Inference

This reference covers deploying AI/ML inference workloads on GKE using Google's
Inference Quickstart (GIQ) and best practices for LLM serving.

> **MCP Tools:** `apply_k8s_manifest`, `get_k8s_resource`, `get_k8s_logs`,
> `get_k8s_rollout_status`, `describe_k8s_resource`, `list_k8s_events`.
> **CLI-only:** `gcloud container ai profiles *`

## When to Use

-   Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
-   Generate optimized Kubernetes manifests for inference
-   Select GPU/TPU accelerators for model serving
-   Configure autoscaling for LLM inference

## Prerequisites

-   A golden path GKE Autopilot cluster (GPU workloads are supported via
    ComputeClasses and NAP)
-   `gcloud` CLI authenticated
-   Sufficient GPU/TPU quota in the target region

## Workflow

### 1. Discovery: Find Models and Hardware

```bash
# List all supported models
gcloud container ai profiles models list --quiet

# Find valid accelerator/server combinations for a model
gcloud container ai profiles list --model=<MODEL_NAME> --quiet

# Example: what can run Gemma 2 9B?
gcloud container ai profiles list --model=gemma-2-9b-it --quiet
```

### 2. Generate Manifest

```bash
gcloud container ai profiles manifests create \
  --model=<MODEL_NAME> \
  --model-server=<SERVER> \
  --accelerator-type=<ACCELERATOR> \
  --target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml
```

**Parameters:**

-   `--model`: Model ID (e.g., `gemma-2-9b-it`, `llama-3-8b`)
-   `--model-server`: Inference server (`vllm`, `tgi`, `triton`, `tensorrt-llm`)
-   `--accelerator-type`: GPU/TPU type (`nvidia-l4`, `nvidia-tesla-a100`,
    `nvidia-h100-80gb`)
-   `--target-ntpot-milliseconds`: Target Normalized Time Per Output Token
    (optional, for latency optimization)

**Example:**

```bash
gcloud container ai profiles manifests create \
  --model=gemma-2-9b-it \
  --model-server=vllm \
  --accelerator-type=nvidia-l4 \
  --target-ntpot-milliseconds=50 --quiet > inference.yaml
```

### 3. Review and Deploy

```bash
# Review for placeholders (HF tokens, PVCs)
cat inference.yaml

# Deploy
kubectl apply -f inference.yaml

# Monitor
kubectl get pods -w
kubectl logs -f <POD_NAME>
```

> Some models require Hugging Face tokens. Create a Kubernetes Secret and
> reference it in the manifest.

## GPU ComputeClass for Inference

For Autopilot clusters, create a ComputeClass to target GPU nodes:

```yaml
apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: l4-inference
spec:
  priorities:
  - machineFamily: g2
    gpu:
      type: nvidia-l4
      count: 1
    minCores: 4
    minMemoryGb: 16
```

## Accelerator Selection Guide

| Accelerator         | Best For                 | Memory      | Relative Cost |
| ------------------- | ------------------------ | ----------- | ------------- |
| NVIDIA T4           | Budget inference,        | 16 GB       | Lowest        |
:                     : lightweight legacy       :             :               :
:                     : models                   :             :               :
| NVIDIA L4 (G2)      | Small-medium model       | 24 GB       | Low           |
:                     : inference, video,        :             :               :
:                     : graphics                 :             :               :
| NVIDIA RTX PRO 6000 | Multimodal AI,           | 96 GB       | Medium        |
: (G4)                : high-fidelity 3D,        :             :               :
:                     : fine-tuning              :             :               :
| Cloud TPU v5e       | Cost-effective           | Varies      | Medium        |
:                     : transformer inference    :             :               :
| Cloud TPU v5p       | High-performance         | Varies      | High          |
:                     : training                 :             :               :
| Cloud TPU v6e       | High-efficiency next-gen | 32 GB/chip  | Medium-High   |
: (Trillium)          : training & serving       :             :               :
| Cloud TPU v7x       | Ultra-scale inference &  | 192 GB/chip | High          |
: (Ironwood)          : agentic workflows        :             :               :
| NVIDIA A100         | Large model inference,   | 40/80 GB    | High          |
:                     : enterprise ML            :             :               :
| NVIDIA H100 / H200  | Frontier model training, | 80/141 GB   | Highest       |
:                     : high throughput          :             :               :
| NVIDIA B200 (A4)    | Blackwell-scale          | 192 GB      | Highest       |
:                     : training, FP4 precision  :             :               :
| NVIDIA GB200 (A4X)  | Rack-scale AI (Grace     | Massive     | Highest       |
:                     : Blackwell Superchip)     :             :               :

## Autoscaling LLM Inference

### GPU-based autoscaling

Use custom metrics for GPU utilization:

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-server
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_duty_cycle
      target:
        type: AverageValue
        averageValue: "80"
```

### Best practices for inference autoscaling

1.  **Use DCGM metrics**: Golden path enables DCGM monitoring for GPU
    utilization metrics
2.  **Set appropriate minReplicas**: At least 1 for always-on serving; 0 for
    batch/on-demand
3.  **Tune scale-down delay**: LLM model loading is slow; use longer
    stabilization windows
4.  **Consider queue depth**: Scale on pending requests rather than pure GPU
    utilization for latency-sensitive workloads

## Optimization Tips

-   **Quantization**: Use quantized models (GPTQ, AWQ) to reduce GPU memory and
    increase throughput
-   **Batching**: Configure model server batch size for throughput vs latency
    trade-off
-   **Tensor parallelism**: Split large models across multiple GPUs within a
    node
-   **KV cache optimization**: Tune `--gpu-memory-utilization` in vLLM for KV
    cache allocation

## Troubleshooting

| Issue              | Cause                    | Fix                         |
| ------------------ | ------------------------ | --------------------------- |
| Invalid            | Unsupported tuple        | Re-run `gcloud container ai |
: model/accelerator  :                          : profiles list               :
: combination        :                          : --model=<MODEL>`            :
| GPU quota exceeded | Regional quota limit     | Request quota increase or   |
:                    :                          : try a different region      :
| OOM on GPU         | Model too large for      | Use larger GPU, enable      |
:                    : accelerator              : quantization, or use tensor :
:                    :                          : parallelism                 :
| Slow cold start    | Large model loading from | Use local SSD for model     |
:                    : registry                 : caching; pre-pull images    :