Serving LLMs on EKS after Kubernetes 1.37: DRA, scale-to-zero and inference-aware routing

Most platform teams I talk to are getting the same request: a product team wants to run an open-weight model themselves, for cost, data residency or latency, and they want it on the Kubernetes platform they already have. For a long time the honest answer was that Kubernetes treated a GPU as an opaque integer. You could ask for nvidia.com/gpu: 1, and that was about it. Sharing, health, topology and autoscaling were all bolted on from outside.

That's changing quickly. Kubernetes 1.37 shipped on 26 August, and Amazon EKS added it on 13 October. KubeCon North America this year added a dedicated AI Inference and Agentic track. The building blocks are now upstream, and the question has moved from "can Kubernetes do this?" to "which parts are production-ready on my managed cluster?"

I run EKS platforms day to day, and this year I took our clusters from 1.33 to 1.36 over about two months, so I tend to read release notes for what they mean in a managed cluster where you can't flip control plane feature gates. This post is that reading for GPU inference: a reference design, real manifests, and the gaps you should plan around.

1. What 1.37 changed for inference

Not every headline feature is usable on EKS yet. Some are on by default, some are beta but off by default, and EKS doesn't let you change control plane feature gates. This is the table I'd hand to a team before they start designing:

The DRA updates post and the HPA scale-to-zero post on the Kubernetes blog go into more detail. The short version: device allocation, device health and scale-to-zero are things you can build on today. Gang scheduling still means Kueue or Volcano on EKS.

2. The reference architecture

Here's the layout I'd draw for a team serving a few models on EKS. Requests come in through a Gateway API implementation that supports the Inference Extension. The route points at an InferencePool instead of a Service. An endpoint picker chooses the model server for each request. Model servers run on a dedicated Karpenter NodePool for GPUs, and each pod gets its GPU through a ResourceClaim.

Three decisions in that picture matter more than they look:

• GPU nodes are their own pool, and they're tainted. Nothing lands on a node that costs several dollars an hour unless it asked for a GPU. That one rule stops a lot of waste.
• The route targets an InferencePool, not a Service. A Service balances connections. An inference pool balances work, which for LLMs is very uneven. I'll come back to why in section 5.
• Autoscaling uses demand signals, not GPU utilisation. GPU utilisation from DCGM tells you a kernel is running, not whether users are waiting. Queue depth and time to first token are the signals that match what users feel.

3. GPUs as claims: DRA with Karpenter on EKS

With the classic device plugin, a pod asks for a number of GPUs and the scheduler counts them. With Dynamic Resource Allocation, the driver on each node publishes a ResourceSlice that describes every device: product, memory, architecture, MIG profile. Pods reference a ResourceClaim that asks for devices matching a CEL expression. The scheduler then allocates a specific device that fits.

On EKS you install the driver yourself. AWS documents both the device plugin and the DRA route in its NVIDIA device management guide, and the AI on EKS guidance has a fuller walkthrough. The driver is NVIDIA's k8s-dra-driver-gpu. Karpenter needs to know about DRA too, because it has to work out which instance type will satisfy a claim before the node exists. Karpenter's DRA support is still alpha, behind a Helm setting:

# karpenter Helm values: DRA-aware provisioning (alpha). Use --reset-then-reuse-values on upgrade.
settings:
  enableDRA: true
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference-dra
spec:
  template:
    metadata:
      labels: { platform.example.com/gpu-mode: dra }   # never mix DRA and device-plugin nodes
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: gpu-al2023 }
      taints:
        - { key: nvidia.com/gpu, effect: NoSchedule }
      requirements:
        - { key: karpenter.k8s.aws/instance-family, operator: In, values: [g6e, g6, p5] }
        - { key: karpenter.sh/capacity-type, operator: In, values: [on-demand] }
      expireAfter: 336h
  limits:
    cpu: "768"                              # a hard ceiling on what a bad HPA can spend
  disruption:
    consolidationPolicy: WhenEmpty           # don't evict a loaded model to save one node
    consolidateAfter: 20m

Then the claim. A ResourceClaimTemplate gives each pod its own claim, so each replica gets its own GPU. The selector is CEL against the attributes the driver publishes, so you ask for "a GPU with at least 40 GiB" instead of naming an instance type:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-40g
  namespace: llm-serving
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.nvidia.com
            count: 1
            selectors:
              - cel:
                  expression: "device.capacity['gpu.nvidia.com'].memory.compareTo(quantity('40Gi')) >= 0"
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: qwen3-8b, namespace: llm-serving }
spec:
  replicas: 1
  selector: { matchLabels: { app: qwen3-8b } }
  template:
    metadata: { labels: { app: qwen3-8b } }
    spec:
      tolerations: [{ key: nvidia.com/gpu, operator: Exists, effect: NoSchedule }]
      resourceClaims:
        - { name: gpu, resourceClaimTemplateName: gpu-40g }
      containers:
        - name: vllm
          image: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/vllm-openai:pinned   # mirrored, scanned, pinned
          args: ["--model", "/models/qwen3-8b", "--max-model-len", "16384",
                 "--gpu-memory-utilization", "0.90"]
          ports: [{ containerPort: 8000, name: http }]
          resources:
            claims: [{ name: gpu }]
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 5
          startupProbe:                           # model load can take minutes
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
            failureThreshold: 60

Two things here are worth slowing down for. First, Karpenter's docs are clear that you shouldn't let a device plugin and a DRA driver manage the same GPUs, and that pods asking for nvidia.com/gpu as an extended resource aren't treated as DRA pods by Karpenter. So run DRA on its own NodePool with its own label, and migrate one model at a time. Second, the startup probe is not optional. Without it, a liveness probe will happily kill a pod that's halfway through loading 16 GB of weights, and you get a restart loop that looks like a GPU problem.

Because Karpenter's DRA support is alpha, I wouldn't make it the only path for a critical model yet. A sensible split for 2026: device plugin with whole GPUs for the production model, a DRA NodePool for new and shared workloads, and a date to revisit once Karpenter's support graduates.

4. Sharing a GPU: whole, MIG or time-sliced

The most expensive mistake in GPU platforms is giving a small model a whole large GPU and leaving it 10% busy. The opposite mistake, packing models onto a shared GPU with no isolation, shows up as noisy-neighbour latency that's very hard to debug. The industry has three standard options, and DRA lets you request any of them through the same claim API.

NVIDIA's MIG user guide lists which GPUs and profiles are supported. On AWS, that generally means the A100 and H100 class instances. The L4 and L40S GPUs in the g6 and g6e families don't support MIG, so on those the choice is whole GPU or time-slicing.

With the NVIDIA DRA driver, time-slicing is a property of the claim rather than a node-wide config file. That's a real improvement, because one team's dev namespace can share GPUs while production on the same cluster doesn't:

# A shared claim: every pod that references it shares the same time-sliced GPU
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: dev-shared-gpu
  namespace: team-search-dev
spec:
  devices:
    requests:
      - name: gpu
        exactly: { deviceClassName: gpu.nvidia.com }
    config:
      - requests: ["gpu"]
        opaque:
          driver: gpu.nvidia.com
          parameters:
            apiVersion: resource.nvidia.com/v1beta1
            kind: GpuConfig
            sharing:
              strategy: TimeSlicing
              timeSlicingConfig: { interval: Long }

The way I'd decide for each model is a short tree. The first question is always size, because a model that doesn't fit on one GPU settles most of the other questions for you.

5. Routing that understands the model server

A normal Service spreads connections roughly evenly. That works for web APIs, where requests cost about the same. LLM requests don't. A 50-token prompt and a 20,000-token prompt with a long answer can differ by a factor of a hundred in GPU time. Round-robin sends the next request to a replica that's already full, while the one next to it sits idle. Worse, it ignores the prefix cache: if a replica already holds the KV cache for a long shared system prompt, sending the follow-up there skips a lot of work.

The Gateway API Inference Extension fixes this with two pieces. An InferencePool, now stable at inference.networking.k8s.io/v1, describes a set of model-server pods. An endpoint picker (EPP), called by the gateway through Envoy's ext_proc, chooses a pod for each request using live metrics from the model servers: queue length, KV cache use, prefix cache affinity and which LoRA adapters are loaded.

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: qwen3-8b
  namespace: llm-serving
spec:
  selector:
    matchLabels: { app: qwen3-8b }
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: qwen3-8b-epp
    port: { number: 9002 }
    failureMode: FailClose          # if the picker is down, fail rather than route blindly
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: qwen3-8b
  namespace: llm-serving
spec:
  parentRefs: [{ name: inference-gateway, namespace: gateway-system }]
  rules:
    - matches: [{ path: { type: PathPrefix, value: /v1/ } }]
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: qwen3-8b
      timeouts: { request: 300s }   # long generations are normal

failureMode is a real trade-off, not a default to copy. FailOpen keeps traffic flowing if the picker is unavailable, but routes it without load awareness, which can overload a replica during exactly the kind of spike that took the picker down. FailClose returns errors instead. For an internal assistant I'd pick FailOpen. For anything with paying users and a tight SLO, I'd pick FailClose and run the picker with several replicas and a PodDisruptionBudget, like any other critical part of the data path.

The gateway is also where tenancy belongs. Rate limits per tenant, token budgets and auth live there, not in the model server. The Google SRE chapter on handling overload applies directly: reject early and cheaply at the edge with a clear 429, rather than letting requests queue inside a GPU server until they time out.

6. Scale-to-zero and where the seconds go

HPA scale-to-zero being on by default is the change most teams will try first, because idle GPUs are the biggest line on the bill. The mechanics are simple: minReplicas: 0 is allowed when the HPA uses at least one Object or External metric. That's because a pod metric can't exist when there are no pods. You also need a metrics adapter (Prometheus Adapter, KEDA's metrics server or a cloud adapter), because EKS doesn't install one.

The hard part isn't the HPA. It's the cold start. Scaling a GPU workload from zero isn't like starting a Go service. Here's where the time goes, with illustrative ranges for a mid-sized model. Measure your own, because they vary a lot with model size and instance type.

The fixes are well known by now. Bake the NVIDIA driver into the AMI rather than installing it at boot. Pre-cache the large serving image on a data-volume snapshot, or use lazy loading, so a new node doesn't pull 10 GB before it can do anything. Keep weights out of the image entirely, and stream them from S3 or a shared file system onto local NVMe. vLLM supports streaming loaders for this. And make the readiness check send a tiny real prompt, so "ready" means "can produce a token", not "process started".

Even tuned, a few minutes is far too long for a chat request to wait. So scale-to-zero is a decision per workload, not a platform default:

For interactive models that you still want to scale down sometimes, the standard pattern is a component that holds the request while the replica starts, like the activator in Knative and KServe or the KEDA HTTP add-on interceptor. Native HPA doesn't do that buffering; it only changes the replica count. For the async case, the native HPA is now all you need:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: summariser-batch, namespace: llm-serving }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: summariser-batch }
  minReplicas: 0                          # allowed: at least one External metric below
  maxReplicas: 6
  metrics:
    - type: External
      external:
        metric:
          name: summariser_queue_depth        # exposed by your metrics adapter from the queue
        target:
          type: AverageValue                 # total / target = replicas, works from zero
          averageValue: "50"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies: [{ type: Pods, value: 2, periodSeconds: 60 }]
    scaleDown:
      stabilizationWindowSeconds: 1800       # a cold start costs minutes; don't flap
      policies: [{ type: Pods, value: 1, periodSeconds: 300 }]

Use AverageValue rather than Value for this. With AverageValue, the HPA divides the total metric by the target to get a replica count, which works from zero. A Value target scales the current replica count by a ratio, which can't grow from nothing. And keep scale-down slow. I've run KEDA in production, and the habit carries over: a scale-down window shorter than the cost of a cold start just turns your autoscaler into a flapping, expensive restart loop.

7. Gang scheduling and multi-node models

A model too large for one node, served with tensor and pipeline parallelism across several nodes, has a property ordinary Deployments don't: a partial replica is useless. If three of four pods schedule and the fourth waits for capacity, you've paid for three nodes' worth of GPUs to do nothing. Gang scheduling means all or nothing: place the whole group or none of it.

Kubernetes 1.37 moved the Workload and PodGroup APIs to beta at scheduling.k8s.io/v1beta1, with a gang policy and a minCount. But the GenericWorkload feature gate is off by default, and EKS doesn't let you turn on control plane gates. So on EKS today, native gang scheduling isn't available. The production route is what the ecosystem has used for a while:

• LeaderWorkerSet to describe one replica as a leader pod plus N worker pods, created, scaled and restarted as a unit. vLLM's multi-node serving guides use it.
• Kueue for admission: a replica group is only admitted when its whole quota fits, with topology-aware placement so the group lands close together, plus fair sharing between teams.
• Capacity you've actually reserved. For multi-node GPU replicas, Karpenter can't launch an instance type the region doesn't have spare. On-demand capacity reservations or capacity blocks are how teams make large GPU shapes predictable.

Even upstream, the beta has limits worth knowing: the gang guarantee covers initial placement only, so after an eviction the group can fall below minCount while still running. Queues and quotas are still Kueue's job. When EKS turns the gate on, it'll make Kueue's life simpler, but it won't replace it.

8. Device taints and sick GPUs

GPUs fail in ways CPUs mostly don't: uncorrectable ECC errors, Xid errors, falling off the bus, thermal throttling. The classic symptom is a pod that starts fine, loads its model and then produces errors or garbage, while the node looks healthy to Kubernetes. Before DRA device taints, the only lever was cordoning the whole node, which on an 8-GPU instance throws away seven good GPUs to avoid one bad one.

With device taints at GA in 1.37, you can taint a single device. In practice, a health controller watches DCGM metrics and driver events and creates a DeviceTaintRule for the bad GPU:

apiVersion: resource.k8s.io/v1
kind: DeviceTaintRule
metadata:
  name: ip-10-0-12-34-gpu-3-ecc
spec:
  deviceSelector:
    driver: gpu.nvidia.com
    pool: ip-10-0-12-34.ap-south-1.compute.internal   # the node's ResourceSlice pool
    device: gpu-3
  taint:
    key: platform.example.com/gpu-unhealthy
    value: ecc-uncorrectable
    effect: NoSchedule        # start here; NoExecute also evicts running pods

Start with NoSchedule while you build trust in the detection, then move to NoExecute for the error classes you're sure about. Pair it with a node-level rule: if more than a set number of GPUs on one instance are tainted, drain and replace the node. With Karpenter, deleting the NodeClaim gets you a fresh instance. AWS's Well-Architected Machine Learning Lens makes the same point in more general terms: automate detecting and replacing unhealthy accelerators instead of relying on people to notice.

9. Upgrading GPU clusters to 1.37

My main lesson from taking our EKS clusters from 1.33 to 1.36 is that the control plane is the easy part. EKS upgrades one minor version at a time, so you plan a sequence of hops, and the real work at each hop is everything installed on top: add-ons, controllers, CRDs and the workloads that depend on deprecated APIs. GPU clusters add a few more things to that list:

• Driver, toolkit and DRA driver versions are a matrix. Check that your NVIDIA driver, container toolkit, DRA driver and Kubernetes version are supported together before you start. Upgrade them on a canary NodePool first.
• Karpenter's version decides what it understands. DRA awareness and new instance types arrive in Karpenter releases, so the Karpenter upgrade is part of the cluster upgrade.
• Node rotation means cold starts. Rolling a GPU NodePool restarts every model server. Do it with PodDisruptionBudgets, during low traffic, and with Karpenter disruption budgets so it rolls one node at a time.
• Check kube-proxy mode. IPVS is deprecated in 1.37. If any cluster still uses it, plan the move now rather than at the version that removes it.
• Metrics adapters and KEDA. Scale-to-zero depends on them, and so does your SLO alerting, so confirm their support for 1.37 before the hop.

10. How inference platforms fail

Most of these will look familiar from any platform. The GPU just makes each one more expensive.

The last row is one I care about because I've done the image side of it: at work we moved our images to ECR with scan-on-push, and the same discipline matters even more when a single serving image is 10 GB and runs on your most expensive nodes.

11. A rollout plan and checklist

You don't need all of this on day one. This is the order I'd follow for a platform team taking on its first self-hosted models.

1. Weeks 1 to 2: one model, boring setup. A tainted GPU NodePool, the NVIDIA device plugin, one vLLM Deployment with whole GPUs, startup and readiness probes, and DCGM plus vLLM metrics in Prometheus. Measure the cold start step by step.
2. Weeks 3 to 4: routing and SLOs. Put a Gateway API implementation with the Inference Extension in front, switch the route to an InferencePool, and define SLOs for time to first token and error rate, with burn-rate alerts like any other service.
3. Weeks 5 to 6: cold start work. Driver in the AMI, cached images, weights streamed from S3. Re-measure. Only now decide which workloads can scale to zero.
4. Weeks 7 to 8: DRA on a second NodePool. Enable DRA in Karpenter, install the DRA driver on a separate, labelled pool, and move dev and batch workloads first. Add device health and taints.
5. Later: multi-node and sharing. LeaderWorkerSet and Kueue for anything that spans nodes, MIG where the hardware supports it, and a review of Karpenter's DRA status each quarter.

And the checklist I'd use in a design review:

1. GPU nodes are on their own tainted NodePool, with a CPU or cost ceiling in limits.
2. DRA and device plugin never manage the same GPUs; DRA pools are labelled and separate.
3. Every model server has a startup probe sized to its worst-case load time.
4. Routes target an InferencePool, and failureMode was chosen on purpose.
5. Rate limits and token budgets per tenant are enforced at the gateway.
6. Interactive models keep at least one warm replica per AZ; only queue-backed workloads scale to zero.
7. Scale-down windows are longer than a measured cold start.
8. Cold start is measured per step, and the driver, image and weights steps have been tuned.
9. Multi-node replicas use LeaderWorkerSet with Kueue admission, never plain Deployments.
10. DCGM health checks can taint a single GPU, and a node with too many tainted GPUs is replaced.
11. Serving images are pinned, mirrored to ECR and scanned; weights are versioned.
12. The upgrade plan covers drivers, the DRA driver, Karpenter and metrics adapters, not just the control plane.

The short version

Kubernetes 1.37 makes GPU inference feel much more like the rest of the platform. Ask for GPUs with claims, keep sick devices out with taints, route with a picker that understands the model server, and scale to zero where a queue can absorb the wait. On EKS, be honest about what's still early: Karpenter's DRA support is alpha and native gang scheduling isn't switched on, so lean on the device plugin, LeaderWorkerSet and Kueue where it matters. Most of the work is the same platform discipline as always: dedicated pools, careful probes, slow scale-down, measured cold starts and an upgrade plan that covers everything above the control plane.