JetBrains is giving away a version of its Junie coding agent, and the company credits a CNCF sandbox project with helping it pay for that. Vladislav Tankov, director of AI at JetBrains, says llm-d now runs in production behind the new Junie Lite plan and “serves a significant share of its traffic,” letting his team “serve more requests per GPU at lower cost and with faster responses.” The llm-d community shipped version 1.0 on October 20 at PyTorch Conference in San Jose, roughly seven months after the project joined the CNCF as a sandbox project and about a year and a half after Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA founded it.
The timing is what grabbed me. Google open-sourced TorchTPU that same day, and Amazon’s TorchNeuron backend for Trainium went public on GitHub alongside it, so two more accelerator families became ordinary PyTorch devices in a single week. A device string gets a model onto a chip, but it says nothing about which replica in a busy fleet should answer the next turn from an agent that has been working all afternoon. That decision happens one layer up, in the Kubernetes router llm-d builds, and 1.0 is the release its maintainers say is ready for “Day 2.”
The Router Picks the Replica Before the Chip Does Any Work
llm-d sits in front of model servers such as vLLM and SGLang as an Endpoint Picker that plugs into the Kubernetes Gateway API Inference Extension. A request arrives at the gateway, the picker scores every pod in the model’s inference pool, and the request goes to whichever replica should answer fastest. Since August the default scoring has followed a rule the maintainers call sticky until saturated, which keeps a request on the endpoint that already has its prompt prefix in KV cache until that endpoint crosses a calibrated load limit, then routes on load alone. The project reports two to three times the throughput of round-robin Kubernetes Service routing on prefill-heavy workloads.
Google’s production numbers explain why the hyperscalers care. In the maintainers’ 1.0 post, Google says its GKE Inference Gateway is built on the llm-d Endpoint Picker and cut time to first token by more than 35% in Vertex AI production while doubling the prefix-cache hit rate from 35% to 70%. Those gains come from skipping prefill work the cluster already paid for, the same KV cache economics our team dug into when Google published TurboQuant in March.
Tenants Get Their Own Lines When the Fleet Fills Up
Flow control is the multi-tenant feature I would read first. Every request gets a flow key built from a fairness ID, usually a tenant or workload, plus a priority band, and each flow gets its own queue. The router dispatches by priority first and then cycles fairly across tenants inside a band, and under saturation it keeps requests waiting in the router instead of piling them onto model servers that are already behind. A probabilistic admission controller sheds load when demand outruns capacity, and the defaults still behave like first come, first served until you write a flowControl policy.
Mistral’s inference engineers say they use it to enforce “strict priority bands across our commercial tiers,” shedding lower-priority requests when pools fill so paid traffic meets its latency targets. The rest of the release goes after upgrade pain. The Endpoint Picker now runs with a hot standby that takes over if the active instance fails or gets replaced during a rollout. A DisaggregatedSet manages split prefill and decode replicas as one unit, upgrading them in lockstep and routing to a pair only after both sides run the same revision, and the router configuration API graduated to v1, which is the signal platform teams wait for before they template anything.
Agents Keep Landing on the Server That Remembers Them
Agent traffic is where 1.0 earns the version number. A session-state plugin tags each request, and the router sends follow-up turns back to the replica that already has that conversation’s KV cache, until that replica fills up and the session rebalances. Turn-priority scheduling ranks queued requests by how deep their conversation already is, since a long session’s prefix is probably still resident and cheap to serve. Native Responses API support, including previous_response_id, lets an agent continue a conversation without resending its whole history for the stack to recompute.
IBM Research shows what that looks like at scale, serving up to 3,000 concurrent coding agents on 544 NVIDIA H100s with a prefix-cache hit rate around 98% and a blended cost near $0.11 per million tokens, which IBM puts at about 11.5 times cheaper than an all-proprietary setup. Read the maturity labels before you copy that design. The turn-priority guide is marked experimental and its strategy is Beta, it needs one fairness ID per session, and the stateful Responses API guide is experimental and validated only on NVIDIA GPUs. The flow-control guide also admits the load harness needed to measure QoS isolation between tenants has not shipped in llm-d’s benchmark tooling yet.
TorchTPU Already Has a Seat, Trainium Is Still Waiting for One
Every TPU overlay in llm-d’s TPU guides runs on vllm-torchtpu, the serving package built on the TorchTPU backend Google open-sourced on Tuesday, and the 1.0 release notes list that image as new. On an agentic coding workload running on TPU 7x, the maintainers report that prefix-aware routing plus host-memory KV cache offload cut median time to first token by more than 85% and raised total throughput 2.4 times compared with a plain Kubernetes Service.
Amazon’s llm-d story is GPUs for now. The maintainers say AWS runs disaggregated inference on SageMaker HyperPod with llm-d, moving KV cache over Elastic Fabric Adapter, and reports up to 70% more tokens per second than standard vLLM. When I searched the llm-d repository this morning, I found no Trainium or Neuron guide anywhere, so TorchNeuron needs someone to pair the vLLM Neuron plugin with a validated deployment before llm-d routes Trainium the way it routes TPUs. DigitalOcean shows the payoff once that work exists, serving the trillion-parameter Kimi K3 across AMD and NVIDIA hardware behind one llm-d stack.
Replay Your Noisiest Agent Before You Trust a Multiplier
JetBrains is handing out a free coding agent and trusting llm-d to keep the GPU bill sane. That is a loud vote for a 1.0 release, and it is why this one stuck with our team at TechForward. We have not run llm-d ourselves, and every cost and latency multiple above belongs to the vendor or lab that reported it.
Start with the v1.0 release notes and the maintainers’ Day 2 post, then stand up the optimized baseline guide and replay your own agent traffic through it before you believe anyone’s multiplier. If TPUs are on your roadmap, Google’s TorchTPU writeup explains the backend underneath llm-d’s TPU guides, and the CNCF AI conformance program we covered last November is the other half of keeping these clusters portable. Red Hat’s launch announcement has the full roster of adopter quotes.
