Does TPU 8i beat the Ironwood capacity you can rent today for serving models? Every platform team on Google Cloud is going to get that question from finance, and Google just handed them four new chips to sort through. TPU 8t and TPU 8i launched in April, and on November 9 Google added TPU 8tx and TPU 8i Max. None of the four has a price yet, and the Google Cloud TPU page still marks all of them as coming soon. Ironwood, meanwhile, rents for $12.00 per chip-hour on demand in us-central1, according to the pricing page.
The launch numbers make the answer look easy. Google says a TPU 8t superpod delivers nearly 3x the compute of an Ironwood pod, and the math behind it is 121 exaflops against 42.5, or 2.85x. The problem is precision, because the 121 is FP4 and the 42.5 is FP8. Google says FP4 doubles MXU throughput on TPU 8t, so if FP8 runs at half the FP4 rate, a TPU 8t superpod comes in near 60.5 exaflops of FP8. A 60.5-exaflop superpod is about 1.4x an Ironwood pod, and Google hasn’t published FP8 peaks to confirm or correct that estimate.
So the headline shrinks from 3x to 1.4x as soon as you match precisions. For serving, even the 1.4x sends you to the wrong chip, because decode barely touches peak compute.
Why Doesn’t the Biggest FP4 Number Decode Fastest?
Every token a model generates means reading the weights, plus the KV cache, out of HBM. At small batch sizes the chip finishes the math long before the next bytes show up, so memory bandwidth sets the pace. Google lists TPU 8t at 6,528 GB/s of HBM bandwidth and TPU 8i at 8,601 GB/s in its architecture post, and the Ironwood docs list 7,380 GB/s. The new training chip has about 12 percent less bandwidth than the chip it replaces.
Now put a 72B-class model at FP8 on each chip, roughly 73 GB of weights, and run it at batch size 1. Every weight crosses the HBM bus once per token, so TPU 8i needs at least 8.5 ms per token, Ironwood needs 9.9 ms and TPU 8t needs 11.2 ms. In tokens per second, you’re looking at about 118, 101 and 89.
The chip with the biggest FP4 number in the family is the slowest single-stream decoder of the three. All three figures are ceilings for one chip, and KV reads, sharding and imperfect overlap will pull real numbers lower, but the ranking comes straight from bandwidth. Divide peak FLOPs by bandwidth and TPU 8t needs about 1,930 FLOPs per byte read before compute becomes the limit, against about 625 for Ironwood at FP8. Small-batch decode runs far below both, so on a serving box most of that FP4 goes unused.
The capacity numbers point the same way. TPU 8i has 288 GB of HBM per chip against 216 GB on TPU 8t, and the Ironwood docs list 192 GiB, which is about 206 GB. Take Qwen2.5-72B, whose BF16 KV cache needs 327,680 bytes per token, and a TPU 8i chip has room for about 656,000 tokens of context after the 73 GB of weights. Google also talks up the 384 MB of SRAM on TPU 8i as a place to keep KV caches on silicon, but for that same model 384 MB fits about 1,170 tokens. I’d plan context around HBM and think of the SRAM as a fast scratchpad.
Which TPU 8 Chip Fits Which Workload?
Once you sort by bandwidth and capacity, the four chips line up behind four different jobs, and the FP4 ranking stops being the useful one.
TPU 8i is the serving chip, since it has the most HBM bandwidth and capacity per chip, and Google pairs it with a new network called Boardfly that cuts the worst-case hop count across a 1,024-chip pod from 16 to seven. MoE models with experts spread across chips pass tokens between them on every MoE layer, so fewer hops should show up as lower decode latency, though Google hasn’t published p99 numbers under load. Google also claims 80 percent better inference performance per dollar than Ironwood, which nobody can check until there’s a dollar figure.
TPU 8i Max is for long-context work that’s starved for bandwidth. Google puts two TPU 8i chips in one package and says that doubles compute, SRAM and HBM bandwidth, which works out to 17,202 GB/s. Capacity doesn’t double with it, because Google Cloud VP Mark Lohmeyer told Data Center Knowledge the package has 384 GB. If he’s right, that’s 192 GB per chip against 288 GB on plain TPU 8i. So if your long-context jobs run out of HBM before they run out of bandwidth, plain TPU 8i may be the better buy.
TPU 8t is the training chip, and training is where FP4 pays off for models that keep their accuracy at 4-bit weights. It also keeps SparseCore, which TPU 8i drops, so embedding-heavy recommendation models belong on it too. If you train in BF16 or FP8, plan around the 1.4x estimate and treat the 2.7x performance-per-dollar claim the way you’d treat any per-dollar number with no dollar attached.
TPU 8tx is for teams that can’t leave model weights or customer data unprotected in memory. It runs TPU 8t silicon with AMD EPYC hosts on the Boardfly network, and Google says it keeps data and weights protected while in use. Any security team will want to see the attestation flow before 8tx goes into a design, and Google hasn’t published one yet.
What to Benchmark on Ironwood Before TPU 8 Prices Arrive
You don’t need TPU 8 hardware to get ready for the pricing conversation. Google says JAX and PyTorch code from Ironwood runs on TPU 8, and TorchTPU went open source on October 20 with vLLM and SGLang backends, so the serving stack you test on Ironwood is the one you’d move. If you already route traffic with llm-d, Google says one llm-d endpoint on GKE can route to both GPUs and TPUs, which makes a side-by-side test cheap to set up.
Run your own model on Ironwood at your p95 TTFT target and record output tokens per second per chip, because cost per token can swing a lot even on identical hardware, as we saw with TensorBench. On bandwidth alone, TPU 8i should come in about 1.17x ahead of that baseline (8,601 against 7,380 GB/s). When Google posts a TPU 8i chip-hour price, compare it with the $12.00 Ironwood rate. Up to about 17 percent more per hour, TPU 8i should pay for itself on bandwidth-bound decode before Boardfly helps at all, and past that point I’d want TPU 8i numbers on my own traffic before signing anything.
