Two GPUs with the same model name ran the same inference workload, and the faster one put out almost 15% more tokens per second. At the same hourly rate, that makes every token from the slower card nearly 15% more expensive, and nothing on the spec sheet or the rental invoice would have told you. Tensor Machines published that finding on October 7 alongside TensorBench, an Apache 2.0 benchmark meant to tie GPU power, heat and throughput to cost per token, and founder and CEO Muneeb Rasool was on the NeoCloud Summit agenda in San Francisco the next morning to present it.
The vLLM team posted the other half of the problem the same day. Its DeepSeek-V4.1-Flash writeup reports 5.3 times more throughput per chip on the same GB300 NVL72 hardware, within three weeks of the model’s release, from software changes alone. Read those two posts together and the price of a token depends on which card you drew, how hard you push it, what serving stack runs on it, and the latency you promised your users.
Same Model Name, Different Token Bill
Power Usage Effectiveness tells you how much energy the facility burns on overhead, and an hourly rate tells you what the card costs, but neither says how much AI output comes out the other end, which is the hole Rasool is aiming at. “The economic value of a GPU comes from the useful work it delivers,” Rasool said in the announcement. According to the company, he previously led product work on Meta Reality Labs’ Presence Platform, while co-founder and CTO Mohamed Abdalla leads the assessment infrastructure and models.
Tensor Machines reports three gaps from running its benchmark on several classes of NVIDIA GPUs. Beyond the 15% tokens-per-second spread, a GPU set to a higher power cap drew about 36% more power than its peers for 21% more FP16 throughput, which by our arithmetic works out to roughly 11% less FP16 work per watt. A group of cards that ran about 12% behind its peers on one inference workload trailed by only about 1% on another, so a ranking built on one workload would misjudge that group on the other.
Those are company numbers. The release does not name the GPU models, the number of cards, or the run lengths behind them, and it says the findings apply only to the workloads and conditions tested. The 15% figure compares the fastest result with the slowest, so the spread across a typical rack is still undisclosed, and our team at TechForward has not run TensorBench to check any of it.
Your Power Contract Belongs in the Score
The TensorBench README describes a single ./run.sh that installs the NVIDIA driver, Docker, the NVIDIA Container Toolkit and ipmitool on Ubuntu 22.04, then runs about 90 minutes of matrix math, memory bandwidth, Llama inference and fine-tuning, FFT and NCCL phases with cooldowns between them. The run logs GPU telemetry through NVML and server telemetry from the BMC over IPMI for the whole run, which is how power and heat land next to throughput. Setup formats a spare drive of at least 500 GB and warns that everything on it is gone, so do not point it at a box you care about.
The score is an estimated dollar cost per million tokens from the Llama inference phase, combining measured throughput and power with four inputs you can override, covering ownership or rental cost per GPU-hour, reserved power capacity, electricity per kWh and utilization. Every value lands in a score.json file, and that file is the part I would hand a finance team, because anyone can swap in their own power contract and recompute the number.
Supported hardware is narrow for now. The README lists V100, H100 SXM5 and H200 GPUs on DGX-1, DGX-2, a Supermicro HGX server and DigitalOcean GPU Droplets, with no Blackwell or AMD profiles. New hardware arrives as a pull request with profiles plus the summary and score files from a complete run.
The Code Landed, and It Pins vLLM 0.5.4
At 9 a.m. PT on October 8 the repo held only the README and the license. The benchmark itself arrived in a single “Initial public release” commit at 9:51 a.m. PT that day, a little over an hour before the 11:05 keynote slot, with run.sh, the stress suite, the telemetry loggers and the scoring code, and as of October 22 nothing has been committed since. The formula in score.py matches the README, charging electricity above idle per token and spreading idle power, ownership and reserved capacity across utilization.
The scored phase runs Llama 3 8B in INT8 at batch 16, with 1,024 input tokens and 128 output tokens per prompt, and scores the last six minutes of a seven-minute run. It starts one vLLM instance per GPU instead of a tensor-parallel group, which the code comments explain is how each card gets graded on its own, and it pins vLLM 0.5.4, an August 2024 release, because it is the last one with a working wheel for the V100’s Volta architecture. Per-prompt time to first token and inter-token latency land in a CSV, but score.py prices tokens from raw throughput, so the useful-token accounting in the announcement is still a roadmap item.
Tensor Machines raised a $1.5 million pre-seed led by Omni VC, and its proprietary physics-based models sit in private beta with unnamed bare-metal and neocloud partners, so the open benchmark is the front door to a closed product. The README offers a comparison against other nodes if you run with --upload, which sends the whole assessment folder to Tensor Machines, anonymously unless you add a contact email. Cato Digital, whose Dean Nelson shared the keynote slot on the NeoCloud Summit agenda, appears under “Working with” on the Tensor Machines site.
vLLM Just Moved the Same Silicon 5.3x
The vLLM chart, sourced from SemiAnalysis InferenceX, pins interactivity at about 150 tokens per second per user and shows throughput per chip rising from 27,000 to 144,000 tokens per second between September 11 and October 2 configurations. vLLM credits the jump to bounded replay of the sliding-window cache, which reruns only the last 128 tokens on a cache hit, plus DeepSeek’s newly released kernels and fused all-reduce work. Nothing about the GB300 changed, and a cost-per-token figure frozen on September 11 would now be off by a factor of five at that latency target. TensorBench freezes its serving stack at a 2024 vLLM release on purpose, which keeps scores comparable across V100s and H200s and also means its dollar figure says nothing about what a current engine would get from the same card.
The same chart shows how much the latency target moves the answer. The optimized curve tops 400,000 tokens per second per chip near 45 tokens per second per user and falls under 10,000 at its fastest interactivity point of 599 tokens per second per user. AgentX, the benchmark behind those numbers, reports the highest throughput a configuration sustains while P90 interactivity stays above a floor, and vLLM’s September post on AgentX used 50 tokens per second per user. That SLO gate is what TensorBench promises and AgentX already enforces, though AgentX has no idea whether your particular card is the slow one.
When I wrote about the token economy last year, the worry was API bills that grow with every call. Teams running their own fleets have the mirror problem, a fixed GPU bill whose output per dollar drifts with silicon, power settings and software releases that nobody reprices.
Find Out Which Card You Drew
Fifteen percent between two cards with the same name adds up fast on a fleet billed by the GPU-hour. If the gap survives independent runs, it belongs in neocloud contract talks and gives you a reason to test nodes before you sign a long reservation. That number is why this launch stuck with our team at TechForward.
The variance figures are Tensor Machines’ own, the scored phase runs a 2024 serving stack, and the SLO-gated score is still a plan. After GTC this spring I wrote that the stories that move this industry will come from enterprises showing how they run things. TensorBench earns that role once people outside Tensor Machines publish runs.
Read the Tensor Machines announcement, clone the TensorBench repo and read score.py before you trust a leaderboard, and pull up vLLM’s DeepSeek-V4.1-Flash post next to the AgentX harness. Then run it on two nodes you swear are identical, skip --upload until you have read what it sends, and compare the two score.json files.
