Google’s TPU v8 chips slash AI costs by cutting Nvidia’s margins

Google has long avoided the premium pricing of Nvidia’s GPUs by building its own AI accelerators. On Tuesday evening in Las Vegas, the company doubled down on this strategy with the unveiling of its eighth-generation Tensor Processing Units (TPUs), a two-chip lineup designed to handle both large-scale AI training and low-latency agentic workloads more efficiently than off-the-shelf hardware.

At an invite-only event at F1 Plaza, Google’s senior vice president of AI and infrastructure, Amin Vahdat, emphasized that the company’s end-to-end control over its AI stack—from hardware to software to models—translates into measurable cost advantages for enterprise customers. "We design every layer together," Vahdat said. "That vertical integration changes the cost-per-token economics in ways competitors can’t easily match."

A two-chip roadmap born from a contrarian bet in 2024

The decision to split the TPU v8 roadmap into two distinct chips wasn’t accidental. According to Vahdat, Google made the call in 2024—a full year before the industry’s pivot toward reasoning models, agentic workflows, and reinforcement learning became mainstream. At the time, it was a bold move. "We realized two years ago that one chip a year wouldn’t be enough," he recalled during a fireside chat. "This is our first step toward two highly specialized chips that address different problems."

For enterprises, this means an end to the inefficiency of renting the same accelerators for both training and inference. TPU v8t is optimized for training frontier models, while TPU v8i targets the memory-intensive, low-latency demands of real-time agent interactions and reinforcement learning.

TPU v8t: Scaling training to a million chips with Virgo networking

TPU v8t represents a generational leap in training performance. Google claims it delivers 2.8 times the FP4 EFlops per pod compared to its predecessor, Ironwood (121 vs. 42.5), doubles the bidirectional scale-up bandwidth to 19.2 Tb/s per chip, and quadruples scale-out networking to 400 Gb/s per chip.

Perhaps the most significant improvement is scalability. TPU v8t clusters—dubbed Superpods—can now scale beyond one million chips in a single training job, thanks to Google’s new Virgo networking fabric. The chip also introduces TPU Direct Storage, which bypasses traditional CPU-mediated data paths by transferring data directly from Google’s managed storage tier into high-bandwidth memory (HBM). For long-running training jobs where time is the primary cost driver, this reduces the number of pod-hours needed per epoch.

TPU v8i and Boardfly: Redefining latency for real-time agent workloads

While v8t pushes the boundaries of training performance, TPU v8i reimagines the hardware requirements for agentic workloads. The improvements are stark: Google reports a 9.8x increase in FP8 EFlops per pod (11.6 vs. 1.2), a 6.8x jump in HBM capacity per pod (331.8 TB vs. 49.2), and a pod size that grows 4.5x from 256 to 1,152 chips.

The key innovation here is the Boardfly topology, a network redesign that prioritizes latency over bandwidth—a critical shift for real-time applications. "Our default network was built for moving large amounts of data, not for minimizing response time," Vahdat explained. In collaboration with Google DeepMind, the team engineered a topology that shrinks the network diameter, reducing the number of hops between any two chips in a pod. Combined with a Collective Acceleration Engine and larger on-chip SRAM, Google claims a fivefold improvement in latency for real-time large language model sampling and reinforcement learning.

Breaking free from the “Nvidia tax” with vertical integration

At the heart of Google’s competitive edge is its refusal to pay Nvidia’s data-center gross margins—the so-called “Nvidia tax” that industry analysts have criticized for years. While OpenAI, Anthropic, xAI, and Meta rely on Nvidia’s H200 and Blackwell GPUs for training, Google designs, fabricates, and packages its own TPUs. This vertical integration eliminates the structural cost disadvantage that comes with renting third-party silicon.

Google’s AI stack spans six layers: energy, data center infrastructure, hardware, software, models (Gemini 3), and services. Vahdat argued that designing each layer in isolation forces compromises at every step. "When you optimize each layer separately, you end up with the least common denominator," he said. "Google’s approach is to build them together."

Evaluating TPU v8: What IT leaders should watch in 2026–2027

For procurement and infrastructure teams, TPU v8 reframes the cloud evaluation process around three key considerations:

• Training workloads: Teams running large proprietary models should prioritize v8t availability, Virgo networking access, and goodput service-level agreements (SLAs) over raw EFlops numbers.
• Agent and reasoning workloads: Organizations deploying real-time agents or reinforcement learning should evaluate v8i availability on Vertex AI, independent latency benchmarks, and HBM-per-pod sizing for context window requirements.
• Gemini Enterprise users: Customers leveraging Google’s Gemini models through Gemini Enterprise will inherit the v8i performance gains, with production deployment ceilings expected to rise significantly through 2026.

The caveats are real. General availability timelines remain unconfirmed, and early adopters will need to benchmark performance against their specific workloads. Still, for enterprises seeking to reduce AI infrastructure costs without sacrificing performance, Google’s TPU v8 lineup offers a compelling alternative to Nvidia’s premium-priced GPUs.

The compute race is far from over, but with TPU v8, Google has sharpened its tools—and its cost advantage.