Google Splits TPU Gen 8 into Dedicated Training and Inference Chips

Google’s latest Tensor Processing Unit announcement marks a pivotal shift in AI hardware: for the first time, the company is splitting its flagship TPU into two distinct chips—one optimized for training, the other for inference and agentic workloads. This architectural separation reflects a growing divide in the demands of modern AI systems, where real-time reasoning and multi-step agent workflows require infrastructure that can keep pace with human-like decision-making.

A Hardware Split Born from Real-World AI Challenges

Traditional chat-based AI systems operate within a predictable latency window. Users submit a prompt, receive a response after a brief delay, and that’s the end of the interaction. Agentic AI, however, functions differently. It breaks down complex goals into subtasks, delegates work to specialized agents, evaluates outcomes, and iterates—all while maintaining near-instant responsiveness. These workflows generate a relentless stream of real-time communication, where even minor delays compound into costly inefficiencies.

Google’s internal deployments of models like Gemini revealed that previous TPU generations, designed as unified chips, struggled to balance the conflicting requirements of training and inference. The result? Sluggish agent loops, inflated operational costs, and scalability bottlenecks. By separating the hardware, Google aims to eliminate these trade-offs, ensuring that each chip excels in its intended role.

Meet the TPU 8t and TPU 8i: Two Chips, Two Missions

TPU 8t: The Training Titan

The TPU 8t is built for the heaviest lifting in AI: training frontier models. A single superpod combines 9,600 chips, delivering 121 exaflops of compute and 2 petabytes of shared memory, connected via high-speed inter-chip interconnects. Compared to its predecessor, the 8t offers triple the compute performance and double the interconnect bandwidth, enabling near-linear scaling even at million-chip clusters.

This architecture enables training clusters that span multiple data centers, effectively turning distributed infrastructure into a unified supercomputer. For labs and enterprises training large proprietary models—such as DeepMind’s Genie 3, a world model enabling millions of agents to refine their reasoning in simulated environments—the 8t redefines what’s possible in terms of scale and efficiency.

TPU 8i: The Latency-Optimized Powerhouse for Agentic Workloads

The TPU 8i introduces several groundbreaking innovations tailored for inference and agentic tasks. Its most notable feature is the Collectives Acceleration Engine (CAE), a dedicated unit that dramatically reduces the latency of reduction and synchronization steps during autoregressive decoding and chain-of-thought processing. This innovation slashes on-chip latency for collective operations by 5x, a critical advantage for agent workflows that depend on rapid, iterative decision-making.

Google also redesigned the 8i’s inter-chip network topology using a high-radix design called Boardfly. Unlike previous generations that prioritized bandwidth, Boardfly prioritizes latency by reducing the number of hops data packets must take. A single Boardfly group can connect up to 1,152 chips, cutting network diameter and achieving up to a 50% improvement in latency for communication-heavy workloads.

The raw specs underscore the 8i’s capabilities:

• - 9.8x the FP8 exaflops per pod compared to prior generations
• - 6.8x the HBM capacity per pod
• - Pod size expanded by 4.5x, from 256 to 1,152 chips
• - 80% better performance per dollar for inference workloads

Practical Applications: Where the Split Makes the Biggest Impact

The separation of TPU 8t and TPU 8i unlocks new possibilities across three key areas:

• - Frontier Model Training: The 8t’s near-linear scaling at million-chip clusters transforms the economics of retraining massive models. Organizations training proprietary models or world models like Genie 3 can now achieve unprecedented scale without sacrificing performance.

• - High-Concurrency Agentic Inference: The 8i shines in environments where thousands of agents operate simultaneously, such as multi-agent pipelines, Mixture-of-Experts (MoE) serving, and chain-of-thought reasoning loops. Its latency optimizations ensure that each step in an agent’s workflow executes with minimal delay, even under heavy load.

• - Reinforcement Learning Post-Training: Google’s new Axion-powered N4A CPU instances complement the TPU split by handling the complex orchestration logic, tool calls, and feedback loops surrounding AI models. This combination delivers up to 30% better price-performance than comparable agent workloads on other cloud providers.

Infrastructure Upgrades: The Hidden Enablers of the TPU Split

The TPU 8t and 8i don’t operate in isolation. Google has paired them with significant upgrades to networking, storage, and interconnects to fully realize their potential:

• - Virgo Network: The new collapsed fabric architecture offers 4x the bandwidth of previous generations, enabling a single data center to connect 134,000 TPUs into one cohesive fabric.

• - Google Cloud Managed Lustre: This storage solution now delivers 10 TB/s of bandwidth—10x faster than last year—with sub-millisecond latency, thanks to TPUDirect and RDMA. Data bypasses the host entirely, moving directly to accelerators for maximum efficiency.

Beyond the Hype: Why This Split Matters for the Agentic Era

While some may frame this announcement as a direct challenge to Nvidia, that perspective misses the point. Google’s strategy is rooted in solving real-world problems, not competing on hardware specs alone. The company is positioning itself as a partner to enterprises and researchers, offering a hardware stack that aligns with the evolving demands of agentic AI.

The implications extend beyond AI workloads. By decoupling training and inference, Google is enabling a future where AI systems can operate with human-like responsiveness, scale seamlessly, and adapt to increasingly complex tasks. This split isn’t just a hardware innovation—it’s a foundational step toward the next era of intelligent computing.