AI uncovers smarter compute strategies that slash LLM token costs by 70%

Researchers from Meta, Google, and several universities have developed a breakthrough framework called AutoTTS that automates the design of test-time scaling (TTS) strategies for large language models (LLMs).

By replacing manual heuristics with algorithmic discovery, AutoTTS enables organizations to dynamically optimize compute allocation during inference, significantly reducing both token consumption and operational costs without compromising accuracy. In experimental evaluations, the framework demonstrated the ability to cut token usage by up to 69.5% while preserving performance parity with traditional handcrafted methods.

The limitations of manually crafted reasoning strategies

Test-time scaling enhances LLMs by allocating additional computational resources during the inference phase, allowing models to explore multiple reasoning paths before finalizing responses. This extra compute is particularly valuable for complex tasks where precision matters.

However, designing effective TTS strategies has long relied on human intuition to define rigid rules for branching, pruning, and stopping reasoning paths. Engineers typically experiment with various heuristics—such as fixed trajectory sampling, early stopping based on confidence thresholds, or parallel probing—to balance accuracy against computational cost.

The core challenge lies in mapping these decisions to a two-dimensional control space: width (number of reasoning branches explored) and depth (how far each branch develops). While methods like self-consistency, adaptive-consistency, and parallel-probe exist, all are manually engineered and inherently limit the exploration of more sophisticated, nuanced strategies. Even advanced approaches using tree search or external verifiers remain constrained by human-defined constraints, leaving vast portions of the optimal resource-allocation space untapped.

How AutoTTS transforms strategy discovery into an algorithmic process

AutoTTS reimagines TTS optimization not as a human-led task but as an algorithmic search problem within a controlled environment. The framework shifts the engineer’s role from crafting rigid rules to defining the discovery environment itself—establishing boundaries such as the control space of states and actions, balancing objectives between accuracy and cost, and implementing feedback mechanisms.

At the heart of AutoTTS is an explorer LLM, which acts as an autonomous agent tasked with designing TTS controllers. These controllers are essentially code-based policies that dictate how an LLM allocates its computational budget during inference. The explorer iteratively proposes, tests, and refines these controllers based on empirical feedback until an optimal policy emerges.

To make this automated search computationally feasible, AutoTTS leverages an offline replay environment. Instead of invoking a base reasoning model to generate new tokens for every strategy test—which would incur prohibitive costs—the framework relies on pre-collected reasoning trajectories from the base LLM. These trajectories include probe signals, intermediate answers that help the controller evaluate progress across different reasoning branches without incurring additional inference costs.

During the discovery loop, the explorer agent proposes a controller and evaluates it against the offline data. It analyzes execution traces to diagnose failure modes—for example, identifying when a controller prunes branches too aggressively in specific scenarios—allowing iterative refinement of the controller’s code to improve the accuracy-cost tradeoff.

Inside the AI-generated controller: non-intuitive optimizations uncovered

Because the explorer agent isn’t bound by human intuition, it can discover highly coordinated, complex rules that engineers would likely overlook. One such controller, named the Confidence Momentum Controller, employs several counterintuitive mechanisms:

• Trend-based stopping: Traditional strategies often instruct models to halt reasoning when confidence reaches a predetermined threshold. AutoTTS’s controller, however, tracks an exponential moving average (EMA) of confidence to avoid premature termination caused by temporary spikes. It only stops reasoning when the overall confidence trend is rising and remains consistently high.

• Coupled width-depth control: Many handcrafted algorithms treat width (number of branches) and depth (exploration depth) as separate variables. The AI-designed controller dynamically couples these dimensions, adjusting both simultaneously based on real-time evidence to maximize accuracy per token spent.

• Adaptive pruning thresholds: Instead of using static confidence thresholds to discard branches, the controller adjusts pruning criteria based on contextual cues, such as the divergence between predicted and observed outcomes across branches.

These innovations collectively enable the controller to make nuanced, context-aware decisions that minimize wasted computation while ensuring robust reasoning outcomes.

The road ahead: from research to production-scale adoption

The implications of AutoTTS extend beyond academic curiosity. For enterprises deploying advanced reasoning models in production—whether for customer support, technical diagnostics, or decision-making tools—the framework offers a pathway to substantially lower operational costs while maintaining or even improving output quality.

As LLMs continue to scale in complexity and cost, automated optimization tools like AutoTTS will play a pivotal role in making advanced AI accessible and sustainable. The next frontier may involve integrating such frameworks with real-time feedback loops, enabling continuous, dynamic optimization of reasoning strategies in live production environments.

The research underscores a broader shift in AI development: where human intuition once dictated the boundaries of innovation, algorithmic discovery is now taking the lead in unlocking unprecedented efficiencies.