Why cheaper AI models can sometimes cost more in practice

A recent experiment revealed a counterintuitive truth about AI model pricing: the model marketed as cheaper didn’t always deliver lower costs in practice. After routing the same single-word prompt to both Claude Haiku and Gemini 2.5 Flash, the results showed a surprising disparity in billing. While Flash boasts a lower per-token rate, its "thinking" phase consumed significantly more tokens before returning a response. Haiku answered in just four tokens, but Flash required about 28 tokens to process the same query—effectively making it nearly 8.6 times more expensive per request. This discrepancy only became apparent through meticulous instrumentation of every API call, tracking tokens, latency, and costs in a Postgres database. The practice isn’t unusual for someone with a background in financial systems, where even minor errors can have serious consequences.

I’ve spent years ensuring accuracy in cross-border real-time payment systems at NPCI, where a rounding error is an operational incident and a system outage can derail an entire weekend. This year, as I built an LLM gateway, I instinctively applied the same principles I’d honed in finance. The underlying challenge wasn’t fundamentally new—it was a variation on a familiar problem. A model API behaves like any other downstream dependency: slow, occasionally unreliable, rate-limited, and billed per interaction. Whether it’s a payment processor, a KYC service, or a model provider, the core issues remain the same—reliability, cost control, and failover mechanisms.

The architecture behind cost control

The gateway required fail-safes to prevent cascading failures, just as payment systems do. Circuit breakers, a staple in distributed systems, became essential here too. When a model provider’s performance degrades, blindly retrying requests only exacerbates the issue, filling worker pools with stuck calls. Instead, the system needed to trip the breaker, fail fast, and gradually probe for recovery—transitioning between CLOSED, OPEN, and HALF_OPEN states. The terminology changed from "partner bank" to "model provider," but the underlying logic remained identical.

Auditing costs was another familiar challenge. Every call had to be logged in real time to track spending, much like an audit trail in financial systems. Across dozens of Go services, idempotency keys ensured that retried requests wouldn’t double-count transactions. Similarly, the cost log relied on request IDs to prevent duplicate billing. Even data types mattered—costs were stored as fixed-precision NUMERIC values rather than floats to avoid rounding errors. Precision isn’t just a best practice in finance; it’s a necessity.

Retries and idempotency: lessons from finance

The distinction between transient and sustained failures is a core concept in both payment systems and AI infrastructure. In finance, a careless retry could result in double-debiting a customer, so timeouts trigger idempotent retries—not blind resubmissions. Sustained failures require halting further attempts. Reapplying this logic to LLM calls shifted the stakes from financial loss to wasted tokens, but the problem’s structure remained unchanged.

What’s genuinely new in AI infrastructure

While much of the work mirrored familiar systems engineering, a few challenges were unique to AI. Token economics, for instance, introduces variables absent in traditional billing models. The 8.6x cost differential wasn’t something I’d encountered in databases or payment processors. Additionally, model outputs are non-deterministic—a single input might yield different responses, making exact string comparisons unreliable. Testing frameworks evolved from asserting equality to evaluating distributions and performance benchmarks. Perhaps most unexpectedly, models can silently consume output budgets through internal reasoning, a problem with no direct parallel in payment systems.

Why backend expertise matters in AI

The gateway now operates reliably, but its success hinges on principles far older than AI itself. Anyone can make a single API call to an LLM in an afternoon. Far fewer can design a system that remains resilient when providers degrade, adapts to shifting token economics, and provides transparent cost tracking. That expertise isn’t rooted in AI—it’s rooted in distributed systems, observability, and cost management. The model is merely the new dependency attached to an existing playbook.

The gateway is operational at llm-gateway-python.onrender.com, with the source code available on GitHub under Yogesh23012001’s repository.