When a tech company publishes a benchmark, the first question should never be "How fast is it?" but rather "How fair is the test?" A recent report from Tolly, commissioned by F5, provides a textbook example of why sponsored benchmarks demand extra scrutiny—even when the underlying engineering is sound.
The March 2026 study set out to compare F5’s BIG-IP Next for Kubernetes against three open-source load balancers: HAProxy, Envoy, and an unnamed third solution. The headline claim? F5’s AI-driven load balancer outperformed the open-source alternatives in token throughput, time to first token, and CPU usage within an AI inference cluster. On paper, the results looked impressive. In practice, the test was structured less like a fair evaluation and more like a setup designed to produce a specific outcome.
The illusion of a level playing field
The core issue wasn’t the data itself—there’s no evidence the numbers were fabricated—but the way the experiment was designed. Before running any tests, the researchers manually loaded 50% of the GPUs in the cluster with background traffic. This load bypassed all load balancers entirely, creating an artificially uneven distribution of work. Then, they measured how each load balancer performed when routing new requests through the congested pool.
F5’s product was configured to detect and avoid overloaded GPUs, a feature built specifically for this scenario. The open-source load balancers, however, were set to static round-robin routing—a configuration that blindly distributes traffic without regard for backend load. This setup guaranteed one result: F5’s intelligent routing would outperform the static configurations every time.
The problem isn’t that F5’s product works; it’s that the test didn’t give open-source tools a fair chance. Engineers rarely use static round-robin in production, especially in uneven workloads. Both HAProxy and Envoy support dynamic, load-aware routing and health-based algorithms designed precisely for scenarios like this. Yet in the report, these features were disabled, leaving the open-source tools to compete with one hand tied behind their backs.
Offload hardware beats host software—no surprise there
Another headline-grabbing claim focused on CPU usage. F5’s solution reportedly used about 2 CPU cores while HAProxy consumed roughly 12 out of 16 available cores—a difference of roughly 80%. At first glance, this suggests F5’s software is far more efficient. But the reality is far simpler: F5’s solution ran on a BlueField DPU, a dedicated offload device with its own ARM cores, while HAProxy ran on the host CPU. Moving network processing to a separate chip reduces host CPU usage by design—regardless of the workload.
Calling this a software efficiency win is like crediting a sprinter for winning a race after they showed up on a motorcycle. The hardware advantage is real, but attributing it to software performance is misleading. Offload hardware is purpose-built to handle specific tasks, and its superiority in this context isn’t surprising—it’s expected.
How to critically evaluate any tech benchmark
Sponsored benchmarks aren’t inherently bad, but they require a higher standard of scrutiny. The techniques used in Tolly’s report aren’t unique to F5; they appear in countless industry studies. Learning to recognize them will help you spot manipulated data before it influences your decisions.
First, follow the money. Who funded the research? Sponsorship alone doesn’t invalidate results, but it does mean the study has a vested interest in positive outcomes. In this case, F5 commissioned the report, raising the bar for transparency and fairness.
Second, count the variables. A well-designed experiment changes only one factor at a time. This study altered multiple variables simultaneously—the hardware platform, the routing algorithm, and the software configuration—then attributed the entire outcome to the software. When too many factors shift, the results become impossible to interpret accurately.
Third, check the baseline. Was the competitor given a fair setup, or was it deliberately handicapped? Static round-robin routing in a lopsided workload is a setup designed to fail. Any benchmark that uses such a strawman configuration should be viewed with skepticism.
Fourth, watch for cherry-picking. The report highlights the most dramatic metrics—like "up to 114% faster" and "406% improvement"—while omitting less impressive comparisons. These numbers often come from testing only against the weakest competitor or focusing on specific scenarios that inflate the results.
Finally, ask whether the results are reproducible. Could an independent team replicate the test with the same configurations and tools? If key details are missing—like the unnamed third competitor, early-access software, or unpublished test setups—then the results aren’t truly verifiable. Real science survives scrutiny; marketing claims often do not.
The next time you encounter a sponsored benchmark, resist the urge to take the headline at face value. Instead, peel back the layers to see what the test was really measuring—and what it was designed to hide. A well-run experiment tells a story. A rigged one tells a conclusion.
Until benchmarks are held to the same standards as peer-reviewed research, the burden of proof falls on the reader to separate signal from noise.
AI summary
Sponsorlu teknoloji benchmarkları neden güvenilir değildir? F5’in bir raporunu inceleyerek, yanıltıcı deney düzeneklerini ve nasıl tespit edeceğinizi öğrenin.
