Open vs Closed AI Models: Performance and Cost Compared

Open-source AI models are no longer just experimental alternatives—they’re now trading blows with closed models on real-world software tasks. A recent evaluation using the Ship-Bench benchmark pitted three top open models—Kimi K2.6, Qwen 3.6 Plus, and DeepSeek v4 Pro—against the same coding workflows typically reserved for proprietary systems. The results challenge assumptions about the performance gap while exposing stark differences in efficiency and cost.

Benchmark Setup: Testing AI in Real Software Workflows

To measure how these models perform beyond isolated tasks, the Ship-Bench benchmark simulates a full software development lifecycle. It assigns five distinct roles—Architect, UX Designer, Planner, Developer, and Reviewer—to each model, requiring them to collaborate sequentially on building a simplified knowledge base application. Each phase produces artifacts that feed into the next, testing not just individual capability but handoff quality in a realistic workflow.

The evaluation used identical hardware and software environments for all three models, running on Windows 11 with Node.js v24 and the Copilot CLI harness. The only variables were the target model and, in DeepSeek’s case, a slightly newer CLI build. The benchmark task—a knowledge base app—was designed to expose differences in architecture, planning, implementation, and quality assurance while remaining constrained enough for fair comparison.

Head-to-Head Results: Quality vs. Token Efficiency

All three open models delivered competitive quality, but their efficiency varied dramatically. DeepSeek v4 Pro led the pack with an average score of 94.18 and passed all five SDLC roles, followed closely by Kimi K2.6 at 93.96. Qwen 3.6 Plus trailed at 90.74, failing one gate due to planning deficiencies.

Metric            Kimi K2.6   Qwen 3.6 Plus   DeepSeek v4 Pro
Average Score     93.96       90.74           94.18
Pass Rate         5/5         4/5             5/5

• DeepSeek v4 Pro excelled in developer efficiency, achieving the highest scores in architecture (95.56) and development (98.75), while maintaining remarkably low token usage at 26.3 million tokens.
• Kimi K2.6 matched DeepSeek’s quality nearly across the board, with standout performance in UX design (98.57) and planning (98.33), though its token consumption ballooned to 64.1 million.
• Qwen 3.6 Plus struggled with planning, failing its gate with only 87.30% good chunks, and used 63.3 million tokens—nearly matching Kimi’s inefficiency despite weaker results.

Where the Models Shone—and Struggled

Each model demonstrated unique strengths across the five SDLC roles, revealing trade-offs between raw capability and practical usability.

Architecture: Clarity and Completeness

All three models produced implementation-ready architecture documents, but differences emerged in structure and foresight. DeepSeek’s output was praised for its organization and completeness, while Kimi received high marks for innovation—proposing a separate API server despite the task’s constraints. Qwen’s planning artifacts were solid but suffered from version drift and weaker maintainability notes.

UX Design: Consistency Across the Board

UX design scores were nearly identical, with all three models achieving 98.60%. This suggests that modern AI systems have standardized their approach to user interface generation, at least for straightforward applications.

Planning: Qwen’s Flaws Exposed Downstream Workflow

Qwen’s planner failed its gate with only 87.30% good chunks, mixing oversized iterations with undersized sub-tasks. This planning failure propagated into development, where Qwen scored just 92.00—well below DeepSeek’s 98.75 and Kimi’s 97.00. The missteps forced developers to rework plans, increasing token usage and delaying completion.

Development: DeepSeek’s Efficiency Wins

DeepSeek’s developer phase stood out for its precision, combining high scores with minimal token burn. Kimi followed closely, while Qwen’s development phase suffered from the earlier planning missteps, requiring more iterative fixes.

Review: All Models Performed Adequately

Review scores were clustered between 82.00 and 85.00, indicating that code review remains the weakest link in AI-driven workflows. None of the models demonstrated exceptional QA capabilities, suggesting room for improvement in automated testing and validation.

The Cost Conundrum: When Efficiency Matters More Than Price

While open models promise cost advantages, the benchmark revealed a counterintuitive reality: token usage—not the model’s base price—dominated the economics. DeepSeek’s efficiency meant it could deliver top-tier results with far fewer tokens, making it the most compelling choice despite similar headline pricing to Kimi and Qwen.

Kimi and Qwen’s heavy token consumption inflated their effective costs, erasing any advantage from being open-source. For teams evaluating these models, the takeaway is clear: efficiency in token usage often outweighs raw model price when calculating total project costs.

The Future of Open Frontier Models

The results confirm that open models have closed the quality gap in software development tasks, but efficiency remains the decisive factor. As AI systems grow more complex, the ability to deliver high-quality output without excessive token consumption will separate the leaders from the laggards.

For developers and engineering teams, the choice between open and closed models is no longer binary. Instead, it’s about matching model strengths to project requirements—whether that’s DeepSeek’s balance of quality and efficiency, Kimi’s architectural creativity, or Qwen’s potential with refined planning strategies. One thing is certain: the era of open models as second-tier alternatives is over.