Ruby’s AI-friendly reputation faces scrutiny under scale

Developers have long touted Ruby on Rails as an AI-friendly stack, praising its concise syntax and convention-over-configuration philosophy for making AI-generated code more reliable. But a fresh analysis of thirteen real-world Ruby codebases challenges this narrative, uncovering a critical gap that AI tools fail to bridge.

The myth of AI-friendly Ruby code

Ruby’s reputation as an AI-friendly language stems from its structured conventions, which guide both human developers and AI models toward predictable code patterns. Rails’ opinionated framework, for instance, anticipates where models, services, and concerns should reside, reducing the cognitive load for AI assistants asked to generate or modify code.

For tasks like "write a service object" or "add a scope to this model," AI performs admirably. The conventions embedded in Rails act as a mental shortcut for the model, minimizing hallucinations and streamlining the generation process. This efficiency is undeniable—yet it only addresses half of the problem.

Why navigating Ruby at scale breaks AI tools

AI’s strength in Ruby lies in reading and generating code, not in understanding the intricate web of dependencies that define a mature codebase. Reading is straightforward: the model scans a file and extracts relevant information. Navigation, however, demands a holistic view of how components interact across hundreds of files, directories, and polymorphic relationships.

In a recent experiment, researchers tested AI’s ability to identify dependent files before making changes to a central model—a task critical for preventing broken deployments. The results were stark. When given a structural map of the codebase, AI’s recall rate skyrocketed:

• Chatwoot’s Inbox model jumped from 29% to 97%
• Mastodon’s Status model rose from 28% to 83%
• GitLab’s MergeRequest improved from 26% to 67%
• Discourse’s Upload model climbed from 35% to 75%
• Solidus’ Spree::Order increased from 40% to 68%

The pattern was consistent: AI without a structural map struggled to identify hidden dependencies, often missing critical connections that could lead to cascading failures. Only in smaller, tightly coupled repositories did AI perform reliably without additional context.

The surprising role of model size and codebase fame

Conventional wisdom suggests larger AI models would outperform smaller ones in navigating complex codebases. Surprisingly, the opposite held true in the study. The smallest model tested—Devstral, an open-source model—achieved the second-highest improvement when paired with a structural map, outperforming even larger frontier models. This counterintuitive result stems from a simple truth: smaller models retain less context in their weights, making external aids like structural maps more valuable.

Equally revealing was the performance of high-profile repositories. Forem, which powers dev.to, and Rails itself, showed that familiarity doesn’t equate to clarity. Models heavily trained on these repositories could recite their structures from memory, leading to false confidence. Yet, when tested on live code, these models still missed critical dependencies, proving that memorization doesn’t replace structural understanding.

The case of langchainrb—a small gem whose training data lagged behind its rapid evolution—highlighted another pitfall. Models recalled an outdated version of the gem, demonstrating how stale training data can mislead even the most advanced tools.

The path forward: structural context as the new standard

Ruby’s AI-friendly reputation isn’t entirely unfounded, but it’s incomplete. AI excels at generating and reading code in Rails’ well-trodden paths, yet struggles to navigate the sprawling dependency networks of mature applications. The solution isn’t better models—it’s better context.

Integrating structural maps, dependency graphs, or even simple cross-references into AI workflows can bridge the gap, transforming AI from a code-generating assistant into a reliable partner for refactoring and maintenance. Without this context, even the most advanced AI tools will remain blind to the hidden connections that define a codebase’s resilience. The future of AI-assisted development in Ruby depends on moving beyond conventions and embracing the complexity of real-world systems.