How GitHub Copilot improved code reviews by refining agent instructions

GitHub Copilot’s automated code review agent was designed to analyze pull requests by scanning differences, exploring surrounding code, and flagging potential issues before they reached production. To do this effectively, the agent relied on specialized code exploration tools tailored to its workflow. When the team migrated to the more robust and widely maintained tools from the Copilot CLI—grep, glob, and view—they expected smoother performance and shared infrastructure benefits. Instead, early benchmarks revealed a surprising regression: the agent’s review costs increased while its ability to catch issues declined.

The root cause wasn’t the tools themselves but the instructions guiding them. Once the team revised the review workflow to align with how human reviewers actually operate—starting from the diff and asking targeted questions rather than broadly exploring the repository—the regression turned into a measurable improvement: a 20% reduction in average review costs without sacrificing quality.

The flawed assumption behind tool migration

Migrating to shared tools like grep, glob, and view seemed logical. These tools power multiple Copilot agent products, including the GitHub Copilot cloud agent, and promise consistency across the ecosystem. The goal was to eliminate duplicated tool implementations, centralize improvements, and ensure that fixes benefited all products using the same infrastructure.

The original Copilot code review tools were not simple wrappers. They automatically included surrounding context when searching directories or reading file ranges, which aligned with the behavior of earlier agentic models. These models made fewer tool calls and struggled to pull in necessary context automatically. Including extra context in each call helped compensate for that limitation.

However, the shared CLI tools followed a different design philosophy. They were optimized for general-purpose code exploration, not for the specific task of reviewing pull requests. The migration plan looked straightforward:

• Replace list_dir and search_file with glob and grep.
• Swap read_code with view.
• Maintain the same functionality.

But the results told a different story. Offline benchmarks showed the agent becoming less efficient and effective. Review costs rose, and the number of useful comments dropped.

Why the agent was searching too broadly

The team’s internal benchmarks are more than just performance metrics—they reveal the agent’s decision-making process. By examining tool calls, output size, and paths taken, engineers could see exactly where the agent went wrong.

Instead of reviewing pull requests like a human—starting from the diff and asking targeted questions—the agent defaulted to broad repository exploration. It would:

• Search widely across the codebase.
• Guess likely file paths.
• Read large sections of unrelated code.
• Accumulate unnecessary context.

This behavior resembles how a coding assistant might operate, mapping the entire repository to avoid breaking changes in unrelated areas. But for code review, this approach is inefficient. Human reviewers focus on the diff and ask precise questions:

• Where is this new function called elsewhere in the codebase?
• Are there other references to this configuration key?
• Does a similar pattern exist in tests or helper functions?
• What is the smallest code range needed to understand this behavior?

The agent, however, was not instructed to prioritize these targeted investigations. Every tool result added to its working context, increasing token usage and diluting focus. Unlike a printed report, which can be discarded after review, an agent’s working context retains all prior tool outputs. Extra, irrelevant information could mislead downstream reasoning.

Refining instructions to match reviewer workflows

The solution wasn’t to replace the tools but to adjust the instructions guiding their use. The agent needed workflows that mirrored human review practices, starting from the diff and narrowing focus efficiently.

The revised instructions emphasized:

• Context minimization: Only retrieve what is necessary to answer a specific question about the diff.
• Targeted exploration: Use tools to answer concrete questions, such as finding references to a new function or validating configuration changes.
• Cost awareness: Avoid broad searches that accumulate unnecessary context.

With these changes, the agent’s behavior shifted from broad exploration to precise investigation. It began asking targeted questions and stopping once it had sufficient evidence, reducing token overhead and improving review quality.

The result was a clear win: a 20% reduction in average review costs while maintaining the same level of issue detection as before the migration.

What this means for AI-powered code review

This experience highlights a critical lesson for AI-powered development tools: tools alone don’t determine success—workflows and instructions do. Even the most advanced tools can underperform if guided by the wrong instincts.

For teams building on agent frameworks, the story underscores the importance of continuously refining workflows to match real-world use cases. The Copilot code review agent’s journey from regression to improvement demonstrates how small adjustments in instruction can unlock significant gains in efficiency and effectiveness.

As AI agents take on more complex tasks, their success will depend not just on the power of their tools but on the clarity of their instructions. The next phase for Copilot and similar systems may involve further automating the alignment between tool capabilities and workflow needs, ensuring agents operate at peak efficiency from day one.