AWS Security Agent’s full pipeline: Lessons from running it on a real project

A common frustration in software development is the post-launch security blind spot. Teams pour effort into threat modeling, code audits, and penetration tests before release—only to watch those practices fade into the background once the app goes live. AWS Security Agent aims to close that gap by offering continuous security checks across design, code, and runtime, not just at launch.

I decided to put the full pipeline to the test on a real project: a real-time Kahoot!-style quiz app built with a serverless stack. What unfolded over three weeks of reviews and pentests wasn’t just a security audit—it was a masterclass in how AI-driven security tools handle ambiguity, custom policies, and integration friction. Here’s what I learned.

How AWS Security Agent Covers the Entire Lifecycle

AWS Security Agent bundles three security checks into a single workflow called Agent Space. Each phase is designed to address a different stage of the product’s life:

• Design Review: Scans architecture docs, requirements, and design specs against AWS security best practices and custom rules.
• Code Review: Integrates with GitHub to post security-focused comments on pull requests.
• Penetration Test: Runs live attacks on the running app using static and dynamic analysis, even generating pull requests to fix vulnerabilities.

As of April 2026, the Penetration Test feature is generally available (since March 31, 2026), while Design Review and Code Review remain in Preview. The unified interface makes it easy to run multiple checks without switching tools.

The Test Subject: A Serverless Quiz App in Production

Rather than testing on a deliberately vulnerable demo app, I chose a personal project with real-world complexity: a Kahoot!-style quiz platform with:

• Frontend: CloudFront + S3
• API: API Gateway (REST + WebSocket)
• Backend: Python Lambda functions
• Database: DynamoDB
• Auth: Cognito for admins, rotating 6-digit codes for participants

The app supports live quizzes with real-time results delivered via WebSocket. Total codebase: ~3,500 lines across 22 API endpoints. It’s not a toy, but neither is it an enterprise system—ideal for a solo developer running a serious security check.

Design Review: AI That Learns as It Reads

Design Review accepts multiple file types—DOCX, MD, PDF, PNG, TXT—and synthesizes their content to evaluate architecture against security policies. I uploaded four Markdown files:

• architecture.md: system design
• requirements.md: functional specs
• security-design.md: security controls
• sequence-diagram.md: auth and data flows

One unexpected strength: the agent doesn’t analyze files in isolation. It reads across all documents and builds a unified model of your system. That means it can trace how authentication flows from a sequence diagram through to the security design—not just flag isolated issues.

Custom Rules Without Friction

Beyond AWS’s managed rules, you can add your own requirements in plain language. I added two Japanese-language rules:

• Log retention ≥ 365 days
• Data must remain in a specific AWS region

Both were enforced correctly. This feature matters because many organizations already write security policies in natural language. Being able to paste those directly—without mapping to a formal schema—reduces translation overhead and makes compliance accessible to non-security teams.

The False Negative That Became a Lesson

After the first run, my design review came back clean. But after adding custom rules, something unexpected happened: a third finding appeared—"Secret Protection Best Practices"—that hadn’t been flagged earlier. The same documents now triggered a violation.

This isn’t a flaw in the tool—it’s a characteristic of probabilistic AI. AI models don’t work like static rules; they generalize patterns from data. A single pass may miss subtle edge cases, just as a human reviewer might overlook a nuance in the first read. The takeaway: run multiple reviews on critical documents. It’s the digital equivalent of having a second set of eyes.

The tool also simplifies iteration. You can clone an existing review and tweak only the rules, making it easy to embed into an agile workflow where designs evolve frequently.

Code Review: GitHub Bots That Actually Add Value

Code Review connects directly to GitHub and posts security-focused comments on pull requests via the aws-security-agent[bot] user. A key constraint: one AWS account can only link to one GitHub account. In multi-account AWS Organizations, you must centralize the integration—no spreading it across sub-accounts.

Once configured, the bot acts like a second reviewer: scanning new changes for secrets, insecure dependencies, or misconfigurations. Unlike traditional static analysis tools, it’s tightly integrated into the PR workflow, so developers see feedback at the exact moment they make the change.

While I only had a few PRs to test, the integration felt seamless. The bot’s comments were concise and actionable, avoiding the noise that often plagues security tools.

What Comes Next: Continuous Security, Not One-Time Checks

AWS Security Agent isn’t a silver bullet—it’s a shift in mindset. Instead of treating security as a milestone before launch, it treats it as a continuous practice. The agent’s ability to re-run design reviews, automate code feedback, and perform live pentests means security doesn’t disappear after deployment.

For solo developers and small teams, that’s a game changer. No more scrambling for budget to hire a pentester every quarter. No more hoping that no one merges a secret into main. Just consistent, automated checks that evolve with your app.

The real test will be how these features mature beyond Preview. But for now, AWS Security Agent shows that AI-driven security isn’t about replacing human judgment—it’s about making it easier to do the right thing, every day.