Playbooks
Why AI-Generated Code Reviews Still Miss the Same Categories of Bugs
AI code review tools consistently miss race conditions, security/authorization flaws, breaking API changes, and business-logic or intent errors — not random misses, but a recurring pattern rooted in how these tools work: they read code as text and pattern-match against known problem shapes, without executing it, understanding the broader system architecture, or knowing what the business actually intended the code to do. A benchmark of 67 real production bugs found even the best-performing tool caught only 49% of them, with widely used tools like GitHub Copilot catching far fewer — and having the same AI model both write and review code compounds the problem by carrying identical blind spots into both jobs.
Last updated Aug 21 · 20 min read
The four recurring miss categories
Across multiple independent benchmark studies and practitioner discussions, the same four categories of bugs come up again and again as what AI code review tools reliably fail to catch:
1. Race conditions and concurrency bugs — issues that only manifest when multiple threads or processes interact in a specific timing sequence, which requires reasoning about runtime behavior, not static code shape.
2. Security and authorization flaws — access-control gaps, privilege-escalation paths, and injection vulnerabilities that depend on understanding how a piece of code is actually invoked elsewhere in the system, not just how it looks in isolation.
3. Breaking API changes — a change that's internally consistent and syntactically correct but breaks a contract some other part of the system (or an external consumer) depends on.
4. Business-logic and intent errors — code that is syntactically valid, passes tests, and does something coherent, but solves the wrong problem or misunderstands what was actually asked for.
These four categories share a common thread: none of them are visible from the code's local syntax or structure alone. Catching any of them requires either running the code under realistic conditions, understanding cross-module or cross-service context beyond the diff being reviewed, or knowing the actual business intent behind the change — none of which a text-based AI reviewer reading a diff has direct access to (diffdojo.com; oreilly.com).
Why these categories specifically go undetected
It's worth being precise about the mechanism, not just the symptom. AI code reviewers, including the more sophisticated ones layered with retrieval and static-analysis integrations, primarily operate by reading a diff (or a diff plus some surrounding context) and pattern-matching it against known-risky shapes — patterns learned from vast amounts of training data showing what buggy code has historically looked like. This works well for issues that have a recognizable textual signature: an unescaped string going into a query, a missing null check, an obviously mismatched type.
It works poorly for the four categories above precisely because none of them have a reliable textual signature in isolation:
– A race condition might look like perfectly reasonable code on its own — the bug only exists in the interaction between this code and other code running concurrently, which a diff-scoped review can't see.
– An authorization flaw might require knowing that this particular endpoint is reachable by an unauthenticated user somewhere else in the routing configuration — information outside the diff.
– A breaking API change might be invisible unless the reviewer knows which other services or clients depend on the exact prior behavior.
– A business-logic error requires knowing what the code was *supposed* to do, which is knowledge that lives in a ticket, a conversation, or a product spec — not in the code itself.
Community discussion on Hacker News reinforces this same conclusion from a different angle: commenters describe AI reviewers functioning mainly as an "advanced linter" — catching style issues, obvious anti-patterns, and known-bad constructs, while missing duplicate code, cross-module coupling, and separation-of-concerns problems that require broader system or business knowledge to recognize (news.ycombinator.com/item?id=46766961; dev.to/jaideepparashar).
The benchmark numbers
The strongest quantitative evidence on this topic comes from a benchmark study using 67 real production bugs — not synthetic test cases, but actual bugs that shipped and were later identified. In that benchmark, the best-performing AI code review tool caught only 49% of the bugs (recall), with a best F1 score of 47.2% (achieved by Entelligence's own tool, worth noting given Entelligence published the benchmark). On the F1 metric specifically, Claude scored 42.8%, Cursor Bugbot 39.4%, Greptile 36.9%, CodeRabbit 33%, and GitHub Copilot scored notably lower at 22.6% (diffdojo.com).
A separate, much larger study covering over 200,000 pull requests found field effectiveness in the 50-60% range — broadly consistent with the smaller benchmark's findings, and suggesting the roughly-half-caught pattern isn't an artifact of one particular test set but a real, reproducible ceiling for current tooling (diffdojo.com).
This plateau echoes older, adjacent research on static analysis specifically: NIST's SATE (Static Analysis Tool Exposition) evaluations found static analysis tools plateau at 50-60% detection for security vulnerabilities, and a 2024 ISSTA study found that 22% of vulnerable commits went entirely undetected across five different tools tested (oreilly.com). AI-based review tools, despite being a newer generation of technology, appear to be converging on a similar ceiling rather than clearly surpassing it — at least on the categories of bugs measured in these studies.
The same-model blind-spot problem
A distinct and increasingly discussed risk is having the same (or a closely related) AI model both write the code and review it. The concern isn't hypothetical — it's a structural argument: if a generating model and a reviewing model share training data, architecture family, or learned patterns, they're likely to share the same misconceptions about what "correct" code looks like. That means the reviewing model may fail to flag exactly the kinds of errors the generating model is prone to making, because both models learned the same incorrect intuitions from similar sources (hackernoon.com; diffdojo.com).
This creates what one analysis describes as confirmation bias at scale rather than independent verification — instead of a genuinely separate check on the work, you get two passes through the same underlying blind spots. This is a meaningfully different (and arguably more concerning) failure mode than simple incompleteness, because it can produce a false sense of security: two "independent-looking" review steps that are actually correlated in their failures.
A related, concrete example of this dynamic is hallucinated or non-existent API usage: AI-generated code sometimes calls functions, methods, or libraries that don't actually exist, or exist with different signatures than assumed, and AI-based review commonly fails to flag this because the referenced code looks syntactically plausible and stylistically consistent with real code — the review model has no straightforward way to verify existence without actually checking against the real, current codebase or library (arxiv.org/pdf/2603.25146).
What AI review actually is good at
None of this means AI code review has no value — the evidence points to a specific, real, and useful role rather than a replacement for human review. It excels at catching the categories of issues that *do* have a reliable textual signature: obvious style violations, common anti-patterns, missing null checks, well-known security-smell patterns (like string concatenation into a query), and inconsistencies with a team's stated conventions. It's also fast and infinitely scalable in a way human review isn't — it can review every single diff, not a sampled subset, and it never gets tired at the end of a long review queue.
The more accurate framing, borne out by a Hacker News commenter's observation, is that an AI reviewer changes *what* a human reviews, not *whether* a human needs to review — automating the mechanical, pattern-matchable first pass so a human's limited attention goes toward the categories AI tools are weakest at: architectural fit, business-logic correctness, and cross-system implications (diffdojo.com).
Automation complacency and the noise problem
Two related, human-side failure modes compound the technical limitations above. The first is automation complacency: when a team sees an AI reviewer catching real bugs regularly, there's a documented tendency to start trusting a roughly 50% catch rate the way they'd trust a 90% one — under-investing in the human review step precisely because the AI reviewer looks competent on the bugs it does catch (diffdojo.com).
The second is noise: practitioners report AI reviewers generating large volumes of speculative or nitpick comments alongside genuinely useful findings, making it hard to separate real signal from noise, with results that can even vary non-deterministically across repeated runs on the same diff (news.ycombinator.com/item?id=46766961). A reviewer buried under dozens of low-value comments per pull request is less likely to carefully scrutinize the two or three comments that actually matter — the tool's volume can work directly against its usefulness.
This combination — a real but partial catch rate, plus a human tendency to over-trust tools that seem competent, plus noisy output that dilutes attention — is arguably a bigger practical risk than the raw miss rate itself. A survey of 105 engineering leaders found 61% shipped a production incident originating from AI-generated code within 90 days, even after that code passed both AI-assisted review and unit tests (news.ycombinator.com/item?id=46312159) — a concrete, real illustration of the gap between "passed review" and "was actually correct."
Practical examples
A financial API endpoint change. An AI reviewer flags an obvious SQL injection risk and a missing input-length check — both real, valuable catches with clear textual signatures. It does not flag that the new endpoint's authorization check runs after (rather than before) a database write in a rare error-handling branch, because that ordering issue only matters given specific runtime failure conditions the reviewer isn't executing against.
A microservices API contract change. A developer changes a shared internal API's response field from a string to a nested object, and the change passes AI review cleanly because it's internally consistent and well-typed within the file being reviewed. Three downstream services that parse the old string format break in production — a breaking-change miss that required knowing about consumers outside the diff's visible scope.
Illustrative example (hypothetical, for clarity). Imagine a feature meant to send a discount email only to first-time customers. The AI-generated code correctly implements "send an email to customers matching certain criteria," passes all existing tests, and gets a clean AI review — but a subtle logic error means it sends the discount to *repeat* customers instead, because the underlying flag it checks was misunderstood. This is a design-flaw or intent-violation bug: syntactically fine, behaviorally wrong, and invisible to a reviewer that has no access to what "first-time customer" was actually supposed to mean in this specific business context — directly analogous to the real Gson null-key bug referenced in the sourced research below, which persisted through years of code review because it was never a syntax problem.
Data and evidence
– A 67-real-bug benchmark found the best AI code review tool achieved only 49% recall, and a best F1 score of 47.2% (from the benchmark publisher's own tool); on the F1 metric, Claude scored 42.8% and GitHub Copilot scored 22.6% (diffdojo.com).
– A larger study across 200,000+ pull requests found field effectiveness in the 50-60% range, consistent with the smaller benchmark (diffdojo.com).
– NIST SATE evaluations found static analysis tools plateau at 50-60% detection for security vulnerabilities; a 2024 ISSTA study found 22% of vulnerable commits went completely undetected across five tools (oreilly.com).
– A survey of 105 engineering leaders found 61% shipped a production incident originating from AI-generated code within 90 days, despite that code passing review and unit tests (news.ycombinator.com/item?id=46312159).
– Human reviewer detection of logic errors also degrades sharply past roughly 400 lines in a single diff, indicating some of AI review's weaknesses aren't uniquely worse than human limitations in every dimension (news.ycombinator.com/item?id=46312159).
– A documented example of a real, long-lived design-flaw bug — Google's Gson library having a null-key handling issue that persisted through years of review — illustrates that intent-violation bugs can survive extensive human review too, not only AI review (oreilly.com).
Comparisons
AI code review vs. human code review. AI review scales to every diff and never tires, but plateaus around 50-60% recall on real bugs and structurally misses architectural/business-context issues. Human review catches more of the context-dependent categories but degrades on large diffs (past ~400 lines) and doesn't scale to reviewing everything. Neither is a strict superset of the other's coverage.
Claude vs. Copilot vs. CodeRabbit for code review accuracy. In the 67-bug benchmark's F1 scores, Claude (42.8%) meaningfully outperformed CodeRabbit (33%) and GitHub Copilot (22.6%); the benchmark's own publisher, Entelligence, reported its own tool scoring highest (47.2% F1) — a real conflict-of-interest caveat worth flagging even though the underlying bug set and methodology appear to be genuine — so treat any single vendor-published benchmark's ranking claim with appropriate caution and check current, independently-run benchmark data before relying on any single tool's marketed accuracy.
Static analysis vs. AI code review. Traditional static analysis and AI-based review appear to converge on a similar 50-60% detection ceiling for security-relevant issues, per NIST SATE and the newer AI benchmarks respectively — suggesting the ceiling may reflect the inherent difficulty of the problem class more than a specific tool-generation limitation.
Real-world use cases
– Fast-moving startups use AI review as a first-pass filter to catch obvious style and security-smell issues before a human reviewer looks at a PR, reducing the mechanical load on senior engineers.
– Teams shipping AI-generated code at volume face a documented, elevated production-incident rate (61% within 90 days per the cited survey) even when that code passes both AI review and tests, making a dedicated human review step for business-logic correctness a practical necessity rather than an optional extra.
– Security-sensitive codebases combine AI review with dedicated static-analysis and manual security review specifically for authorization and injection-class issues, given the documented 50-60% ceiling on catching these with automated tools alone.
Common mistakes
– Treating a passing AI review as equivalent to a passing human review. The 49%-best-case recall figure from the 67-bug benchmark shows this substitution risks missing roughly half of real bugs.
– Using the same model family to generate and review code with no independent check. This risks confirmation bias at scale rather than genuine independent verification, per the shared-blind-spot argument above.
– Letting AI review comment volume dilute human attention. Noisy, non-deterministic AI review output can bury the few comments that actually matter.
– Assuming AI review understands architectural or cross-service context. It generally doesn't — breaking-change and coupling issues require context outside a single diff.
– Not verifying AI-generated code's referenced APIs and libraries actually exist as assumed. Hallucinated function signatures pass syntactic review cleanly and require execution or explicit lookup to catch.
– Skipping dedicated concurrency and security review because "the AI reviewer already checked it." Race conditions and authorization flaws are specifically among the categories AI review is weakest on.
Best practices
– Use AI code review as a fast first-pass filter for style, obvious anti-patterns, and known-bad-pattern detection — not as the sole review step for anything security- or business-logic-sensitive.
– Keep a human reviewer explicitly responsible for business-logic correctness, architectural fit, and cross-service impact — the categories the evidence shows AI review consistently misses.
– If possible, use a reviewing model that differs meaningfully in training or architecture from the generating model, to reduce shared-blind-spot risk.
– Add dedicated, execution-based or static-analysis tooling specifically for race conditions and security/authorization flaws, rather than relying on text-based AI review for these categories.
– Verify AI-generated code's referenced functions, libraries, and APIs actually exist and match the current codebase — don't rely on syntactic plausibility alone.
– Watch for automation complacency explicitly — periodically audit a sample of AI-approved PRs with a rigorous human pass to check whether the team's trust level matches the tool's actual catch rate.
– Tune or filter AI review output to reduce noise, so genuinely important findings aren't buried under low-value nitpicks.
Key takeaways
– Race conditions, security/authorization flaws, breaking API changes, and business-logic errors are the four recurring categories AI code review consistently misses, because none have a reliable textual signature in an isolated diff.
– The best AI tool in a 67-real-bug benchmark caught only 49% of bugs; a larger 200,000+ PR study found 50-60% field effectiveness — a real, reproducible ceiling, not a one-off result.
– Using the same (or closely related) model to both write and review code risks correlated blind spots rather than genuine independent verification.
– AI review is genuinely useful for style, common anti-patterns, and known-bad-pattern detection — the right framing is that it changes what a human reviews, not whether a human needs to.
– Automation complacency and review noise are real, human-side risks that compound the technical miss rate — a documented survey found 61% of teams shipped an incident from AI-generated code within 90 days despite it passing review and tests.
Relevant tools.scult.in resources
If your team is dealing with structured data, config files, or API payloads as part of your review or debugging workflow, the JSON Formatter & Validator on tools.scult.in is a quick way to catch malformed structures before they become a harder-to-trace bug. For prompt patterns that help structure AI-assisted development and review work more deliberately, see the Claude, Cursor, and GitHub Copilot prompt libraries.
If your team is scaling up AI-assisted development and wants the human-review, testing, and architecture-review layers built around it properly rather than bolted on after an incident, that's the kind of gap SCULT's custom software development and AI agents & automation services are built to close — worth a conversation if this article's miss categories sound familiar from your own incident history.
Frequently asked questions
What is AI code review?
An automated process where an AI model (often integrated into a CI/CD pipeline or PR workflow) analyzes code changes and generates comments about potential bugs, style issues, or security concerns.
What bugs does AI code review commonly miss?
Race conditions, security/authorization flaws, breaking API changes, and business-logic/intent errors are the most consistently documented miss categories.
How accurate is AI code review, in real numbers?
In a benchmark of 67 real production bugs, the best tool caught only 49% (F1 up to 47.2%); a larger 200,000+ PR study found 50-60% field effectiveness.
Can AI code review catch race conditions?
Generally no, reliably — race conditions depend on runtime timing behavior that a static, text-based review can't observe.
Can AI catch security vulnerabilities in code review?
Partially — it catches well-known, textually recognizable security-smell patterns but plateaus around 50-60% detection on harder security-relevant issues, similar to traditional static analysis tools.
Why does AI code review miss business-logic errors specifically?
Because business intent lives outside the code itself (in tickets, specs, or conversations), and a reviewer analyzing only the diff has no access to what the code was actually supposed to do.
Is AI code review reliable enough to replace human review?
No — the evidence consistently shows it catches roughly half of real bugs at best, missing categories that require runtime behavior, system context, or business intent that human reviewers can access but AI review currently can't.
Why is a same-model generate-and-review setup risky?
Because a model reviewing its own (or a closely related model's) output shares its training-derived blind spots, risking correlated rather than independent verification.
Does AI code review change what humans review?
Yes — it's more accurate to say it shifts what a human focuses on (toward architecture and business logic) than to say it eliminates the need for human review.
What is "automation complacency" in this context?
A documented tendency for teams to trust a tool's roughly 50% catch rate as if it were much higher, simply because the tool looks competent on the bugs it does catch.
Do AI code review tools generate false positives?
Yes — practitioners report significant noise from speculative or nitpick comments, sometimes varying non-deterministically across repeated runs on the same diff.
Can AI code review understand a codebase's architecture?
Not reliably — it primarily analyzes the diff and immediate surrounding context, missing duplicate code, cross-module coupling, and separation-of-concerns issues that require broader system knowledge.
Does AI-generated code lead to more production incidents even after passing review?
Yes — a survey of 105 engineering leaders found 61% shipped a production incident from AI-generated code within 90 days, despite it passing review and tests.
Do AI reviewers detect hallucinated or non-existent APIs?
Commonly no — code referencing invalid functions or libraries can look syntactically plausible and pass review, since verifying existence requires checking against the real, current codebase rather than pattern-matching text.
How does human reviewer performance compare to AI on large diffs?
Human detection of logic errors also degrades sharply past roughly 400 lines, so AI's weaknesses aren't uniformly worse than human limitations across every dimension.
What is a "design flaw" or "intent violation" bug?
Code that's syntactically correct and passes tests but solves the wrong problem — for example, correct-looking output that doesn't match what was actually intended, a category neither automated review nor casual human review reliably catches.
Why do static analysis tools plateau around 50-60% detection?
NIST SATE evaluations and a 2024 ISSTA study attribute this to the inherent difficulty of the vulnerability classes being tested, not a fixable tooling gap specific to one product.
Is this miss-rate problem specific to any one AI code review tool?
No — the pattern shows up across multiple tools and studies (Claude, GitHub Copilot, and traditional static analysis all measured in the plateau range), suggesting a structural limitation rather than one product's flaw.
What tools are teams actually using for human review of AI-assisted code?
A mix of AI-reviewer bots (CodeRabbit-style tools) plus continued manual review discipline; practitioners are split on whether dedicated review tools add meaningful value over prompting a general-purpose model directly.
Is there an "AI code review bubble" — is the category overhyped?
This is a live debate — a widely discussed Hacker News thread argues the category carries more hype than proven value at current accuracy levels, though proponents point to real time savings on the mechanical first-pass work.
How do I catch logic bugs that AI code review misses?
Keep a dedicated human review step focused specifically on business intent and architectural fit, since these require context an AI reviewer scoped to a diff doesn't have.
How do I set up AI code review without creating false confidence?
Explicitly communicate the tool's real catch rate (roughly 50%, per available benchmarks) to the team, and periodically audit a sample of AI-approved PRs with rigorous human review to check actual performance against that baseline.
How do I review AI-generated code effectively given these blind spots?
Prioritize checking business-logic correctness, cross-service impact, and whether referenced APIs/functions actually exist — the categories most likely to slip through both AI-generated code and AI-assisted review.
How do I reduce noise from AI code review comments?
Configure severity thresholds or filters where the tool supports them, and consider having a human triage AI comments before they reach the PR author, so genuinely important findings aren't buried.
How do I check for hallucinated APIs in AI-generated code?
Actually compile/run the code or explicitly look up referenced functions and libraries against your current dependency versions — text-level review alone won't catch this.
How do I catch race conditions that AI review misses?
Use dedicated concurrency-testing tools, stress tests under realistic load, and runtime analysis — static or text-based review isn't the right tool for this category.
How do I catch breaking API changes that pass AI review?
Use contract testing and dependency-impact analysis across services, since this requires visibility into consumers outside the diff being reviewed.
How do I structure a review process that compensates for AI's blind spots?
Layer AI review for fast, mechanical first-pass catches, add dedicated static/security analysis for security-class issues, and require human sign-off specifically on business-logic and architectural questions.
My AI code review tool approved a PR that later broke production — what should I check first?
Check which of the four miss categories (race condition, security/authorization, breaking API change, business-logic error) the bug falls into — this tells you whether it's a known, documented gap rather than a fluke.
Should I use a different model to review code than the one that generated it?
It's a reasonable mitigation for the shared-blind-spot risk, though it doesn't eliminate the broader structural miss categories (architecture, business intent) that apply regardless of which specific model is reviewing.
Are AI code review tools improving over time on these specific miss categories?
General capability is improving, but the available benchmark data doesn't show a clear trend of the four core miss categories (concurrency, security, breaking changes, business logic) being specifically resolved — treat any claim of a fixed ceiling as provisional and check current benchmarks.
Can retrieval-augmented AI review (giving the model more codebase context) fix the architectural blind spot?
It can help with some cross-module visibility, but the available evidence doesn't show this fully resolving business-logic or intent-based misses, since those require knowledge (product intent) that isn't necessarily present anywhere in the codebase itself.
Does adding more automated tests reduce the risk from AI code review's blind spots?
Partially — tests catch behavior that diverges from expected output, but the survey finding that 61% of teams shipped incidents despite passing tests and review shows tests alone aren't sufficient either.
Is there a way to quantify how much an AI reviewer is actually catching on my own codebase?
Yes — periodically sample AI-approved PRs for rigorous human re-review and track your own team's effective catch rate over time, rather than relying solely on general industry benchmarks.
Does the same-model blind-spot problem apply if I use two different AI review tools from different vendors?
It's reduced but not eliminated — different vendors' models may share less training data overlap, but underlying model families and training approaches can still converge on similar blind spots.
AI code review vs. human code review — which catches more bugs overall?
Neither dominates cleanly — AI review scales but plateaus around 50-60% recall and misses context-dependent categories; human review catches more context-dependent issues but degrades on large diffs and doesn't scale to full coverage.
Claude vs. Copilot vs. CodeRabbit — which is most accurate for code review?
In the one available head-to-head benchmark's F1 scores, Claude (42.8%) outperformed CodeRabbit (33%) and GitHub Copilot (22.6%) on real bugs, though the benchmark was published by Entelligence, whose own tool scored highest (47.2% F1) — a conflict-of-interest caveat worth weighing alongside the numbers.
Static analysis vs. AI code review — is AI strictly better?
Not clearly — both converge on a similar 50-60% detection ceiling for security-relevant issues in the available research, suggesting AI review hasn't yet clearly surpassed traditional static analysis on this specific measure.
Is a dedicated AI code review product better than just prompting a general-purpose model like Claude directly?
Practitioner opinion is split — some report dedicated tools add workflow integration value (PR comments, CI gating) without necessarily higher raw accuracy than a well-prompted general model.
Do enterprise AI code review tools outperform free/open alternatives on these miss categories?
The available evidence doesn't show a clear, verified advantage specifically on the four core miss categories (concurrency, security, breaking changes, business logic) — pricing tier doesn't appear to be the deciding factor for these structural limitations.
My team's AI reviewer keeps approving PRs with obvious security issues — what's wrong?
Check whether the issue falls into the security/authorization category specifically — this is a documented, structural weak point (50-60% detection ceiling) rather than necessarily a misconfiguration.
My AI code review tool gives inconsistent results on the same diff across runs — is that expected?
Yes — practitioners have reported non-deterministic AI review output on identical diffs; this is a known limitation, not necessarily a bug in your specific setup.
My team has started ignoring AI review comments because there are too many — what should we do?
This is the documented noise problem; tune severity filters where available and consider a human triage step so real findings aren't lost in volume.
We had a production incident from AI-generated code that passed both tests and AI review — is this common?
Yes — a survey of 105 engineering leaders found 61% of teams experienced exactly this within a 90-day window, so it reflects a documented pattern rather than an isolated failure.
Our AI reviewer missed a breaking API change that broke three other services — how do we prevent recurrence?
Add contract testing and cross-service dependency checks, since this category requires visibility outside a single diff that text-based AI review structurally doesn't have.
What's the best AI code review tool to buy?
There's no single verified "best" tool independent of your stack and needs — evaluate based on your specific miss-category risk profile (e.g., security-heavy vs. business-logic-heavy codebases) rather than marketed accuracy claims alone.
Is it worth paying for CodeRabbit, Entelligence, or a similar dedicated code review product over using Claude or Copilot directly?
This depends on whether you value the dedicated workflow integration (PR-native comments, CI gating, team dashboards) enough to justify the cost — current, verified head-to-head accuracy comparisons across all of these specific products are evidence not sufficiently verified.
How much should a team budget for AI code review tooling?
Pricing varies too widely by tool and team size to state a single verified figure here — evidence not sufficiently verified; evaluate based on a trial against your own real bug history rather than list price alone.
Should a startup rely on AI code review alone to save on senior engineering review time?
Not advisable given the roughly 50% best-case recall on real bugs — use AI review to reduce the mechanical review burden, not to eliminate senior human review for business-logic and architectural correctness.
What's the single most important step to take if we're already using AI code review?
Explicitly assign a human owner for the categories AI review reliably misses — race conditions, security/authorization, breaking changes, and business-logic correctness — rather than assuming the AI tool has those covered.
Sources
- https://diffdojo.com/blog/ai-code-review-tools-miss-half
- https://www.oreilly.com/radar/ai-code-review-only-catches-half-of-your-bugs/
- https://news.ycombinator.com/item?id=49321400
- https://news.ycombinator.com/item?id=46766961
- https://news.ycombinator.com/item?id=46312159
- https://hackernoon.com/the-problem-with-using-ai-to-review-ai-written-code
- https://arxiv.org/pdf/2603.25146
Need this built into your business?
The free tools and prompts on this site handle the small, solved problems. If what you need is bigger — custom software, built and maintained for you — that's Scult's day job.
