AI-generated security patches fail to properly fix vulnerabilities 74% of the time, 1Password study finds

Aug 7, 2026

A study by 1Password's Off-By-1-Labs reveals that AI-generated security patches fail to properly fix software vulnerabilities 74% of the time. Evaluating 6,080 patch attempts by ChatGPT 5.5 and Claude Opus 4.8 across six complex vulnerabilities, researchers found that only 26% of fixes were fully successful. The remaining attempts failed to close exploit paths, altered application behavior, or introduced new security flaws. While AI patch generation is cheap, experts warn that the high cognitive load of manually verifying these often-fragile fixes means skilled human oversight remains essential.

AI-generated patch failure rates

▪The 1Password study evaluated 6,080 patch attempts generated by frontier AI models ChatGPT 5.5 and Claude Opus 4.8 across six recently disclosed open-source software vulnerabilities.
▪AI-generated patches introduced a new security vulnerability or exploit path in approximately 2.2% to 2.3% of the tested cases.
▪A study by 1Password's security research team, Off-By-1-Labs, found that AI-generated security patches failed to properly fix software vulnerabilities 74% of the time.
▪Only 26.0% of the AI-generated patches completely resolved the targeted vulnerabilities without introducing unintended side effects or altering application behavior.
▪The 1Password study found that 20.1% of the AI-generated patches resolved the vulnerability but altered the application's typical behavior.
▪The 1Password study showed that 49.3% of the AI-generated patches failed to fix at least one existing exploit path.

Common AI patching defects

▪More than one-third of the AI-generated patches rated as successful were flagged as fragile because they blocked specific exploit symptoms rather than addressing the underlying root cause.
▪AI models frequently exhibited 'tunnel vision' by focusing narrowly on the specific exploit demonstrated in a proof-of-concept while overlooking similar vulnerable code paths elsewhere.
▪For the Chromium vulnerability (CVE-2026-8512), AI models routinely secured only one part of the memory handling process while leaving the second critical component unguarded.
▪When patching the Linux kernel Copy Fail vulnerability (CVE-2026-31431), roughly one-third of the patches from both AI models regenerated a flawed revert containing an off-by-one heap write.

Freenginx vulnerability case study

▪Both the Freenginx maintainers' patch and the rejected Patch the Planet patch introduced a new secondary flaw that allowed an ordinary client to crash the server.
▪In a test case involving a memory-related use-after-free flaw in the Freenginx web server, an AI-generated patch submitted through Patch the Planet was rejected by maintainers because it only repaired two of three vulnerable spots.
▪Off-by-1 Labs set an automated patcher using ChatGPT 5.5 to write 270 attempts for the Freenginx flaw, resulting in 114 attempts that closed the original hole but introduced a new bug in every single instance.

Prompt quality impact

▪Providing AI models with accurate remediation guidance resulted in a 65.0% fix-success rate, whereas providing incorrect guidance collapsed the success rate to 15.2%.
▪AI models frequently followed flawed prompt instructions even when tool execution data directly contradicted those instructions.
▪Providing richer, more detailed prompts improved the AI models' fix-success rates from 51.8% to 76.3% but did not reduce the likelihood of introducing new vulnerabilities.

Model performance variations

▪Claude cleanly fixed the Exim remote code execution bug (CVE-2026-45185) about 75% of the time, but achieved under a 1% success rate on the Gemini CLI trust-bypass advisory (GHSA-wpqr-6v78-jr5g).
▪The 1Password researchers noted that because the study evaluated one CVE per codebase, they could not separate whether the codebase or the specific bug caused the wide variations in model performance.

Need for human oversight

▪1Password researchers concluded that expert human review remains essential because the cognitive load of reviewing mostly incorrect AI patches can exceed the effort needed to write patches manually.
▪While an individual AI patch attempt and validation cycle is inexpensive, costing between $2.00 and $3.00, the primary expense lies in the manual engineering effort required to verify patch security.
▪1Password has released its testing and evaluation framework, named FLAWED, on GitHub to allow organizations to run local evaluations against their own codebases.