Only 0.229: What Cisco's New Agentic Security Benchmark Reveals About AI Agents and Vulnerability Detection

Every security team has the same backlog: thousands of known vulnerabilities, too few engineers, and a growing pile of AI tools promising to close the gap. The pitch is easy to believe. If an AI agent can write code, surely it can find the file that holds the bug.

A new benchmark from Cisco suggests that belief deserves a hard look. On October 23, 2026, Cisco researchers introduced agentic evaluation in FAITH, an open-source framework extension for testing AI agents on multi-step security tasks. Its first benchmark asks a deceptively simple question: given a repository, can an agent point to the files affected by a known vulnerability? The answer, so far, is “not reliably.”

This post covers what the benchmark measures, what the numbers say, and what they mean for any business planning to put AI agents near its codebase or security workflow.

What Cisco Actually Built

FAITH is Cisco Foundation AI’s open-source evaluation framework. Until now it handled question-answering style cybersecurity tests. The new extension lets it run agentic tasks too, where a model chooses an action, receives tool output, and decides what to do next, all inside one loop that FAITH manages and records.

The VLoc benchmark

The first agentic benchmark is VLoc, a vulnerability localization test. It has two phases:

Phase A: find the affected files in a repository that still contains the vulnerability. The metric is File-set F1, which balances finding the right files against naming wrong ones.
Phase B: look at the patched version of the repository and recognize that the fix is in. The primary metric is the true-negative rate, meaning how often the agent correctly says “nothing to see here.”

The benchmark covers 500 vulnerabilities across 290 repositories, six package ecosystems and 147 distinct weakness categories (CWEs).

Why the design matters

Phase B is the clever part. Many security tools look impressive because they flag a lot. A tool that cries wolf on every repository would ace Phase A and fail Phase B. Testing both phases measures what a real security team needs: accurate findings and a low false-alarm rate.

What the Numbers Say

The headline result is sobering. According to the benchmark page, the best-performing model, GPT-5.5, achieves only 0.229 File F1. The same page reports that 190 of the 500 tasks, or 38%, were not solved by any model tested.

A secondary write-up of Cisco’s results from Data Studios adds more detail, which you should treat as less authoritative than Cisco’s own pages:

Traditional static analysis performed worse on Phase A. The best static analyzer, Semgrep, reached an F1 of 0.086.
Agents ran with read-only terminal tools in an isolated container with networking disabled, a limited command budget, and results averaged over three runs.
Small, specialized models came close to the frontier model on Phase A, but some flagged almost every patched repository as still vulnerable.
A full Phase A sweep ranged from under a dollar for a small local model to over $100 for a frontier model through an API.

Cisco’s benchmark page also notes that performance falls 13 times from the smallest to the largest repositories. That matters, because enterprise codebases are rarely small.

How to read an F1 of 0.229

An F1 score runs from 0 to 1. A score of 0.229 does not mean the agent is “23% accurate.” It means that the balance of precision and recall on the files it names is low. In practice, the agent will often miss the right file, name wrong ones, or both. For a developer triaging a real vulnerability report, that is a starting hint, not a verdict.

This is a hard task by design: the agent works with limited tools and no internet access, and real vulnerabilities often span unfamiliar code. But hard is exactly what production security work looks like.

Why This Matters Beyond Security Teams

You do not need to run a security operations center for this to affect you. Three business consequences follow.

1. Autonomy claims need task-level evidence

Vendor demos show agents solving curated problems. A benchmark like VLoc shows what happens on 500 messy, real vulnerabilities. We made a related point when we looked at why one success is not reliability in AI agents: a single good run says little about the tenth or the hundredth.

2. The false-alarm problem is a cost problem

An agent that flags clean code wastes engineer hours and erodes trust. Phase B exists because a noisy tool quickly gets ignored, and an ignored security tool provides no protection.

3. Humans become reviewers, not bystanders

When automated output is unreliable, the human role shifts. We explored this pattern in our piece on on-call engineers becoming approvers. The benchmark reinforces it: an agent can narrow the search, but a person still has to confirm before anything ships.

The Governance Gap Around Security Agents

Weak accuracy would be less worrying if agents were tightly controlled. Many are not. Security startup Opsin, which raised $21 million in October 2026, reported from its own analysis of customer environments that 93% of misaligned agent tools had unrestricted access to systems and only 1 in 200 required human sign-off before acting. Those are vendor-reported figures and should be read that way, but the direction matches what other surveys describe. See our look at how many organizations lack control of their agents.

Put the two findings together: tools that are not yet accurate at security tasks, deployed with broad access and little oversight. That is the risk combination to avoid.

A Practical Playbook

Here is how to apply the lessons without abandoning AI-assisted security.

Evaluate on your own code

Public benchmarks are a floor for scrutiny, not a substitute for testing. Take a sample of vulnerabilities you have already fixed, hide the answer, and see whether the tool finds the right files. Frameworks like FAITH are open source, so you can run similar evaluations internally.

Measure both directions

Track missed findings and false alarms separately. A tool with a high detection rate and a high false-positive rate may cost more than it saves.

Keep access read-only by default

The benchmark agents worked with read-only tools. Mirror that. Give a triage agent permission to look, not to change, merge or deploy, until it has earned more trust.

Require human sign-off on consequential actions

Treat agent output as a lead. A named engineer should approve any change that touches production code or security configuration.

Re-test when models change

Scores move with every model release. Build evaluation into your upgrade process rather than assuming a newer model is a safer one.

Conclusion

Cisco’s benchmark does not say AI has no place in security. It says that today’s agents, even the best, are far from dependable at one of the most basic tasks in the job: locating the vulnerable code. That is useful, honest information for anyone budgeting for automation.

Actionable takeaways:

Ask any AI security vendor for task-level results, including false-positive rates, not just demo footage.
Run a small internal evaluation using vulnerabilities you have already fixed.
Start agents in read-only mode and widen access only with evidence.
Keep a human approver on every consequential change.
Re-run your evaluation whenever the underlying model changes.

Promact helps teams design, evaluate and govern AI agent workflows so that automation adds speed without adding blind spots. If you are weighing where agents belong in your engineering or security process, we would be glad to help you test before you trust.

Frequently Asked Questions

What is Agentic FAITH?

It is an extension of Cisco Foundation AI’s open-source FAITH evaluation framework. It lets teams run and score multi-step, tool-using AI agent tasks alongside traditional question-answering security benchmarks using one set of commands.

What does the VLoc benchmark test?

VLoc tests whether an AI agent can find the files affected by a known vulnerability in a repository, and whether it correctly recognizes the same repository as clean after the fix is applied.

Does a 0.229 F1 score mean AI is useless for security?

No. It means the best tested agent is unreliable at this specific, difficult task under constrained conditions. AI can still help with triage and drafting, but its output needs human verification.

Can I run these evaluations myself?

FAITH is open source and published by Cisco Foundation AI on GitHub, with documentation at faith-evals.com. Check the repository for current licensing and setup instructions before adopting it.

How should businesses use AI agents in security workflows today?

Start with read-only, assistive roles, measure both missed findings and false alarms on your own code, and keep a human approver on any action that changes production systems.

Sources

Cisco Blogs: Introducing Agentic Evaluation in FAITH - Cisco’s October 23, 2026 announcement of the agentic extension and the VLoc benchmark
VLoc Vulnerability Localization Benchmark - Benchmark page with task counts and headline scores
Cisco Foundation AI: FAITH on GitHub - Open-source evaluation framework repository
FAITH Documentation - Setup and usage guides, including the agentic quickstart
Data Studios: Cisco Agentic FAITH and VulnLoc - Secondary write-up with additional model and cost detail
Pulse 2.0: Opsin Raises $21 Million - Funding announcement and vendor-reported agent access statistics