The Verification Gap: What Five Autonomous Pentest Agents Claim vs. What They Actually Produce

A code-level teardown of PentestGPT, Strix, PentestAgent, Shannon, and PentAGI, and the one hard question the whole category is still working on: does it prove a finding, or assert one?

July 12, 2026 · 33 min read · AI Security LLM Agents Penetration Testing AppSec

Every “autonomous pentest agent” shipping today sells the same sentence, in some form: it finds and confirms real vulnerabilities. Not scans. Not suggests. Confirms. The entire pitch rests on that one verb. The promise is that the false-positive tax, all those hours a human burns triaging a scanner’s noise, is finally gone, because the agent validates its own findings with working exploits before it ever hands them to you.

I re-cloned five of these tools and read their code to see how close they actually get. They span the category: an academic assistant (PentestGPT, USENIX‘24) since rebranded as autonomous; two commercially-flavored “AI hacker” products (Strix, PentAGI); a focused academic exploitation tool (PentestAgent, AsiaCCS‘25); and a white-box exploit-prover from a security startup (Shannon). Five teams, three languages, subtly different jobs, and a great deal of genuinely good engineering between them.

They also share one honest limitation, and it is the single most useful thing to understand about all of them: none of the five yet confirms a finding with an independent programmatic signal. Every one ultimately trusts the language model’s own account of whether an exploit worked. The best of them wraps that self-report in a beautifully typed schema with referential-integrity checks, and it is still a self-report.

Here is the whole story in one picture. Follow any tool from the claim at the top down to the finding at the bottom, then read the box below it: the checks that would turn an assertion into proof, which the field has not standardized yet.

This piece is about that gap: the distance between the confident language of “validated findings, not false positives” and the code that decides what gets written into the report. It is where most of the interesting story lives, and if you are choosing one of these tools or building your own, it is the part that decides whether you can trust the output unattended.

How to read a claim

Before the teardowns, the method, briefly, because it is what keeps this fair.

For each tool I separated three layers and never let them collapse into each other: (a) what it claims (README, paper, marketing), (b) what the code does, and (c) what it produces on a target. Most of the interesting findings are the deltas between these layers: a README that describes an iteration loop the code does not contain, a “proof of concept” field that is never executed, a “RAG knowledge base” that is really live web scraping.

I judged each tool at the job it was built for first, and only then against a common “autonomous agent” yardstick. This matters for two of the five. PentestGPT was designed as an advisory assistant with a human in the loop, so I judge its current autonomous rebrand against its current README while crediting the advisory tool it actually is. PentestAgent explicitly disclaims red-teaming, chaining, and post-exploitation, so I evaluate it strictly at single-known-CVE exploitation and do not penalize it for scope it deliberately excluded.

The single lens I care about most is verification. For each tool I traced the exact path by which a candidate becomes a reported finding, and asked whether any signal the model cannot fake from inside its own transcript is required to confirm it: an out-of-band callback, a control-versus-injected differential, an observed JavaScript execution, a cross-session access diff. That is the difference between proof and assertion, and between a tool you can trust unattended and a tool that produces excellent leads for a human to check.

Everything below is grounded in code at a pinned commit. Where I could not verify something by execution, I say so.

PentestGPT: a research contribution, and an autonomous rebrand

Built for: interactive, human-in-the-loop pentest advice (USENIX Security ‘24). Now claims: “AI-Powered Autonomous Penetration Testing Agent.”

PentestGPT is really two tools sharing a repository, and the distinction is the whole story. The pentestgpt_legacy/ package is the genuine article from the paper: three persistent LLM sessions (a reasoner, a generator, a parser), coordinating around a Pentesting Task Tree, a structured external memory of the engagement that the reasoning session updates and then consults through a separate task-selection call. The parser is explicitly instructed, “You only summarize. You do not conclude.” It is a thoughtful design, it earned its citation, and the recent work on it (a clean multi-provider LLM layer) is real.

The flagship pentestgpt/ package, the one the README calls an autonomous agent, is a different shape. Its actual runtime, verified end to end, is a fixed, linear, three-stage pipeline (core/pipelines.py). For a pentest that is asset-identification, then vulnerability-identification, then report; for CTF it is recon, then exploit, then walkthrough. Each stage spins up a fresh backend that shells out to the Claude Code CLI as a subprocess:

claude -p <prompt> --output-format stream-json --verbose \
  --system-prompt <sys> --permission-mode bypassPermissions --model <model>

That is core/backend.py:169-188. The autonomy you see when you run it is real, but it is Claude Code’s autonomy, invoked three times; PentestGPT’s own contribution is the stage decomposition and the prompts. Cross-stage memory is the previous stage’s output truncated to 4000 characters and spliced into the next prompt (prompts/stages.py:138-139). There is no shared session between stages, and, the load-bearing part, there is no loop. The README’s centerpiece “What’s New in v1.0 (Agentic Upgrade)” describes an iteration loop that “runs continuously, maintains a context file, and terminates on flag capture or max iterations.” None of that exists in the code yet. max_iterations defaults to 300, is read nowhere at runtime, and the documented --max-iterations 5 command-line flag is not defined at all, so the example invocation in the README would error. The pentestgpt_context.md file the docs describe is never written or read. The Pentesting Task Tree, the paper’s actual contribution, is absent from the flagship (grep for it and you get zero hits outside pentestgpt_legacy/).

Verification. Here is how the flagship decides it has captured a flag: it runs a set of regular expressions over the model’s own narration (core/controller.py:343-358). The patterns include the expected (flag{...}, HTB{...}) alongside a very broad [A-Za-z0-9_]+\{[^\}]+\} that matches any word{...} construction, and a bare \b[a-f0-9]{32}\b that matches any MD5 or 32-hex token the model happens to print. Under case-insensitivity the specific patterns are redundant with the generic one. A regex hit becomes a captured flag. The project’s own integration test shows the mechanism plainly: a mock backend that merely says the string "Found flag{test123}!" produces a recorded capture, with no target and no execution. Grep the flagship for submit, oracle, collaborator, oob, or differential and you get nothing. The PENTEST report stage then compiles Stage-2’s asserted vulnerabilities into CVSS-scored “Confirmed” findings, using an instruction to “estimate CVSS scores where possible.” So an unverified Stage-2 claim can arrive in the report as “Critical, CVSS 9.8.”

Genuinely good: the ideas in the legacy package are worth borrowing and largely absent from the newer tools. A typed task tree as external memory is exactly what these agents need at long horizons. A summarizer placed in front of the reasoner, compressing noisy tool output before the expensive reasoning step and forbidden from drawing conclusions, is a clean pattern. The BFS-for-pentest, DFS-for-CTF split is a real strategic insight.

Where it can go next: the highest-leverage addition is a FindingVerifier stage between exploitation and report that refuses to mark anything “Confirmed” without an execution-grounded signal, such as a flag read from a tool result rather than model prose, or an out-of-band callback for a web finding. And regenerating the docs from the code, plus a CI doc-lint that greps documented symbols against the tree, would catch the drift above.

Intent-aware verdict: as the advisory PTT assistant it was designed to be, PentestGPT is genuinely useful, and its lack of an automated oracle is by design, because a human is the oracle. As the autonomous agent the flagship now markets, it is a thin three-stage wrapper whose findings are not independently checked, and whose documentation currently describes features the code does not contain. The research ideas are the durable part; the flagship’s autonomous findings are best read as leads.

Strix: a serious exploitation engine, and the last mile to proof

Built for / claims: a “fully autonomous AI hacker” that finds and confirms vulnerabilities, promising “Real exploit validation, working PoCs, not false positives.”

Strix is the most serious piece of autonomous-exploitation engineering in this group, and it makes the verification question especially vivid precisely because everything around it is done so well. It is built on the OpenAI Agents SDK with LiteLLM; its own IP is a coordinator, a 54-document skills library, a Caido-based proxy layer, a dedup/reporting pipeline, and a Textual TUI. The agent loop (core/execution.py, run_agent_loop then _run_cycle, not the “performAgentChain” that some summaries name) wraps the SDK runner with a default max_turns=500 and a lifecycle-gated termination: the run only ends when the model calls finish_scan with success=true, and a text-only turn triggers a force-continue recovery loop that re-injects “call exactly one tool.” Sub-agents are spawned as async tasks in the same process.

The standout engineering is the proxy design, and it is genuinely worth stealing. Strix exports http_proxy/https_proxy/ALL_PROXY into /etc/environment, profile.d, wgetrc, and every shell in the sandbox (containers/.../docker-entrypoint.sh). The consequence is that every child process the agent spawns is transparently captured by the proxy, with no per-tool integration required. The captured traffic is queryable from inside the sandbox via an importable proxy API and an HTTPQL query language. This is the correct shape for a tool feedback loop: structured, universal, low-friction. It is the best single idea in any of the five codebases.

Verification. This is where the shared gap shows up. The README sells “working PoCs, not false positives.” When the model reports a vulnerability, it must fill a required poc_script_code field, described to it as “the actual exploit/payload.” I traced every use of that field across the codebase. It is checked for exactly one thing, non-emptiness. It is then stored to vulnerabilities.json, rendered as a Markdown code block, represented in SARIF as a single script_available: true boolean (the body is deliberately withheld, which is a responsible choice), and syntax-highlighted in the CLI and TUI. That is the complete list of its destinations. It is never executed. There is no run_poc, no run_exploit; grep for subprocess, exec, eval, Popen, shell=True and none of the hits touch poc_script_code. The dedup step does not even compare it. So the “working proof of concept” that is meant to distinguish Strix from a false-positive-prone scanner is a string that flows to the report and a syntax highlighter, and is never handed to an interpreter.

So how does a candidate become a finding? Field-presence checks, some CVSS/CVE/CWE regex, and a single LLM dedup call. That dedup is one O(n) call that can quietly drop a real finding it judges a duplicate, with an instruction not to retry. The “validation subagent” some people imagine is not an independent adjudicator: it is an ordinary child agent, same model, same skills, seeded with a verbatim JSON dump of the parent’s history, filing findings through the same tool. An out-of-band client (interactsh-client) ships in the image, but it is a technique the model may choose to use, wired to no finding gate.

Where it can go next: the shortest path to proof here is remarkably short, because Strix already has a sandbox, a universal capture proxy, and an OAST client. Execute the poc_script_code the model is already required to write, and accept the finding only if an interact.sh token fires or a Caido replay shows a control-versus-injected differential. That one change would move Strix from asserting to proving for a large class of bugs. Secondary opportunities: compact context (there is zero compaction at 500 turns), give children a structured brief instead of a verbatim history dump, and enforce scope at the network layer rather than in the prompt, since scope is currently advisory (“empty allowlist = allow all”) while the sandbox carries NET_ADMIN/NET_RAW.

Intent-aware verdict: judged at its own “find and confirm” bar, Strix delivers strongly on find and is still landing confirm. The find half is autonomous and well-built. The confirm half is the part still to arrive: no reported finding carries an independent signal, and the “working PoC” is not run. It is a powerful exploitation engine whose reporting is assertion-grade today, and it is one focused sprint from becoming the first tool here that proves its findings.

PentestAgent: a reusable scoring idea, and self-reported success

Built for: automated exploitation of single, known-CVE VulHub targets. Explicitly disclaims red-teaming, chaining, and post-exploitation. (This is nbshenxm/pentest-agent; a same-named GH05TCREW/pentestagent is a different, unaffiliated tool.)

Evaluated at its actual, deliberately narrow task, PentestAgent is a research prototype with one genuinely reusable idea and two places where the README says more than the code does. Architecturally it is three manually-chained scripts (recon, planning, execution) with no orchestrator; a human edits a YAML config and runs them in sequence. That is appropriate for a research artifact, and I do not hold the absence of a controller against it.

The idea worth stealing is its exploitability-scoring rubric. It decomposes “how exploitable is this CVE” into roughly eleven orthogonal, LLM-assessed features: vulnerability type (weighted from code-exec at 10 down to DoS at 2), remote-versus-local, an explicit exploit-maturity ladder (None, PoC, PoC+, Exploit, as a 0.3 to 1.0 multiplier), and six attack-complexity sub-features. It combines them with fixed weights in a deterministic formula, bins the result into easy, medium, hard, and validates the whole thing against CVSS and EPSS using Cohen’s kappa (the computation is in the code at utils/cve_info.py:300, though the reported kappa values are in the paper only). This is a legitimately good pattern: it turns a fuzzy judgment into a decomposed, inspectable, tunable score with orthogonal features and a validation methodology. You could reimplement it on its own tomorrow.

The one caveat on the implementation is cost: scoring one candidate exploit repo takes about 17 sequential LLM calls, and a single CVE can surface several repos, so a three-repo CVE runs 50 or more calls before it is even ranked. The idea scales; this implementation is expensive. The fix is one structured, function-calling request that returns all eleven features at once, keeping the deterministic combiner.

On the “RAG.” The README advertises “RAG over a pentest knowledge base” and an index directory “to store vector indexes.” There is no embedding-based semantic retrieval in the code; grep for VectorStoreIndex returns zero hits. What actually happens is live GitHub and ExploitDB scraping per CVE, summarized with a LlamaIndex SummaryIndex and looked up through a SimpleKeywordTableIndex keyword match. The only “embedding” tokens in the codebase are counters that read zero because no embedding model is ever invoked. It is a perfectly defensible design of scrape-and-summarize with a keyword index; it is just not what the word “RAG” usually implies.

On verification, the crux. How does PentestAgent decide an exploit succeeded? After running model-proposed shell commands until the model emits a stop token, it sends an EXECUTION_SUMMARY prompt asking the same model for {"summary", "successful"}, where “the ‘successful’ field should be TRUE if the exploit executed successfully.” Actor and grader are the same model on the same transcript. There is a config field that looks like a real oracle, command_to_execute, defaulting to touch /tmp/success, but it is currently dead code: read once at execution_agent.py:138 and never referenced again. Nothing runs it; nothing checks whether /tmp/success exists. And the exploit commands execute on the operator’s own host, so a PoC can “succeed” without the target changing state at all. (The advertised “voting” for self-consistency is also inactive: every caller passes no_vote=1, so it always returns after a single sample.)

Intent-aware verdict: at single-CVE exploitation, this is a plausible prototype whose scoring rubric is a real, publishable contribution. The exploitation half cannot yet prove exploitation, because it asks the model whether it won and the one knob designed to be a ground-truth check is not wired in. Borrow the rubric with confidence; treat the success numbers as provisional until an independent oracle replaces the self-report. (One housekeeping note for the authors: partial substrings of real OpenAI API keys are committed in utils/cve_info.py as model-selection fingerprints. They are partial, not full, but worth scrubbing from history.)

Shannon: the strongest verification framing of the five

Built for: autonomous white-box (source-required) proof-by-exploitation for five vulnerability classes: injection, XSS, SSRF, broken authentication, broken authorization. Black-box is reserved for the closed Keygraph platform.

Shannon is the tool that takes verification most seriously, and it is worth studying precisely because it gets so much right and still stops just short of independent proof. It is a durable Temporal workflow that runs five vuln-analysis-then-exploitation pipelines in parallel, each agent a Claude Agent SDK instance (maxTurns: 10_000, bypassPermissions), with the target repository mounted read-only and only narrow .shannon/ scratch directories writable. The containment posture (source immutable to a bypass-permissions agent, ephemeral per-scan workers on a durable backbone) is the best of the five.

Three things about its verification framing are genuinely excellent and directly reusable:

1. A typed, falsifiable hypothesis format. The analysis phase emits an exploitation_queue (ai/queue-schemas.ts) that forces the model to name the taint source, the sink call, the slot_type, the sanitizer it observed, the post-sanitizer concatenation, why it thinks there is a mismatch, and a minimal witness payload. That is a far more disciplined artifact than “the model found SQLi.”

2. A proof ladder that encodes the right distinction. The exploit prompts define four levels and state, in as many words, that “reflected in HTML is NOT JavaScript executed” and that the agent “MUST reach at least Level 3 with evidence to mark as EXPLOITED” (exploit-xss.txt:52-70). Someone who has actually triaged XSS wrote that.

3. Referential integrity via a Zod refinement. This is the cleverest single mechanism in any of the five. The exploit-collector builds its schema over the set of IDs the analysis phase actually produced, and pins the reported vulnerability_id with .refine((id) => validIds.has(id)) (mcp-server/exploit-collector.ts:105-119). The model cannot record a finding against an ID that does not exist in this run’s queue; a typo or hallucinated ID returns a retryable structured error listing the valid options, and unprocessed IDs are surfaced rather than silently dropped. It is real per-run referential integrity, and it is exactly the kind of constraint the other tools lack.

Where it stops short. Here is the precise line. Two programmatic gates exist. The analysis-to-exploit gate checks only that files exist, that the queue is valid JSON, and that the array is non-empty. The exploit-output gate, the .refine() schema, pins the ID, enforces the exploited-or-blocked discriminator, and requires the fields to be present. But proof_of_impact and exploitation_steps are validated as z.string().min(1): non-empty strings. Their content, the actual claim that the exploit worked, is authored by the same Sonnet agent that ran and graded the exploit. The per-agent exploit validator is a literal no-op (session-manager.ts:153-155: async () => true). No code anywhere re-executes the steps, diffs a control against an injected response, observes a real JS-execution signal, or catches an out-of-band callback. In the sample reports, the “Exploitation Steps” are copy-pasteable curl (a human can re-run them, which is more than the others offer), but the “Proof of Impact” is prose (“Successfully bypassed authentication…”) with a truncated JWT, not a captured artifact.

So the typing constrains shape, identity, and prompt adherence, but not truth. It is self-attestation, the best-framed in the group, and still self-attestation. Shannon’s own docs/safety.md says so plainly: “final reports can still contain weakly supported or incorrect details. Human review is essential.” That is the honest posture, and it is worth noting that it contradicts any “zero false positives” framing. To Shannon’s credit, that claim does not actually appear anywhere in the repository.

Where it can go next: close the last mile. The witness payload and typed queue are already there. Add a post-agent replay activity that re-runs exploitation_steps, captures the HAR, and gates status: "exploited" on an observed signal: a CDP console nonce for XSS, an OOB token for SSRF and blind injection, a response differential for SQLi. That single change would make Shannon the first genuinely proof-grade tool here. Secondarily, the externally_exploitable=true filter and the report-only-what-you-can-exploit design trade away recall quietly, because real bugs it cannot actively exploit simply do not appear; recording them as visible “candidates” would make that trade explicit.

Intent-aware verdict: at autonomous white-box exploit-proving, Shannon is the most intellectually honest and best-architected tool of the five, and the queue schema and refine-pinned IDs deserve to be copied widely. It enforces “proof by exploitation” as proof by well-typed assertion: the agent that runs the exploit also grades it, and the only programmatic check on the grade is “non-empty string, real ID.” It is one replay-and-observe activity away from deserving its own marketing.

PentAGI: exceptional observability, and the trust question

Built for / claims: a “fully autonomous” multi-agent system with “complete isolation” and “20+ professional security tools including nmap, metasploit, sqlmap.”

PentAGI is the most operationally mature codebase here, a Go backend of roughly 223k lines with a real Flow, Task, Subtask, agent-chain hierarchy and, unusually, genuine per-role model routing: thirteen configurable routing buckets, each independently pinned to a model, a reasoning setting, and its own price. Execution within a flow is strictly serial (one input channel, one worker goroutine); you scale by launching many flows. The reliability engineering is thoughtful: an argument-normalized repeat detector that canonicalizes tool calls to catch thrashing, a self-healing tool-call-argument fixer, result-size caps, and a Graphiti temporal knowledge graph for reusable methodology memory whose data anonymization is code-enforced (a deterministic replacer), not merely prompt-instructed.

Its observability is the best of the five by a wide margin: a hand-rolled, fully-typed Langfuse SDK threaded through every agent, chain, generation, and tool span; a complete OpenTelemetry stack (Grafana, Loki, Jaeger, VictoriaMetrics, ClickHouse); and live GraphQL streaming of a run. If you want to watch an autonomous pentest happen in real time, this is the tool.

Two things to know. First, the headline “20+ integrated tools.” If you grep the Go code (not the README, not the frontend) for those integrations, you find that the pentester agent’s entire registered toolset is a generic terminal, a file tool, a search tool, a browser, and some delegation helpers. The terminal is literally cmd := []string{"sh", "-c", command} (tools/terminal.go:177), returning raw TTY. The “sqlmap” grep hits are a database column-mapper variable named sqlMappers; the “nmap” hits are substrings of unrelated identifiers and argument-schema examples. Tool knowledge lives in a 433-line prose prompt that names sixteen tools as text. In fairness, the Kali binaries do ship in the sandbox image, and a capable model can run them through the shell, so this is a gap in integration depth (no structured adapter, no XML/JSON parsing, no msfrpc), not a claim that the binaries are missing. But “seamless integration with 20+ tools via function calling,” repeated five times in the README, is not what the code does. And the missing structured-result layer is exactly what a verification oracle would need to consume.

Second, there is no verification step yet. The terminal returns raw TTY, truncated to 16 KB and LLM-summarized if longer, so the evidence is already a lossy paraphrase before anyone reasons over it. The closing reporter template presents itself as a “TASK EXECUTION EVALUATOR” doing “critical analysis” and “independent judgment,” which reads convincingly, but mechanically it only reads the LLM-authored execution log and emits a boolean. It never re-runs anything. A hallucinated shell result or an over-claimed finding flows unverified into the subtask result, into the execution log, and out through a reporter that certifies it “SUCCESS.”

The safety posture compounds this. For a system executing attacker-chosen sh -c, the container adds NET_RAW unconditionally, drops no capabilities, sets no seccomp or AppArmor profile, imposes no pids, memory, or CPU limits, and the orchestrator’s compose file mounts the host Docker socket and runs as root. The pentester prompt is a jailbreak-style “AUTHORIZATION FRAMEWORK” instructing the model to “never request permission” and “proceed immediately and confidently.” Cost is meticulously tracked and aggregated, but never capped; only iteration limits halt a flow.

Where it can go next: the observability that makes PentAGI special is telemetry, not control. The one in-loop corrective (the “mentor”) triggers on thrashing (call counts) rather than on being wrong, and is off by default. The opportunities are clear and mostly orthogonal to the good parts: add a verification oracle that re-tests each claimed finding before it is marked successful; build the structured tool adapters the marketing already promises (and feed their parsed output to that oracle); harden the sandbox (drop caps, add seccomp, set resource limits, stop mounting the host socket); and turn the cost it already tracks into a hard budget cap that halts the run.

Intent-aware verdict: the “fully autonomous multi-agent” claim is substantiated, and this is real, careful engineering. The “20+ integrated tools” and “complete isolation” claims are not yet matched by the code. It is the observability and reliability standout of the field, around a core whose trust model is still “the model said so.” Today it is a superb autonomous co-pilot for a consenting lab, and the path to the self-verifying production system its marketing describes is clear.

Benchmarks are not ground truth

Four of these tools cite a headline percentage. Before you let any of them into a procurement decision, look at what the benchmark’s oracle actually is, because it determines what the number can possibly mean.

The recurring trap is that CTF and flag benchmarks have a built-in oracle, and that is exactly why they mislead here. A captured flag is unambiguous, scriptable success, which is wonderful for measuring whether an agent can reach and exploit a planted bug. But on a single-flag target, “did you get the flag?” imposes no penalty for also emitting three findings that are not real. Real applications have no flag; the cost of running one of these agents is the human triage of its output, and that cost is dominated by false positives, precisely the quantity a flag oracle cannot see. A tool can score 96% on XBEN and still produce a lot of noise on a real app.

Two subtler points. First, the oracle lives in the harness, not the tool. Strix’s 96% is real, but the flag check happens in XBEN, not in Strix; in production there is no harness, so the tool falls back to its own (self-attesting) verification, which means the benchmark measures a capability the shipped tool does not have on real targets. Second, watch for self-curated and unreproducible numbers. Shannon’s 96.15% and its “~85% for top agents and humans” comparison do not appear anywhere in the repository (I grepped the pinned commit; there is no harness and no results directory), and the variant is described as source-aware, which makes the comparison to black-box baselines apples-to-oranges by construction, since source access is strictly more information. PentestAgent’s numbers are likewise absent and deferred as stale, and because VulHub images advertise their CVE, the tool may simply be recalling a public exploit, a confound that rigorous VulHub-derived benchmarks control for by anonymizing the images, which PentestAgent does not.

None of this makes the tools worthless. It means every percentage in this space should be read as a claim about reachability on planted targets, not as real-world efficacy, and not as evidence of a low false-positive rate.

The common yardstick

Scoring all five on one “autonomous pentest agent” rubric, 1 to 5. The apples-to-oranges warnings are load-bearing: PentestAgent disclaims the autonomy and scope this rubric asks for, so its low Autonomy, Scalability, and Tool-depth scores reflect “not built for this axis,” not failure; and PentestGPT’s Verification score judges its autonomous rebrand, not the advisory tool it originally was.

*judged at single-CVE exploitation, its stated task, not as a red-team agent.

The verification column is the one that matters, and it is where the whole field is still early: the best score is a 3, three of five score 1, and nobody scores 4 or 5, because nobody has an independent oracle. The rest of the table is a study in how easy it is to over-value the wrong things. PentAGI pairs the best observability (5) with the lowest verification (1): you can watch a confidently-wrong run in beautiful real time. Strix pairs the deepest toolkit (5) with an assertion-grade pipeline (2), a reminder that depth of action is not depth of proof. Autonomy and observability are easy to score high, and they say nothing on their own about whether you can trust the output.

If you are building one: what good looks like

If you are building a pentest agent, the fastest way to avoid the mistakes above is to design for eight properties. No tool here has more than three or four, and, tellingly, none satisfies the first two, which are the two that decide trust.

1. An independent verification oracle. A signal the model cannot fake: an out-of-band callback, a control-versus-injected differential, a real JavaScript-execution event, a cross-session access diff. This is the missing piece in all five. Shannon’s typed queue is the scaffolding it would plug into.
2. Precision and recall measured on real-app-like targets, not CTF flags. No tool measures this yet, and the benchmarks in circulation cannot.
3. Scope and safety enforced at the network layer, not by prompt. Shannon’s read-only source mount is closest; Strix, PentAGI, and PentestAgent rely on prompt instructions.
4. A durable, typed world model at long horizons. Shannon’s typed queue is closest; Strix does no context compaction at 500 turns, and PentestGPT truncates to 4000 characters between stages.
5. Structured tool adapters with real feedback loops, not sh -c and raw TTY. Strix’s capture proxy is the one strong example; parse tool output into typed results a verifier can consume.
6. A durable evidence chain: the actual request and response that justified a finding, not a prose summary. Strix captures the raw material but does not yet bind it to specific findings.
7. Horizontal scale with per-target isolation. Shannon’s ephemeral per-scan workers are right; Strix shares one container, and PentAGI serializes within a flow.
8. A hard budget cap that halts the run, not just an iteration limit. Strix and Shannon treat budget as a control; PentAGI tracks cost meticulously and never caps it.

The encouraging part is that these are mostly last-mile gaps, not research problems. Strix and Shannon already own the hard-to-build substrate, a universal capture proxy and a typed, ID-pinned finding queue, that an oracle would plug into. The distance from asserting to proving is an engineering sprint: execute the PoC you already collect, gate acceptance on an out-of-band callback or a control-versus-injected differential, and record the captured signal as the finding’s evidence. Whoever ships that first has the first genuinely trustworthy tool in the category.

Which tool for which job

• You want an autonomous engine to point at in-scope targets and review the output. Strix. The deepest toolkit and the best proxy design of the five, with the understanding that you, or a later verification pass, confirm each finding.
• You have source and want white-box exploit-proving on injection, XSS, SSRF, or auth. Shannon. The best verification framing and the most reusable schema, with human review of the proof text.
• You want to watch an autonomous engagement in real time in a lab. PentAGI. Unmatched observability; add your own verification before trusting a finding.
• You want interactive methodology help or CTF ideation with a human driving. PentestGPT’s legacy PTT mode, where you are the oracle by design.
• You want to automate single known-CVE exploitation, or you want a great exploitability-scoring rubric to reuse. PentestAgent, treating its success numbers as provisional.

Closing verdict

If you are evaluating one of these for real work, here is the honest map. Shannon is the one to watch: the best-architected and most intellectually honest of the five, and if it closes its last-mile verification gap it will be genuinely proof-grade. Today, treat its output as high-quality, well-structured leads that still need human confirmation. Strix is the most capable exploitation engine and the best codebase to learn from, especially that proxy design, and its “validated, not false positives” claim will be backed the day it runs the PoC it already writes. PentAGI is the one to run for unmatched observability over an autonomous lab engagement, with the understanding that it verifies nothing yet and that “complete isolation” and “20+ integrated tools” are still ahead of the code. PentestGPT’s legacy PTT assistant is a fine interactive advisor with good ideas; read its flagship autonomous findings as leads. PentestAgent is a research prototype whose exploitability-scoring rubric is worth reimplementing and whose success numbers are self-graded.

And the thing that unites all five: none of them proves a finding yet. Every reported vulnerability from every one of these tools, regardless of its CVSS score, regardless of the 96% on the benchmark, regardless of how confidently the report says “successfully exploited,” is a lead for a human to triage rather than a proven vulnerability. The marketing says confirms; the code, today, asserts. The good news is that the distance between the two is an engineering problem, not a research one, and whoever closes it first will have the first genuinely trustworthy tool in the category.

Methodology, per-tool dossiers with file:line evidence, the full scorecard with per-cell justifications, the benchmark-integrity analysis, and reproduction commands are published as a companion analysis repository. Every tool was re-cloned at a pinned commit and read; claims are tagged verified-from-code, from-docs, or inferred. Corrections to any specific finding are welcome; bring the code.