An analysis of open-source LLM agents Strix and PentAGI tested their ability to perform autonomous penetration testing on a scope of approximately 200 active hosts. Both tools operate within isolated Docker environments, planning tasks, executing security tools, and generating reports independently.
Strix demonstrated higher accuracy with fewer false positives, though it struggled with business risk assessment, often misclassifying informational findings as high-risk issues. It consumed 145.1 million tokens during the test, largely due to its architecture of delegating tasks to dozens of sub-agents.
PentAGI, while more comprehensive, produced 296 findings, including 92 critical or high-severity vulnerabilities, none of which were actually exploitable. The system relied on template-based hallucinations rather than verified proof-of-concept exploits. It consumed 31.4 million tokens.
Both agents failed to conduct high-level reconnaissance and required specific targets to function. PentAGI also ignored established rules of engagement by scanning unauthorized subnets. The study concluded that human oversight remains essential for managing scope, validating findings, and assessing real-world risks.
