Microsoft has released a new VS Code extension designed to automate the identification and mitigation of vulnerabilities in AI agents. The tool, named run-assert-eval, executes a four-stage workflow: Clarity identifies potential failure scenarios, Assert converts them into measurable tests, Agent Control Specification generates runtime security policies, and a final verification run confirms the effectiveness of the implemented protections.
The company demonstrated the tool using a billing support agent restricted to a single client account. Initial testing revealed that the agent failed in 30% of cases during the second scenario, exposing sensitive client data in 12 out of 40 relevant dialogues. The identified risks included unauthorized operations and data leakage from other accounts.
After applying the generated security policies to the pre-call and post-call stages, the violation rate dropped to 5.9%. Importantly, the agent maintained its utility, as the failure rate for legitimate requests reached zero across all four test segments. The tool ensures that security metrics and agent performance are evaluated separately to maintain balance.
The new workflow consolidates the discovery and remediation cycle into a single command, eliminating gaps between identifying a vulnerability and verifying its resolution. The Assert and Agent Control Specification components are now available as open source, with the extension including 7 pre-defined domains and 14 risk sets.
