SecurityScorecard has launched an open-source benchmark designed to evaluate artificial intelligence models in third-party risk management. The framework assesses how effectively models handle security telemetry and vendor questionnaires, while also testing their resistance to adversarial techniques like prompt injection and jailbreaks.
The benchmark assigns models an autonomy readiness level ranging from L0 to L4 based on performance in repeated trials. Attack resistance is graded on a scale from A to F. The system includes deceptive vendor self-attestations to measure how false claims impact model accuracy, with scores measured against a baseline.
The company released version 2026.09 of the framework, which includes documentation and a public leaderboard. Users can run their own models and submit results, while certified evaluations require direct engagement with SecurityScorecard. The initiative aims to provide a transparent method for assessing AI reliability in enterprise security environments.
Early testing covered closed frontier, open-weight, and international models. According to SecurityScorecard, open-weight models are narrowing the performance gap in supply chain security tasks but remain highly susceptible to adversarial attacks. The company did not disclose specific scores or the names of the models tested.
