The GuardRate system operates as an Evaluation-as-Code pipeline, ensuring consistent testing conditions by utilizing Docker environments and vLLM instead of cloud APIs. The process is fully automated via CLI, allowing users to run evaluations with a single command using YAML configurations.
The framework evaluates models across 19 benchmarks, covering categories such as jailbreak attacks, false refusals, and multilingual capabilities. The system calculates metrics including F1, FNR, FPR, precision, recall, and accuracy, aggregating them into a final Integral Score.
The scoring methodology employs harmonic means for group-level assessment and geometric means for the final ranking. This approach prioritizes model stability and penalizes significant performance gaps, often resulting in smaller, more balanced models outperforming larger ones in the leaderboard.
Data from experiments, including raw predictions and container logs, are stored in artifacts directories and published to Hugging Face buckets. The frontend automatically updates to reflect these results, allowing users to analyze specific model failures and compare performance across different benchmarks.
For production use cases, the system provides latency metrics alongside safety scores. For instance, the HiveTraceGuard-Pro 0.6B model achieves a p95 latency of approximately 29 ms while maintaining a high ranking, offering a performance advantage over larger, slower models.
