MWS AI, a division of MTS Web Services, has released RESON, an open-source Python library for evaluating automatic speech recognition (ASR) models. The tool supports over 10 metrics, including WER, CER, MWA, and a proprietary WIS indicator designed to identify words critical for specific business scenarios, such as brand names.
RESON generates interactive HTML reports and can be integrated into existing development pipelines via command line or Python API. It allows for deployment within private infrastructure, enabling teams to compare different model versions or third-party solutions efficiently. The tool reduces analysis time from hours to 10–15 minutes.
Alongside the library, MWS AI launched the RESON ASR OSD benchmark, containing over 12,000 labeled audio recordings totaling approximately 38 hours of data. The dataset covers conversational speech, numeric sequences, and various acoustic environments, including background noise and real-world interference.
MWS AI tested several open-source models using the new benchmark. Sber's GigaAM v3 demonstrated the best performance with an average Word Error Rate (WER) of 7.42%. Vosk followed with 11.07%, while Nvidia's Parakeet and Nemotron models recorded 15.83% and 25.52% respectively.
