The developer integrated a visual avatar for an AI voice assistant, Aika, which operates via Yandex Realtime. The system uses WebRTC to stream audio and video, with the avatar service synchronizing lip movements to the audio stream provided by the server.
The integration involved testing Simli and Anam services. The developer noted that Anam provided more realistic facial expressions, while Simli’s performance was limited by account restrictions. Technical challenges included managing audio sampling rates, as Yandex outputs 24 kHz audio while the avatar services required 16 kHz.
The system architecture uses a FastAPI server to manage session tokens, limiting usage to 6 tokens per hour per IP address. To ensure smooth operation, the developer implemented a generation counter to handle asynchronous start and stop commands, preventing race conditions when users interrupt the AI.
Mobile compatibility, specifically on iPhone Safari, required a workaround for autoplay policies. The developer triggered a play command on a canvas element to bypass restrictions. Additionally, the system dynamically adjusts the Voice Activity Detection (VAD) threshold, switching between 1500 ms and 2200 ms depending on whether the user is dictating numbers or engaging in general conversation.
