The Yandex.Telepult browser extension allows users to modify their video stream in Yandex.Telemost. By intercepting the getUserMedia method, the tool replaces the camera feed with a canvas-based stream, enabling real-time effects such as background replacement, masks, stickers, and gesture-based controls.
The extension utilizes Google MediaPipe Tasks Vision, running models directly in the browser via WebAssembly. The package includes Face Landmarker, Hand Landmarker, and Image Segmenter, with a total size of approximately 20 MB. To optimize performance, the extension processes frames in a background page and drops frames if the model is busy to prevent latency.
The 'disappear' feature functions by capturing a reference image of the empty room. It calculates differences between the live feed and the reference to segment the user, applying corrections for auto-exposure and white balance. The 'Thanos' effect and other transitions are rendered by animating pixel transparency based on the generated mask.
Gesture recognition is performed locally using hand tracking. A snap gesture, detected by monitoring the distance between fingers over 300 ms, triggers the disappearance effect. Other gestures, such as palm or victory signs, allow users to display reactions, toggle the microphone, or show images within the video stream.
