Moore Threads has officially open-sourced its Megatron-MUSA-Titan infrastructure project. The platform is designed to support the training of trillion-parameter MoE models on clusters consisting of tens of thousands of GPUs.
The infrastructure is built on the company's proprietary MUSA architecture. It covers both pre-training and reinforcement learning scenarios, providing developers with specific system optimizations and training adaptations for the MUSA platform.
The project has already been deployed in large-scale cluster environments, demonstrating high model FLOPs utilization and stability during long-term training cycles. It integrates with existing frameworks like Slime and Miles for reinforcement learning.
Megatron-MUSA-Titan introduces support for blockwise FP8 training to leverage low-precision computing capabilities. Additionally, it utilizes the DeepEP and ACE asynchronous communication engine to optimize token distribution and expert parallelism.
