Amazon has launched TorchNeuron, an open-source toolkit designed to streamline the execution of PyTorch workloads on AWS Trainium chips. Previously, running these projects on Trainium-based instances required significant manual configuration, which increased development costs. The new toolkit minimizes code changes, requiring developers only to specify the target chip.
The toolkit supports the latest Trainium2 and Trainium3 accelerators for both training and inference tasks. The Trainium3 processor includes eight computing cores, each with four specialized calculation modules, and utilizes 154 gigabytes of onboard memory. Servers equipped with these chips can house up to 144 accelerators, delivering performance of up to 362 quadrillion calculations per second in MXFP8 format.
TorchNeuron integrates the NKI programming language and compiler, allowing for low-level model optimizations. It provides pre-packaged NKI operators that assist in implementing mixture-of-experts architectures and reducing memory usage. Additionally, the toolkit supports TorchTitan, an AI training engine that automates workload distribution across chips using fully sharded data parallelism.
Early adopters of the toolkit include the voice agent startup Cartesia AI Inc., which utilizes it for inference workloads, and the Lieber Institute for Brain Development, which is applying the technology to optimize medical research workflows.
