Building a supercomputer requires at least two servers connected via InfiniBand and a shared task. InfiniBand is a low-latency communication technology used in computing clusters, particularly for AI workloads, to facilitate data exchange between machines.
For tasks like neural network processing, which involve prefill and decode stages, servers must exchange intermediate results such as KV-cache. RDMA technology allows data transfer between memory areas with minimal CPU involvement, which is critical for maintaining performance in distributed computing.
Network management in InfiniBand is handled by a subnet manager like OpenSM, which discovers topology and configures routes. To isolate traffic between different users, engineers use partitions, which are logical groups of ports defined by P_Key keys.
Manual configuration of server ports is prone to errors, especially when dealing with multiple accelerators per machine. The team at Cloud.ru automated this process by creating a system that collects port identifiers from servers and generates configurations automatically.
To modify cluster composition without downtime, the team uses OpenSM with configuration reloads via SSH. They also implemented a redundant subnet manager setup, where configuration files are delivered to both primary and backup managers to ensure high availability.
The concept of 'supercomputer as a service' allows users to dynamically group servers and manage their connectivity through a centralized control system, with future plans to integrate with Terraform.
