Ozon Tech engineers developed the j2c tool to automate the transfer of machine learning experiments between the JupyterHub development cluster and the high-performance Compute cluster.
The platform addresses the challenge of moving experiments by separating dependencies. Instead of creating heavy, self-contained archives, the system uses pre-configured Docker images for system-level dependencies like Python, PyTorch, and CUDA.
The j2c tool packages only the user-specific virtual environment and source code. By utilizing the venv module with the system-site-packages option, the system ensures that the experiment correctly links to the pre-installed libraries in the target cluster.
This approach avoids the overhead of traditional methods like conda-pack, which can take over an hour to build and result in archives tens of gigabytes in size. The new workflow significantly reduces preparation time and storage requirements for ML tasks.
