Spark Cluster 2 is a desktop application designed to automate the deployment of Apache Spark clusters on virtual private servers. Developed using Python and PySide6, the tool provides a graphical interface to simplify infrastructure setup for educational purposes, replacing manual configuration and command-line operations.
The updated architecture in version 2 shifts from a multi-server setup to a single-VPS model. All components, including the Spark Master, two Spark Workers, ClickHouse, and MinIO, are deployed within Docker containers on one server. This change reduces the number of required VPS instances from four to one per student, facilitating easier management for large groups.
The application is distributed as ready-to-use builds for two platforms, eliminating the need for users to install Python or external dependencies manually. While the software automates the deployment process, the user remains responsible for renting the underlying VPS from a provider of their choice.
The project integrates both software and educational components. Beyond infrastructure deployment, it includes learning materials to help students practice distributed data processing. Future development plans focus on expanding the library of practical assignments for users.
