The Prod is Down platform offers a browser-based Linux simulation for practicing incident response. Users receive a root console to troubleshoot broken services, such as PostgreSQL failures, OOM Killer, or Nginx 502 errors. The platform currently features 13 distinct scenarios, each with three randomized failure modes to prevent users from relying on memorized commands.
To ensure security, the platform uses gVisor (runsc) to isolate containers, preventing direct access to the host kernel. Each environment is restricted by resource limits, including 512 MB of RAM, no swap, and a 10-minute inactivity timeout. Network access is blocked via internal Docker networks, and SSH access to the host is disabled to prevent malicious activity like mining or spamming.
The system validates solutions by checking service health rather than monitoring user commands. To prevent tampering, verification scripts run from the host, and the system monitors for unauthorized changes to system binaries. An internal Python-based emulator mimics systemd behavior, allowing users to use familiar tools like journalctl and systemctl within the restricted container environment.
The developer plans to expand the platform with new scenarios focusing on cascading and distributed failures. Future updates include an 'incident of the week' feature with a leaderboard and infrastructure improvements to handle higher traffic. Currently, the platform allows users to test their skills with a free introductory case without requiring registration.
