The Data Operations Dashboard is a custom monitoring solution designed for Data Lakehouse platforms using Spark, Iceberg, and Airflow. It integrates with GitLab, Trino, and Docker to provide a unified view of platform operations, replacing fragmented diagnostic processes.
The dashboard features an interactive Data Lineage tool built with Python, NetworkX, and Cytoscape.js. It visualizes both upstream and downstream dependencies, allowing engineers to identify the root cause of delays or failures by analyzing the logical structure and operational status of DAGs.
A Gantt-based visualization tool displays multiple DAGs on a single timeline to identify peak loads and resource contention. It distinguishes between sensor waiting times and actual calculation phases, helping teams track SLA compliance across different data layers.
The Operational Health module provides real-time monitoring of system incidents, including failed tasks, SLA risks, and data quality issues. It offers direct links to Kibana and the Spark History Server for deep-dive diagnostics of long-running or failed processes.
