Integrating Neo4j into Modern Data Engineering Pipelines

The project integrates Neo4j into a data engineering stack alongside Kafka, ClickHouse, dbt, Airflow, and Metabase. The system processes AIS vessel movement data, using ClickHouse for historical time-series storage and Neo4j for managing current graph states, relationships, and network topology.

Data flows from the source into Kafka, where it is consumed by both ClickHouse and a Neo4j Sink connector. The latter updates vessel entities in real-time. Neo4j also functions as an analytical engine, utilizing the Graph Data Science (GDS) library to run PageRank and Louvain algorithms for port importance and community detection.

Calculated graph metrics, such as PageRank scores and community identifiers, are exported from Neo4j back into ClickHouse. This allows these features to be integrated into BI tools like Metabase via dbt. The architecture treats Neo4j as a component within the data pipeline, orchestrated by Airflow.

To ensure reproducibility, the project utilizes a custom Docker image containing Neo4j, APOC, APOC Extended, GDS, and the ClickHouse JDBC driver. The CI pipeline includes automated smoke tests for Kafka and Neo4j integration, ensuring consistent configuration across local and production environments.