Modern data platforms require a SQL engine to query data stored in Apache Iceberg format within S3 buckets without moving it to a separate database. This engine acts as a layer that accepts SQL queries, plans their execution, and retrieves results via a catalog.
Engine selection depends on workload patterns, ecosystem maturity, and network topology. Distributed MPP engines like Trino and Presto focus on external data, while systems like ClickHouse prioritize their own storage formats. DuckDB is designed for local prototyping and single-user tasks.
A benchmark analysis using TPC-DS with a 3.5 TB dataset on a 32 vCPU/256 GiB node revealed performance differences. Presto took 203 seconds, while Trino finished in 47 seconds. StarRocks and Doris showed high speed in isolated tests but struggled under concurrent loads.
Independent tests comparing Trino and Spark on a 10 TB TPC-DS dataset show that Trino is approximately 3.5 times faster, completing tasks in 4,441 seconds compared to Spark's 15,678 seconds. However, the choice of engine remains dependent on specific infrastructure needs.
