A recent benchmark tested pandas, Polars, DuckDB, and PySpark on a machine with 5.8 GB of RAM and two CPU cores to determine performance limits when processing data. The study used a custom tool, TableBench, to run five common tasks, including filtering, grouping, and joins, based on the TPC-H schema.
Pandas struggled with memory, failing to load 18 million rows due to high memory overhead. While using the pyarrow backend extended this limit to 60 million rows, it did not improve performance for grouping and sorting. Polars outperformed pandas in speed, though its standard mode failed on 180 million rows. Using the streaming engine allowed Polars to process 180 million rows in 7.35 seconds.
DuckDB proved highly efficient, handling all tasks where disk space allowed, often outperforming other tools. It processed 180 million rows in 5.84 seconds using 0.4 GB of RAM by reading directly from Parquet files. Spark was the slowest on a single machine, often taking significantly longer than DuckDB, though it was the only tool to successfully cache 180 million rows.
The author concluded that Spark is better suited for distributed clusters rather than single-machine optimization. For local data processing, Polars and DuckDB offer faster and simpler alternatives to pandas. The study also highlighted that code optimization within pandas itself can sometimes yield greater performance gains than switching libraries entirely.
