Groq is reportedly planning to integrate 3D-stacked DRAM into its future Language Processing Unit (LPU) inference chips. The company aims to increase memory capacity and bandwidth to support large AI models with trillions of parameters, while maintaining high throughput.
The proposed design utilizes 3D-stacked DRAM connected via through-silicon vias (TSVs). Groq intends to expand vertical DRAM capacity through standard interfaces, prioritizing scalability over achieving maximum possible bandwidth numbers.
Currently, the Groq 3 LPU features 500 MB of on-chip SRAM, providing 150 TB/s of bandwidth per chip. A 256-chip rack configuration currently offers 128 GB of total SRAM, which limits the ability to store large models locally.
Groq was established in 2016 by Jonathan Ross, a former designer of Google TPU chips. In December 2025, Nvidia entered into a deal valued at approximately $20 billion to license Groq's intellectual property and hire its key personnel.
