Researchers from Carnegie Mellon University developed LegoGPT, also known as BrickGPT, a transformer-based model designed to generate physically stable LEGO structures from text prompts. The system functions by predicting the next block in a sequence, similar to how language models predict tokens, while evaluating geometry and structural integrity within a 3D grid.
The model is a fine-tuned version of LLaMA-3.2-Instruct-1B, trained on a dataset of 47,000 LEGO assemblies and 28,000 unique 3D figures. It operates within a 20x20x20 grid, encoding block types, positions, and orientations as text tokens. The system incorporates a physical engine to simulate gravity, friction, and connection strength, allowing it to discard unstable configurations.
A key feature of the model is its backtracking mechanism. Without it, only 24% of generated structures were stable, whereas the ability to revert unsuccessful steps increased the success rate to 98.8%. The model currently supports approximately 20 categories of simple objects, such as furniture and vehicles, though it lacks internal moving parts.
Potential applications for this technology include generative design for furniture, architectural planning under specific environmental constraints, and robotics for assembly line optimization. The researchers have released the StableText2Brick dataset to support further development in this field.
