Researchers from the University of Maryland and AWS developed LEGO-Anything, a project that uses coding agents to generate editable 3D scenes from single photos. The agent writes and executes code for Blender, iteratively refining the output to match the original image.
To evaluate performance, the team created LEGO-Bench, which includes 208 images from 104 indoor and outdoor scenes using 443 assets. The benchmark scores models on validity, geometric accuracy, and visual similarity by comparing results against simulator-based ground truths.
Testing showed that while six GPT configurations consistently produced working scenes, geometric accuracy varied. GPT-6 Astra achieved 53.4 percent accuracy on indoor scenes and 39.6 percent on outdoor scenes, while weaker models scored around 15 percent.
The study identified that agents struggle with self-assessment, often failing to determine if their revisions improve the scene. To address this, the team introduced LEGO-Plugin, which replaces internal model judgment with concrete measurements and prevents regressive edits.
The plugin improved performance across all tested models, with weaker agents seeing gains of up to 62.7 percent. Despite these improvements, the researchers noted that current reconstructed scenes are not yet accurate enough for specialized vision tasks.
