Modern e-commerce search systems are moving beyond simple visual image matching. While traditional visual search converts product photos into embeddings to calculate distance, it often fails to distinguish between visually similar but distinct items, such as different shoe models or materials.
To address these limitations, researchers are integrating multimodal approaches that combine visual data with text descriptions, attributes, and user behavior. Early examples like Fashion IQ introduced interactive search, allowing users to refine results with natural language feedback rather than relying solely on image similarity.
The development of models like FashionBERT and CLIP marked a shift toward joint training of visual and textual encoders. CLIP, trained on 400 million image-text pairs, allows for zero-shot classification and text-based queries. However, e-commerce requires specialized models like e-CLIP to handle specific product attributes and catalog nuances.
A key challenge in e-commerce is managing multiple images per product and preventing the model from treating identical items from different sellers as negative examples. Solutions like e-CLIP use catalog identifiers to group these items, while newer architectures like MOON utilize Multimodal Large Language Models (MLLM) to aggregate multiple images and text into a unified representation.
The current trend involves using MLLM, such as Qwen2.5-VL, to process multiple photos and product details simultaneously. This approach, featured in research accepted at WSDM 2026, also incorporates real-world user purchase data to refine search relevance beyond simple visual or textual matching.
