The company developed a pipeline for normalizing master data, processing millions of records with 98% accuracy. The system operates on a standard server equipped with 8 cores and 16 GB of RAM, ensuring high performance in enterprise environments.
The classification system includes approximately 1,000 classes, with an average of 7 attributes per class, though complex items may require up to 30. Maintaining a 95% accuracy threshold is critical to prevent data duplication and reporting errors.
Initial attempts using regular expressions and grammar-based parsers proved inefficient, requiring 4 days of setup per class. Transitioning to custom-trained BERT-based models and NER architectures reduced the preparation time to 1–2 days per class.
The team evaluated Large Language Models (LLMs) for data normalization but found them unsuitable for industrial use. LLMs were 30 times slower than local ML models and prone to hallucinations, often generating non-existent attributes that compromised data integrity.
The current solution utilizes a hybrid approach: local ML models extract data fragments, while deterministic rules handle the standardization of values. This method avoids the pitfalls of generative models and maintains strict control over the output format.
