MWS Data Scout is a service designed to automate the documentation of database tables using large language models. The system addresses the challenge of managing large data landscapes, where manual documentation is inefficient, requiring up to six hours per table.
The platform aggregates multi-layered context, including data samples, table classification, domain affiliation, and existing documentation. Experiments revealed that overly complex prompts, specifically those including rigid formatting or outdated documentation, can negatively impact the quality of the generated descriptions.
To optimize performance, the team implemented a prompt matrix tailored to specific table classes and business domains. The system currently supports over eight database connectors, including Oracle, PostgreSQL, and ClickHouse, and integrates with DataHub and OpenMetadata.
The automation results for MTS databases show that 8,180 tables were documented in under three days, with generation taking approximately 25 seconds per table. This increased individual productivity from 30 to 2,000 tables per month, with 95% of descriptions requiring no manual edits.
Beyond text descriptions, the tool generates ER diagrams, identifies hidden data relationships, and detects personal data for compliance with Russian Federal Law No. 152-FZ. To manage costs, the company uses dedicated GPU A100 and A40 servers and offloads simple tasks to lightweight classifiers.
