The project to build a Crimean Tatar language corpus faced a lack of usable datasets. Initial efforts relied on a cleaned speech set, but the majority of historical materials existed only as scanned PDFs or audio recordings without transcripts. The developer processed these materials to create sound-text pairs for speech recognition and pure text for language models.
Analysis of the archive revealed that 79% of Crimean Tatar pages already contained a digital text layer, requiring no OCR. The remaining 20% required optical character recognition. The team implemented automated fixes for common errors, such as incorrect character substitutions caused by Russian-trained OCR engines and encoding mismatches in Latin-script texts.
To ensure data quality, the developer used a dictionary-based metric to evaluate pages, checking for valid word prefixes and character distribution. This process increased the usable training text by approximately tenfold. Duplicate content was identified and removed by comparing text segments, which accounted for 7.0% of the pages.
A significant challenge emerged with Turkish language influence in the training data, particularly in periodicals published outside Crimea. The developer created specialized detectors to identify Turkish-style forms, such as specific verbal suffixes. These detectors helped isolate problematic segments without deleting them, allowing for manual review and quality control.
Attempts to automatically normalize Turkish-style spellings into Crimean Tatar were abandoned after tests showed the process created valid words with incorrect meanings. The developer opted to keep dialectal forms in the corpus, as they represent the language's diversity, while maintaining strict standards for the model's output.
