Developers often face data processing challenges when handling the Cyrillic letters 'e' and 'yo'. In Unicode, 'yo' can be represented as a single character (U+0451) or as a sequence of 'e' (U+0435) and a combining diaeresis (U+0308). Standard Python functions like len() treat these as different lengths, and normalization forms like NFC or NFD do not equate 'e' with 'yo'.
PostgreSQL 18 includes an 'unaccent' extension that maps 'yo' to 'e' and removes the U+0308 combining mark. While this improves search recall, it creates semantic collisions where words like 'mel' (chalk) and 'myol' (swept) become indistinguishable. Because unaccent functions are marked as STABLE, they cannot be used directly in immutable functional indexes.
Beyond 'e' and 'yo', developers must handle 'confusables'—visually similar characters from different scripts, such as Latin 'a' and Cyrillic 'a' (U+0430). Unicode normalization (NFC, NFD, NFKC, NFKD) does not detect these substitutions. Protection requires implementing restriction levels or confusable skeleton checks as defined in Unicode Technical Standard #39.
Best practices for data handling involve maintaining the original user input for audit purposes while creating separate projections for search and canonicalization. Developers should define specific rules for each attribute rather than applying global transformations, ensuring that search-optimized fields do not overwrite the source data.
