Early search engines relied on lexical matching, using the Bag of Words model to index site content. This approach failed to account for word order, synonyms, or negations, often requiring users to refine queries manually.
Methods like TF-IDF and BM25 introduced weighting to prioritize relevant terms over common ones, while PageRank, introduced by Google, utilized link structures to determine document authority and ranking.
By the late 2000s, search engines incorporated morphological analysis and lemmatization to handle word forms and synonyms. In 2009, Yandex implemented MatrixNet, a machine learning model for ranking that automated the weighting of various signals.
Google launched the Knowledge Graph in 2012 to identify entities and their relationships, helping disambiguate queries. Later, in 2015, Google introduced RankBrain, a deep learning system designed to map queries to concepts.
In 2016, Yandex released the Palekh update, which utilized DSSM neural networks. This technology converts queries and page titles into numerical vectors to match content based on semantic meaning rather than exact keyword overlaps.
