The fourth installment of the deep learning series focuses on implementing a neural network designed to predict missing words in text. This approach aims to improve the model's understanding of linguistic nuances by requiring it to infer context from surrounding words.
The architecture utilizes a hidden layer with 50 neurons and a vocabulary-sized output layer. To optimize training, the implementation employs negative sampling, which reduces the computational load by calculating probabilities for a small subset of words rather than the entire vocabulary.
The training process involves iterating through movie reviews from the Kinopoisk dataset. For each review, the model uses a window of two words to the left and two words to the right of a target word. The network is trained to identify the correct word among a selection of one true target and five randomly chosen false candidates.
The model updates weights using binary cross-entropy and sigmoid activation. The first layer weights are initialized with random values between -0.1 and 0.1, while the second layer weights are initialized as zeros and transposed to facilitate efficient matrix operations during backpropagation.
