The author has developed a custom Gated RNN architecture named LSWM (Long-Short Working Memory), which utilizes a modified mathematical approach compared to traditional LSTM and GRU models. The architecture replaces standard activation functions with scaled softsign functions to achieve a 0 to 1 output range.
The LSWM model consists of two main stages: Short Working Memory (SWM) and Long Working Memory (LWM). The design intentionally omits the Constant Error Carousel (CEC) typically found in LSTMs, relying instead on a matrix-based attention mechanism to manage long-term dependencies.
The model was tested on a 'multi-hop branching' task, designed to evaluate logical reasoning across sequences. During training, the network achieved 100% accuracy on the test set after 1000 epochs, demonstrating the ability to handle branching logic and sequence extrapolation.
Benchmarking results indicate that while LSWM outperforms LSTM and GRU in accuracy for this specific task, it remains slower in training speed. The author noted that the model's performance is sensitive to random initialization seeds, leading to further refinements in the mathematical implementation.
