Periodic functions like sine and cosine offer an alternative to traditional activation functions in neural networks. Unlike hyperbolic tangent (tanh), which saturates at large input values and causes gradient decay, periodic functions maintain non-zero gradients, allowing the model to continue learning.
The use of periodic functions forces the model to precisely adjust weights to match specific values across the function's period. While functions like sine and cosine keep outputs within a stable range of [-1, 1], others like tangent, cotangent, secant, and cosecant are unsuitable for neural networks because they tend toward infinity, which can lead to NaN errors and training failure.
In comparative tests between attention-based models using tanh and sine, the tanh-based model initially showed better performance. However, as training progressed, the sine-based model surpassed it, likely due to the non-saturating nature of the sine function's derivative.
Implementing periodic functions in deep networks presents challenges regarding gradient stability. While these functions prevent gradient vanishing caused by saturation, they require careful control of input values to avoid instability or noise, as maintaining stable RMS values becomes increasingly difficult with greater network depth.
