New Microsoft and Cornell Method Boosts LLM Prediction Accuracy

Researchers from Microsoft and Cornell University introduced a new training technique for large language models called Free Pause Tokens. The method, detailed in a paper submitted to arXiv on September 3, creates a parallel prediction stream that shares weights with the main model through gated feedforward networks.

Unlike previous methods that inserted explicit pause tokens into input sequences, this approach avoids increasing context lengths or key-value cache sizes. As a result, the model maintains standard inference speeds without requiring additional hardware or infrastructure redesigns.

Testing on a 1 billion parameter model showed a 2 to 3 centinats improvement in next-token prediction. The training process takes 1.09 to 1.14 times longer than standard pipelines, while showing performance gains across isoflop, isoparameter, and isotoken metrics.

The team achieved low training overhead by using two-pass splitting for FlashAttention compatibility, a prediction window of w=0, and shared gated feedforward networks that utilize existing model parameters.