Researchers tested replacing system prompts with initial states in RWKV-7 models ranging from 0.4B to 2.9B parameters. Experiments were conducted on a MacBook M4 with 16 GB of RAM using the SwiftRWKV framework.
The study compared three configurations: models without prompts, models with standard text prompts, and models with trained initial states. Results show that text prompts can reduce accuracy by 3–15 points, while trained states consistently perform as well as or better than models without prompts.
Trained states offer significant performance benefits, including faster initial token generation—up to 13 times faster for long prompts. Training a state takes between 5 and 50 minutes and requires 1.6–5.4 GB of memory, resulting in a state file size of 6.5–22 MB.
The method uses context distillation where the model learns to replicate desired outputs without a prompt. This approach is particularly effective for small models or tasks where the model ignores text instructions, such as news classification, where accuracy improved significantly compared to standard prompting.
While trained states are effective for maintaining roles and formatting, they do not add new knowledge. The study highlights that filtering training data is crucial to avoid inheriting errors, and gradient clipping is necessary to prevent training failure.
