Salesforce AI Research has published a preprint detailing a method to reduce computational costs in reinforcement learning. The study suggests that training individual specialists for specific domains and distilling their knowledge into a single model can match the performance of joint reinforcement learning.
In a test involving math and coding tasks, the multi-teacher on-policy distillation (OPD) approach achieved results comparable to joint Group Relative Policy Optimization (GRPO) while consuming 42% fewer training FLOPs. The distillation process itself accounted for 5.2% of the total training cost in this two-domain scenario.
When the researchers expanded the setup to four domains, the cost of the distillation step increased to 8.0% of the specialist training FLOPs. The study notes that using self-GRPO teachers, derived from earlier checkpoints of the student model, proved highly efficient, reducing distillation FLOPs by up to a factor of 50.
The paper, titled Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO (arXiv:2611.08453), was released on November 5, 2026. The authors caution that the findings are based on a preprint that has not yet undergone peer review and that performance varies when using external teachers instead of in-house experts.
