The TokenRouter system addresses the challenges of token-level LLM routing, which aims to optimize the balance between service cost and output quality. Unlike session-level or request-level routing, token-level routing distributes individual tokens across different models, though it currently faces significant implementation hurdles.
Existing infrastructures struggle with token-level routing due to frequent synchronization issues and delays when adding requests to batches. TokenRouter proposes a new serving architecture based on the principle of programming from the request perspective while executing from the model perspective to improve efficiency.
