Perplexity has detailed its GPU embedding stack, which utilizes existing LLM infrastructure components named Ivy, Tulip, and ROSE to serve the pplx-embed model. The company opted to reuse prefill and decode kernels from its LLM stack rather than building a dedicated engine, as embedding models function as small Transformers.
The Tulip scheduler manages request accumulation, prioritizing simplicity because dense layer costs dominate attention costs for the models used. Perplexity employs full-model CUDA graphs to mitigate CPU-side kernel launch overhead. To manage the high volume of graphs, the team implemented lazy capture, which performs an eager warmup run before triggering capture on the second request.
The ROSE component handles multiple attention backends, including FlashInfer 2, FlashInfer 3, and FlashAttention 4, selecting the optimal one based on the specific model and sequence length. ROSE avoids KV cache instantiation to prevent padding issues. Additionally, the system uses a LazyTensor abstraction to allow Rust async tasks to process batches without blocking device execution.
Perplexity benchmarked its stack against vLLM version 0.22.0 in BF16 precision. Testing covered four scenarios: low-latency embeddings, low-latency scoring, high-throughput embeddings, and high-concurrency embeddings. The company verified that cosine similarity divergence remains within 0.1 percent during these performance evaluations.
