A backend developer at Yandex Food's platform team discovered a significant CPU overhead in their PHP monolith. During a load increase, the developer noticed that the requests per second (RPS) in peak evening hours closely matched the number of CPU cores allocated to the monolith, with 100% load on one core occurring every 2.5 RPS.
To identify the cause, the team used Perforator, a profiling tool installed on most data center host machines. However, the team had to implement a custom version of Perforator with PHP support, as the standard version lacked it. This process involved overcoming eBPF kernel verifier limits, where the program size exceeded the 1 million instruction limit, causing the load to fail.
The developer resolved the eBPF issue by modifying the loop structure in the Python stack unwinding function to simplify the verifier's state tracking. Additionally, a debugging macro (PHP_TRACE) was causing the loop to appear infinite to the verifier by pushing the loop counter to the stack. Moving the debug print statement fixed this issue.
The profiling revealed that the CPU consumption was caused by a function used for OpenTelemetry semantic conventions. Specifically, the getRouteCollection method was rebuilding the route list from YAML files and PHP annotations every time it was called, instead of using a cached version. The team fixed this by caching the routes during the container build stage, which eliminated the CPU overhead.