Engineers identified a chain of six bugs causing system instability during network maintenance. The issues led to segmentation faults, massive socket leaks, and spikes in TCP connection attempts.
A critical bug in glibc, present since 2015, caused processes to crash with a segmentation fault. It occurred when the resolver re-initialization failed due to resource exhaustion, leaving the resolver state corrupted.
The investigation revealed that a library for socket management leaked UDP sockets when the bind operation failed due to ephemeral port exhaustion. The system limit for these ports is 28 232.
A regression in the Async concurrency library caused DNS resolution attempts to hang indefinitely. Additionally, the Kafka client was found to duplicate parallel DNS queries, further straining network resources.
A migration error in the task runner caused file descriptor limits to be reduced by a factor of eight. This triggered the resource exhaustion that exposed the underlying bugs in the other components.
The team successfully reproduced the crashes by limiting file descriptors and forcing single-threaded execution. A fix for the glibc bug has been accepted and released in version 2.44.
