The engineering team at GRI implemented end-to-end logging to improve observability within a complex Django-based monolith and microservices architecture. The project involves various integrations, including Kafka, REST, SOAP, and Celery tasks, which previously made tracking event lifecycles during errors difficult.
The team transitioned from the standard Python logging module to structlog to optimize log collection for their ELK stack. During the setup, they encountered issues with Sentry, where error grouping and JSON formatting were broken. These issues were resolved by redefining the fingerprint for Sentry events and implementing specific decoding for JSON and Unicode characters.
To achieve end-to-end tracking, the team used structlog.contextvars to inject a unique request_id into the thread context for every API call. For Celery tasks, they utilized task_prerun signals to propagate this ID across task boundaries, ensuring that the entire chain of events remains traceable even when multiple tasks are triggered sequentially.
For external integrations, the team developed a plugin for the Zeep library to include the request_id in SOAP headers. For REST requests, they initially attempted to override the requests library, but this caused memory leaks and connection errors. Consequently, they shifted to a manual approach where developers explicitly include the ID in request headers.
