Log deduplication helps reduce network traffic, storage volume, and data retrieval time by filtering repetitive entries. Different log shippers implement this functionality in various ways, ranging from built-in stateful deduplication to rate limiting or UI-based filtering.
OpenTelemetry Collector uses the log_dedup processor, currently in alpha status. It aggregates identical logs over a configurable interval, adding metadata like repeat count and timestamps. Users must carefully define include_fields to avoid high cardinality issues that can impact memory usage.
Vector provides a stable stateful dedupe transform using an LRU cache. It allows users to match specific fields to identify duplicates. While it is highly efficient and predictable, it does not natively track the total count of discarded events, and its state is lost upon service restart.
Fluent Bit lacks a native stateful deduplication filter, requiring custom Lua scripts to manage cache and TTL logic. Similarly, vlagent and Grafana Alloy do not offer built-in content deduplication, focusing instead on delivery reliability and log collection, leaving deduplication to backend processing.
