The Cross-Cleaner project optimizes memory usage by replacing the standard full-file JSON loading with a streaming approach. Previously, using serde_json::from_reader to load an entire database into a Vec caused significant memory spikes due to temporary string allocations within the Helper structure during deserialization.
The new implementation uses a custom iterator, iter_json_array, which processes JSON array elements one by one. This wrapper handles array delimiters and utilizes StreamDeserializer to ensure that internal buffers do not accumulate between elements. By using BufReader, the system avoids excessive system calls, maintaining performance during streaming.
This transition allows the application to process databases that exceed available RAM, provided that global aggregation or sorting is not required. The peak memory consumption is now limited to the size of a single record rather than the entire dataset. Existing optimizations, such as string interning and Arc-based data representation, remain fully compatible with this streaming method.
The approach is applicable to various components of the system, including CleanerIndex and RegistryIndex, which already utilize efficient deserialization techniques. For databases structured as top-level arrays, this method provides a scalable way to handle data without modifying existing Deserialize implementations.
