Flowwow migrated its legacy S3 storage, which contained over 65 TB of data and tens of millions of objects. The company aimed to optimize storage costs, which exceeded 150,000 ₽ per month, and resolve issues with data ownership and unused files.
The team implemented a migration strategy without downtime by creating a new bucket and using a dual-read, single-write architecture. The application read from the new bucket first, with a fallback to the old one if a file was missing. This allowed for gradual data migration based on actual traffic.
To identify necessary files, the developers mapped business entities to S3 keys using a MediaCleanupProcessorManager. This tool processed database records in batches of 1,000, sending copy tasks to a RabbitMQ queue. The system was configured to handle errors by distinguishing between infrastructure failures and missing files.
During the process, the team encountered stability issues with RabbitMQ when the queue exceeded 500,000 messages. They optimized the system by adjusting the number of consumers and refining retry logic to avoid redundant operations on non-existent keys.
The migration successfully reduced the storage volume from 65 TB to approximately 45 TB. About 90% of requests were served by the new bucket almost immediately, and the fallback mechanism eventually reached near-zero usage without impacting user experience.
