The author developed a custom S3 balancer to eliminate the need for manual configuration updates when migrating data between Ceph clusters. Previously, each cluster had unique addresses, keys, and users, leading to complex management during hardware or data center transitions.
The balancer identifies the target cluster by extracting the access key from the S3 request header or URL. It maintains a routing table provided by the CAM management panel. If a key is missing, the balancer queries CAM, with a rate limit of 20 requests per second to prevent abuse.
For anonymous requests or temporary STS keys, routing is based on bucket names. The balancer operates in memory, updating its routing snapshot every five seconds via CAM. It remains functional even if the management panel is offline, using the last cached state saved to disk.
The system supports multisite configurations by grouping clusters. It handles failover by monitoring replication lag and metadata synchronization. Large multipart uploads are supported via stream buffering, with timeouts extended to 15 minutes to accommodate cluster processing times.
High availability is achieved by deploying multiple balancer replicas across different nodes. An L4 balancer using Katran manages traffic distribution. If a node fails, the scheduler migrates the replica, and the system automatically excludes unresponsive nodes from the traffic flow.
