The integration project required DATAREON Platform to process millions of product labeling codes from the Honest Sign system. The system fetches data via REST API, stores it in PostgreSQL, and performs daily updates to check the current status of each code.
Initially, the system struggled with performance as the database grew to 90 million rows. Calculating the latest record version on the fly caused delays, with batch processing of 1,000 objects taking over 50 seconds. The team shifted to explicit status tracking using an is_current flag.
The developers implemented partial PostgreSQL indexes to optimize queries for new and existing records. They also introduced parallel processing using 10 workers, utilizing the FOR UPDATE SKIP LOCKED mechanism to manage concurrent access to the database without processing the same record twice.
To handle authentication, the team moved to a singleton-based approach for API tokens. Instead of dynamic creation, a single record with a fixed EntityId is updated by workers when tokens expire, preventing duplicate entries in the data bank.
These optimizations, combined with clear separation of responsibilities between DATAREON and PostgreSQL, increased the processing speed to approximately 1.1–1.2 million objects per hour.
