Arenadata has released version 1.4.0 of its ADH Cloud module, part of the Arenadata Hyperwave platform. The update focuses on enhancing Spark performance within Kubernetes environments by introducing a dedicated Spark operator, a resource scheduler, and a remote shuffle service.
The new release integrates the YuniKorn scheduler to manage task queues and resource allocation among teams, featuring support for Kerberos, SPNEGO, and LDAP authentication. Additionally, the Celeborn component has been added to offload high-load shuffle operations from local node disks to external storage systems like HDFS, Ozone, and S3.
Monitoring capabilities have been expanded through integration with VictoriaMetrics and Prometheus. Users can now utilize PodMonitor, ServiceMonitor, VMPodScrape, and VMServiceScrape for tracking infrastructure performance. A new Spark History Server is also included for analyzing logs and metrics of completed applications.
Security improvements include a centralized mechanism for managing Kubernetes secrets across all platform components. Furthermore, the Arenadata Cloud CLI has been updated with new commands, such as 'adc list', to provide administrators with faster access to infrastructure status without needing external tools.
