Storing sensitive data like database passwords or cloud access keys in plain text within Hadoop XML configuration files poses significant security risks. These files are often accessible to numerous users and services, increasing the likelihood of accidental exposure or unauthorized access.
The Hadoop Credential Provider API, introduced in version 2.6, allows applications to retrieve secrets from a secure JCEKS keystore instead of reading them from XML files. This mechanism uses the getPassword() method to fetch credentials, which are then decrypted at runtime.
JCEKS files store secrets as key-value pairs encrypted with a master password. While the format relies on the 3DES algorithm, which NIST has deprecated for new data as of 2024, it remains a standard solution for Hadoop. The security of this method depends heavily on the protection of the master password.
Managing the master password can be handled via environment variables or, more securely, through dedicated files on the disk or in tmpfs. Manual rotation of credentials involves creating a new JCEKS file, distributing it across all cluster nodes, and restarting the affected services.
Platforms like Arenadata's ADCM automate credential encryption, though security depends on proper file permissions. Misconfigured access rights, such as overly permissive settings on HDFS files, can still leave secrets vulnerable to unauthorized users within the cluster.
