Secret Management Configuration

The Secret Management Service is an optional PAA component responsible for integrating with secret stores and providing secrets to other PAA services. Currently, the service supports these use cases:

• Passwords/secrets for data source connections in the PIP-Operator
• Passwords for the PDP Authorization Audit Database connection settings
• REDIS authentication passwords for PAA services
• A private key for the PDP to sign Authorization Decisions returned as JWTs

Supported Secret Stores

Configuring the Secret Management Service

By default, the Secret Management Service is not included in the PAA deployment. To enable it, add a secretsMgmt section to the values-custom.yaml file and follow the configuration instructions.

Enabling and Configuring the Service

After adding to the values-custom.yaml file the secret-mgmt section, set it as enabled: true and define additional configuration parameters, such as ports, logging, and most importantly, secretStore definitions.

Example:

secretsMgmt:
  enabled: true
  replicaCount: 1
  command: []
  # Allows you to add any config files to /app/config
  plainIDConfig:
    config.yaml:
      server:
        port: 8080
        # Which IP is allowed to make requests to secret-mgmt
        bindIp: 0.0.0.0
      management:
        port: 8081
      log:
        level: "info" #debug,trace
        format: "json"
        logTo: "console" # rolling, file
      rabbitmq:
        enabled: false
      gin:
        mode: release # debug, release
      # Secret Store configuration
      secretStore:
        - id: fileSecret
          type: File
          isDefault: true
          decoder: Base64
          details:
            path: /app/config/filename.txt
        # Examples below for HashiCorp Vault based Secret Store
#       - id: vault
#         type: Vault
#         isDefault: true
#         decoder: None # Base64, None
#         details:
#           PathPrefix: prefix
#           defaultPath: vault_
#           skipVerify: true
#           url: vault
#           timeout: 3s
#           enginePath: custom
#
# Authentication method should use a Token
#         auth:
#           method: Token # Token, Kubernetes
#           tokenValue: token
#           tokenFileName: ${ VAULT_TOKEN }  - instead of token value
#
# OR Authentication method should use vault Kubernetes authentication
#         auth:
#           method: Kubernetes
#           path: /var/path-to-file/file.txt
#           role: secrets-mgmt

# Examples below for Environment Variables based on the Secret Store
#       - id: ENV_VAR_SECRET
#          type: Environment
#          isDefault: false
#          decoder: Base64
#          details:
#            name: ENV_VAR_SECRET

Optional Settings for the Secret Management Service

Configuring a Secret Store

The Secret Store retrieves relevant Secrets (e.g., Private Keys). The Secrets Management Service supports various store integrations, allowing multiple stores to be configured for different purposes. The id defined in this configuration identifies the store integration when setting up a Secret's usage. Refer to the specific Secret Stores below to learn more about specific parameters.

General Store Parameters

secretStore:
  - id: vault
    type: Vault
    isDefault: true
    Decoder: None # Base64, None
    details:
      defaultPath: vault_
      skipVerify: true
      url: vault
      timeout: 3s
      enginePath: custom
      pathPrefix: prefix
    auth: #by Token or by K8s authentication
      method: Token # Token, Kubernetes
      tokenValue: token
#    tokenFileName: ${ VAULT_TOKEN } - alternate for token value
#
#  auth:
#    method: Kubernetes
#    path: /var/path-to-file/file.txt
#    role: secrets-mgmt

The parameters below are defined under the secretStore section.

HashiCorp Store

Configuring a HashiCorp Vault Secret Store

The following parameters are required to configure the HashiCorp Vault Secret Store:

Environment Variables Store

Configuring an Environment Variable Secret Store

The Secret Management Service can also obtain private keys from Environment Variables. To configure a Secret Store that reads private keys from Environment Variables, use the following configuration:

Sample Configuration

secretsMgmt:
  ...
  plainIDConfig:
    ...
      # Secret Store configuration
      secretStore:
       - id: ENV_VAR_SECRET
          type: Environment
          isDefault: true
          decoder: Base64
          details:
            name: ENV_VAR_SECRET

Text File Store

Configuring a Text File Secret Store

The Secret Management Service can also obtain private keys from text files, to configure a secretStore that reads the private keys from text files, use the following configuration:

Sample Configuration

secretsMgmt:
  ...
  plainIDConfig:
    ...
    # Secret Store configuration
	secretStore:
	 - id: fileSecret
	   type: File
	   isDefault: true
	   Decoder: Base64
	   details:
	     path: /app/conf/filename.txt
   filename.txt: "BASE64 ENCODED SECRET"

An example use case involves injecting a certificate as a Base64-encoded file into the Secrets Management file. This process typically entails encoding the certificate into Base64 format, including it in the configuration file used for Secrets Management, and securely storing it there. Once stored, the Application or system can reference this certificate as needed. This enables secure access to sensitive information without hardcoding credentials or certificates directly into the application code.

AWS Secret Manager Store

AWS Secrets Manager provides secure credential management. Integrating with this store ensures seamless access to sensitive credentials and enables automatic rotation and secure retrieval of secrets. This store supports storing and retrieving secrets, including credentials for external services like Redis.

Prerequisites for AWS SM

To integrate with AWS SM, set access permissions that allow the PlainID Secrets Management Service to connect, authenticate, and fetch secrets.
The best practice is to set up an AWS role with the necessary permissions policy and attach it to the Kubernetes cluster that runs the PlainID Secrets Management Service.

Set AWS Role - The role assigned to the PAA's Secrets Management Service must grant permission to read the Secret containing the relevant passwords. The following resources and actions must be allowed in the AWS Policy:

• Action: secretsmanager:BatchGetSecretValue
  Resource: *
• Action: secretsmanager:GetSecretValue
  Resource: The secret's ARN containing a relevant password, such as a Redis password used for Redis authentication.

Note: the secretsmanager:BatchGetSecretValue permission must be granted on the * resource; It cannot be granted on specific Secret ARNs. See the BatchGetSecretValue row in this table for more
information.

Granting the above permission does not allow reading all Secrets. Each Secret returned by the BatchGetSecretValue API must be explicitly allowed by the secretsmanager:GetSecretValue permission. Therefore, the latter must be granted only on the specific ARN of the relevant secrets required by the PlainID integration (Redis, Data Sources, etc.).

Check out an AWS example policy that explains the above principle here.

Example JSON for an AWS Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "fetchSpecificSecret",
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue"],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:redis-rotated-pw"
    },
    {
      "Sid": "batchFetchSecrets",
      "Effect": "Allow",
      "Action": ["secretsmanager:BatchGetSecretValue"],
      "Resource": ["*"]
    }
  ]
}

Configuring the AWS SM Secret Store

Use the following parameters in your configuration:

Example

secretsMgmt:
  ...
  plainIDConfig:
   ...
    # Secret Store configuration
	secretStore:
	 - id: AWS_SM_STORE
      type: AWSSecretsManager
      isDefault: false
      details:
         auth:
            region: ${AWS_AUTH_REGION}
            accessKeyId: ${AWS_AUTH_ACCESS_KEY_ID}
            secretAccessKey: ${AWS_AUTH_SECRET_ACCESS_KEY}

AWS RDS IAM Authentication Store

AWS RDS allows an IAM Authentication flow, replacing regular Database connections with a short lived signed connection token. Enabling IAM-based authentication for RDS and integrating it with the Secrets Management Service ensures seamless access to RDS databases and enhances security by eliminating the need for static passwords.

Prerequisites

To enable the integration with AWS RDS IAM Auth, you are required to set access permissions allowing PlainID Secrets Management Service to connect with AWS, authenticate and fetch connection tokens.
The best practice is to set up an AWS Role with the necessary permissions policy and attach it to the Kubernetes cluster running the PlainID Secrets Management Service.

Set AWS Role - The role assigned to the PAA's Secrets Management Service must grant permission to use the RDS Authentication. The following resources and actions must be allowed in the AWS Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["rds-db:connect"],
      "Resource": ["arn:aws:rds-db:<region>:<account>:dbuser:<cluster-resource-id>/<database-user-name>"]
    }
  ]
}

Define RDS IAM Grants - Follow these steps to prepare your AWS store integration with the PlainID Secret Management Service:

1. The RDS IAM must be enabled for the relevant RDS cluster.
2. Create a dedicated user in the RDS database:

CREATE USER <username> WITH LOGIN;
GRANT rds_iam TO <username>;

-- Note that from this point if the user existed and had its own password, it will no longer function.

3. Connect to your Database instance using IAM Authentication. Refer to the AWS documentation on this topic.

Configuring the AWS RDS IAM Auth Secret Store

The AWS RDS IAM store can be configured in the PlainID Secret Management Service by using the store type AWSRDSIAMAuth.

In addition, you are required to authenticate to AWS using these parameters:

Example

secretsMgmt:
  ...
  plainIDConfig:
   ...
    # Secret Store configuration
	secretStore:
	 - id: AWS_RDS_IAM_AUTH_STORE
      type: AWSRDSIAMAuth
      isDefault: false
      details:
         auth:
            region: ${AWS_AUTH_REGION}
            accessKeyId: ${AWS_AUTH_ACCESS_KEY_ID}
            secretAccessKey: ${AWS_AUTH_SECRET_ACCESS_KEY}

See more details on how to use this store for RDS connection tokens under Defining RDS Secrets Using the AWS RDS IAM Auth Store

Secrets Integration Use Cases

Here are some examples for specific secrets integration use cases supported by the Secrets Management Service and its integration to supported Stores.

AWS ElastiCache Redis Authentication

PlainID PAA services use Redis as a data store for various purposes, such as PIP settings, policy metadata, and PDP caching. If you are using an AWS Managed Redis you can leverage PlainID's integration with AWS Secrets Manager to obtain AWS Redis passwords and serve them to the different services. This integration also supports seamless Secret rotation with no downtime or a need to restart the PAA pods, based on the AWS SM Secret rotation and password update in ElastiCache.

To use this integration pattern you are required to:

1. Setup an AWS SM integration
2. Define an AWS Role to allow access from Secrets Management Service to AWS SM
3. Define the Redis passwords in different services to use secret key from Secrets Management Service

Configuring AWS SM as a Store

Follow the configuration steps detailed under AWS Secret Manager Store

AWS SM Preparations

To access AWS Secrets Manager, IAM credentials with permissions are required. There are two available methods to grant these permissions:

• Grant the Secrets Management Service an IAM role directly via EKS, as detailed here (recommended).

• Configure an AWS Key ID and Secret access key in the Secret Management service’s configuration file.

Make sure the Redis secret key is granted access permission using the specific Secret ARNs.

PAA Configurations

Make sure to add the externalRedis section to your values-custom.yaml and define the connection parameters to use Secrets Management Service with the proper details, such as storeID and secret key. See example below:

Redis Configuration with secret

PAA services that use Redis get the credentials from the Secret Manager (SM) service do not require special configurations. Once an SM is configured and the externalRedis parameter is defined, all other services seamlessly utilize this configuration.

Sample Configuration

secretsMgmt:
  enabled: true
  ...

  secretStore:
  - id: AWS_SECRETS_MANAGER_STORE
    type: AWSSecretsManager
    isDefault: false
    details:
      auth:
        region: ${AWS_AUTH_REGION}
        accessKeyId: ${AWS_AUTH_ACCESS_KEY_ID}
        secretAccessKey: ${AWS_AUTH_SECRET_ACCESS_KEY}

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/redis-rotating-pw-fetch

redis:
  enabled: false # this capability is supported for external redis in AWS, so the internal redis needs to be disabled

externalRedis:
  port: 6379
  tls: true
  host: "master.redis-pw-rotation1.0wgi71l.use2.cache.amazonaws.com"
username: |
    "{{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.username}}"
password: |
    "{{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.password}}"

Additional Optional Configurations in PDP, PIP and Agent Services

Secret Caching Duration & Refresh In the PAA services (PDP, PIP, Agent) that use Redis connections, you can define the cache duration for the Redis Secret retrieved from SM. Configure this using the optional environment variable (not limited to Redis):
SECRET_MGMT_CLIENT_CACHE_MANAGER_DURATION_SECONDS (default: 3600 seconds).

Troubleshooting SM Integration Each PAA service connecting to Redis for key retrieval uses the password provided by the SM service. To enable detailed logging for troubleshooting, configure the following settings and set the log level to TRACE:

• For the agent and pip-operator:

SECRETMGMTCLIENT_LOGGING_DETAILED_ENABLED: "true"

• For the runtime, use the Environment Variable:

RTCONF_SECRETS_PROVIDER__LOGGING__DETAILED__ENABLED: "true"

This logging configuration enables either of the following logs in PAA services:
Success message:

logger.detailed("Successfully parsed input: store=" + store);

Error message:

logger.detailed("identified as not a pattern, treating as password...");

This guide helps you configure and troubleshoot Redis connections for PAA services, whether using PlainID's Helm deployment or custom Kubernetes setups Leveraging the integration with the Secret Manager Service ensures secure and streamlined access to Redis credentials, while optional configurations and logging enable enhanced flexibility and visibility for your deployment.

Customers Managing Their Own K8s Deployment

If you are not using the PlainID Helm deployment and instead opt to build your own Kubernetes deployment procedures, you are required to configure the Redis user and password keys in the PAA services under the relevant Redis configuration entries.
Use the following key pattern to integrate with the SM service and obtain the credentials:
{{store=<store name>,key=<AWS SM key>,jsonpath=<AWS SM value jpath>}}

Example:

• Redis Connection Username: {{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.username}}
• Redis Connection Password: {{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.password}}

AWS Secrets Manager Secret Rotation

AWS SM supports rotating ElastiCache for Redis secrets via Lambda, a service provided by Amazon. PlainID password rotation support, assuming that the AWS SM password rotation is implemented in the manner recommended by Amazon. See this guide for more information.

Defining RDS Secrets Using the AWS RDS IAM Auth Store

After setting up the new store you can define secrets, referencing the store based on the following structure {{store=<storeID>,key=<secretName>,refreshInterval=3s}}

• storeID
  • The ID defined in the secret management service configuration.
• secretName
  • The secret key name you want to consume from the store.
  • For the AWSRDSIAM store the key will be in the form user@host:port/region based on the AWS user you defined at the prerequisite.
• refreshInterval
  • The interval used to fetch updates from the store.
  • Can be defined with s/m/h/d (sec/min/hours/days).
  • Note the default refresh interval for this store type is 10m which should be sufficient to accommodate AWS RDS's 15-min expiration policy.

Using Secrets for PIP Data Sources

If integrating with PlainID’s Secret Manager, reference credentials using a {{secret name}}.
For Secret Manager integration, use the syntax specific to your secret store. See Secret Management Configuration for details.
Example: {{general store example}}

For AWS RDS DB with AWS IAM Auth:
{{store=AWS_RDS_IAM_AUTH_STORE,key=test_user@shared-partner-mgmt-dev.cluster-g6wgs3hs1zff.eu-east-2.rds.amazonaws.com:5432/eu-east-2}}.

Using Secret for PDP Audit DB

If an AWS RDS database needs to be defined as the PDP Authorization Audit database, the database connection is configured in the PDP settings using a few configuration keys. The configuration of the Secret itself will use similar secret syntax, but as its defined within a yaml configuration the double curly secret syntax needs to be wrapped with a printf command to be valid.

Here is a set of Runtime config keys defining a connection to an AWS RDS DB:

Refer to our Runtime Service Configuration, Authorization Audit Configuration, and Policy Information Point articles for more specific information on how to use Environment Variables for generating tokens and passwords.

Using Secrets for PIP Data Source Connections

When configuring databases and other data sources in the UI, you can use secrets to manage authentication credentials securely. Instead of storing credentials in plain text, you can reference them from a secret store, enabling seamless secret rotation and enhanced security.

Using Secret Annotations in OAuth Configuration

In OAuth authentication, you can reference a secret from a secret store within the OAuth XML configuration file. The following example demonstrates how to use a secret annotation for the client-secret field:

<application-policy name="ps-auth2-massive">
    <authentication>
        <login-module code="com.plainid.pip.auth.oauth.clientcredential.OAuth20ClientCredentialLoginModule" flag="required">
            <module-option name="access-token-uri">https://login.microsoftonline.com/cd6168ed-fe82-4528-b2eb-4e030d322878/oauth2/v2.0/token</module-option>
            <module-option name="client-id">19f4ef6c-d0dd-42f0-844c-b529b609a171</module-option>
            <module-option name="client-secret">{{ printf "{{store=AWS_SECRETS_MANAGER_STORE,key=path1/path2/key}}" }}</module-option>
            <module-option name="scope">https://graph.microsoft.com/.default</module-option>
        </login-module>
    </authentication>
</application-policy>

Secret Management and Rotation

The Client Credentials settings use a secret stored in Secret Manager (SM) for token retrieval. Secrets typically have a longer lifespan than access tokens, and Secret Manager integration ensures seamless rotation without downtime. This allows for periodic secret updates without requiring manual configuration changes.

When integrating with a secret store, both the old and new secrets can be handled simultaneously. This allows for graceful secret transitions, avoiding disruptions when rotating credentials. The system automatically retrieves the latest valid secret, ensuring continuous operation.

Using Secret Manager for PDP JWT Signing

The PlainID PDP (Runtime service) supports obtaining the authorization response as a signed JWT. This functionality requires retrieving a private key via the Secret Management Service, which is used for JWT signing, along with additional setup described below.

Scope Level Configuration

In addition to configuring the Secret Management Service and Secret Store in the Policy Authorization Agent (PAA), additional settings relating to JWT need to be configured in the PlainID Policy Administration Point (PAP).

JWT Sign In Settings

This section includes the relevant configuration attributes for obtaining the private key used to sign the PlainID Policy Decision Point (PDP) JWT.

JWT Response Settings

JWT Response Attribute Settings