Secret Management Configuration

About Secret Management

The Secret Management Service is an optional PAA component responsible for integrating with Secret stores and providing Secrets to other PAA services. Currently, the service supports two key use cases:

• Providing a private key to the PDP for signing Authorization Decisions returned as JWTs.
• Supplying AWS ElastiCache for Redis passwords to the Agent, PIP-Operator, and PDP for authenticating ongoing connections to Redis.

Supported Secret Stores

The PlainID Secret Management Service currently supports the following integrations:

• Private Keys used for PDP JWT signing:
  • HashiCorp Vault
  • Environment Variable
  • Text File
    
    
• AWS ElastiCache for Redis Passwords
  • AWS Secrets Manager

Configuring the Secret Management Service

By default, the Secret Management Service is not included in the PAA deployment. To enable it, you are required to:

• Add a secret-mgmt section to the values-custom.yaml file.
• Set it as enabled: true and define additional configuration parameters, such as ports, logging, and most importantly, secretStore definitions.

Example:

secrets-mgmt:
  enabled: true
  replicaCount: 1
  command: []
  # Allows you to add any config files to /app/config
  plainIDConfig:
    config.yaml:
      server:
        port: 8080
      # Which IP is allowed to make requests to secret-mgmt
        bindIp: 0.0.0.0
      management:
        port: 8081
      log:
        level: "info" #debug,trace
        format: "json"
        logTo: "console" # rolling, file
      rabbitmq:
        enabled: false
      gin:
        mode: release  # debug, release
      # Secret Store configuration
      secretStore:
        - id: fileSecret
          type: File
          isDefault: true
          decoder: Base64
          details:
            path: /app/config/filename.txt
        # Examples below for HashiCorp Vault based Secret Store
#       - id: vault
#         type: Vault
#         isDefault: true
#         decoder: None # Base64, None
#         details:
#           PathPrefix: prefix
#           defaultPath: vault_
#           skipVerify: true
#           url: vault
#           timeout: 3s
#           enginePath: custom
#         
          # Authentication method should use a Token
#         auth:
#           method: Token # Token, Kubernetes
#           tokenValue: token
#           tokenFileName: ${ VAULT_TOKEN }  - instead of token value
#         
          # OR Authentication method should use vault Kubernetes authentication
#         auth:
#           method: Kubernetes
#           path: /var/path-to-file/file.txt
#           role: secrets-mgmt
        
        # Examples below for Environment Variables based on the Secret Store
#       - id: ENV_VAR_SECRET
#          type: Environment
#          isDefault: false
#          decoder: Base64
#          details:
#            name: ENV_VAR_SECRET

Optional Settings for the Secret Management Service

Configuring a Secret Store

The Secret Store retrieves relevant Secrets (e.g., Private Keys). The Secrets Management Service supports various store integrations, allowing multiple stores to be configured for different purposes. The id defined in this configuration identifies the store integration when setting up a Secret's usage.

General Store Parameters

secretStore:
 - id: vault
   type: Vault
   isDefault: true
   Decoder: None # Base64, None
   details:
     defaultPath: vault_
     skipVerify: true
     url: vault
     timeout: 3s
     enginePath: custom
     pathPrefix: prefix
   auth: #by Token or by K8s authentication
     method: Token # Token, Kubernetes
     tokenValue: token
#    tokenFileName: ${ VAULT_TOKEN } - alternate for token value
#         
#  auth:
#    method: Kubernetes
#    path: /var/path-to-file/file.txt
#    role: secrets-mgmt
        

The below parameters are defined under the secretStore section.

Configuring a HashiCorp Vault secretStore

The following parameters are required for configuring the HashiCorp Vault Secret Store:

Configuring an Environment Variable secretStore

The Secret Management Service can also obtain private keys from environment variables. To configure a Secret Store that reads private keys from environment variables, use the following configuration:

Sample Configuration

secrets-mgmt:
  ... 
  plainIDConfig:
    config.yaml:
      server:
        port: 8080
      # Which IP is allowed to make requests to secret-mgmt
        bindIp: 0.0.0.0
      management:
        port: 8081
      log:
        level: "info" #debug,trace
        format: "json"
        logTo: "console" # rolling, file
      gin:
        mode: release  # debug, release
      # Secret Store configuration
      secretStore:
       - id: ENV_VAR_SECRET
          type: Environment
          isDefault: true
          decoder: Base64
          details:
            name: ENV_VAR_SECRET

Configuring a Text File secretStore

The Secret Management Service can also obtain private keys from text files, to configure a secretStore that reads the private keys from text files, use the following configuration:

Sample Configuration

secrets-mgmt:
  ... 
  plainIDConfig:
    config.yaml:
      server:
        port: ${APP_PORT:8072}
        bindIp: ${IP_BIND:127.0.0.1}
    management:
      port: ${MANAGEMENT_PORT:8077}
    log:
      level: "debug"
      format: "json"
      logTo: "console"
    gin:
      mode: ${GIN_MODE:release}  # debug, release
    # Secret Store configuration
	secretStore:
	 - id: fileSecret
	   type: File
	   isDefault: true
	   Decoder: Base64
	   details:
	     path: /app/conf/filename.txt

Using Secret Manager for PDP JWT Signing

The PlainID PDP (Runtime service) supports obtaining the authorization response as a signed JWT. This functionality requires retrieving a private key via the Secret Management Service, which is used for JWT signing, along with additional setup described below.

Scope Level Configuration

In addition to configuring the Secret Management Service and Secret Store in the Policy Authorization Agent (PAA), additional settings relating to JWT need to be configured in the PlainID Policy Administration Point (PAP).

JWT Sign In Settings

This section includes the relevant configuration attributes for obtaining the private key used to sign the PlainID Policy Decision Point (PDP) JWT.

JWT Response Settings

JWT Response Attribute Settings

Using Secret Manager with AWS Secrets Manager & AWS ElastiCache for Redis

PlainID PAA services use Redis as a data store for different purposes, like PIP settings, Policy metadata and PDP caching. If you are using an AWS Managed Redis you can leverage PlainID's integration with AWS Secrets Manager to obtain AWS Redis passwords and serve them to the different services.
This integration also supports seamless Secret rotation with no downtime or a need to restart the PAA pods, based on the AWS SM Secret rotation and password update in ElastiCache.

Prerequisites

Connecting to the AWS SM
To access AWS Secrets Manager, IAM credentials with sufficient permissions are required. There are two available methods to grant these permissions:

• Grant the Secrets Management Service an IAM role directly via EKS, as detailed here (recommended).

• Configure an AWS key ID and Secret access key in the Secret Management service’s configuration file.

AWS Roles

The role assigned to the PAA's Secrets Management Service must grant permission to read the Secret containing the Redis password. The following resources and actions must be allowed in the Policy:

• Action: secretsmanager:BatchGetSecretValue
  Resource: *

• Action: secretsmanager:GetSecretValue
  Resource: the ARN of the Secret containing the Redis password.

Note: the secretsmanager:BatchGetSecretValue permission must be granted on the * resource; It cannot be granted on specific Secret ARNs. See the BatchGetSecretValue row in this table for more information.

Granting the above permission does not allow reading all Secrets. Each Secret returned by the BatchGetSecretValue API must be explicitly allowed by the secretsmanager:GetSecretValue permission. Therefore, the latter must be granted only on the specific ARN of the Redis password Secret.

Check out an AWS example policy that explains the above principle here.

Example JSON for an AWS Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "fetchSpecificSecret",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:redis-rotated-pw"
    },
    {
      "Sid": "batchFetchSecrets",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:BatchGetSecretValue"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Configuring the AWS SM secretStore

This capability is supported with the AWS Secrets Manager integration. Use the following parameters in your configuration:

Sample Configuration

secretsMgmt: 
  enabled: true 
  ...
  
  secretStore: 
  - id: AWS_SECRETS_MANAGER_STORE 
    type: AWSSecretsManager 
    isDefault: false 
    details: 
      auth: 
        region: ${AWS_AUTH_REGION} 
        accessKeyId: ${AWS_AUTH_ACCESS_KEY_ID} 
        secretAccessKey: ${AWS_AUTH_SECRET_ACCESS_KEY} 

serviceAccount: 
  annotations: 
    eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/redis-rotating-pw-fetch 
    
redis: 
  enabled: false # this capability supported for external redis in AWS so the internal redis needs to be disabled
  
externalRedis: 
  host: "master.redis-pw-rotation1.0wgi71l.use2.cache.amazonaws.com" 
username: |
    "{{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.username}}"
password: |
    "{{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.password}}"
  port: 6379 
  tls: true

AWS Secrets Manager Rotation

AWS SM supports rotating ElastiCache for Redis Secrets via Lambda, a service which Amazon provides. PlainID password rotation support assumes that AWS SM password rotation is implemented in the manner recommended by Amazon. See this guide for more information.

PAA Service Configurations

PAA services that use Redis get the credentials from the Secret Manager (SM) service do not require special configurations. Once an SM is configured and the externalRedis parameter is defined, all other services seamlessly utilize this configuration.

Customers Managing Their Own K8s Deployment

If you are not using the PlainID Helm deployment and instead build your own Kubernetes deployment procedures, you are required to configure the Redis user and password keys in the PAA services under the relevant Redis configuration entries.
Use the following key pattern to integrate with the SM service and obtain the credentials:
{{store=<store name>,key=<AWS SM key>,jsonpath=<AWS SM value jpath>}}

Example:

• Redis Connection Username: {{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.username}}
• Redis Connection Password: {{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.password}}

Optional Configurations in PDP, PIP, and Agent Services

Secret Caching Duration & Refresh
In the PAA services (PDP, PIP, and Agent) that use Redis connections, you can define the cache duration for the Redis Secret retrieved from SM. Configure this using the optional environment variable:
SECRET_MGMT_CLIENT_CACHE_MANAGER_DURATION_SECONDS (default: 3600 seconds).

Troubleshooting SM Integration
Each PAA service connecting to Redis for key retrieval uses the password provided by the SM service. To enable detailed logging for troubleshooting, configure the following settings and set the log level to TRACE:

• For the agent and pip-operator:

  secretMgmtClient.logging.detailed.enabled = true
  

• For the runtime:

  "secretsProvider": { 
    "logging": { 
      "detailed": { 
        "enabled": true 
      } 
    } 
  }
  

  Alternatively, use can also use this environment variable:

  RT_secretsProvider__logging__detailed__enabled=true
  

This logging configuration adds the following logs in the PAA services while establishing Redis connections:

logger.detailed("Successfully parsed input: store=" + store);  
logger.detailed("identified as not a pattern, treating as password...");

By following this guide, you can effectively configure and troubleshoot Redis connections for PAA services, whether using PlainID's Helm deployment or managing custom Kubernetes setups. Leveraging the integration with the Secret Manager Service ensures secure and streamlined access to Redis credentials, while optional configurations and logging enable enhanced flexibility and visibility for your deployment.