Secret Management Configuration
    • 24 Nov 2024
    • 11 Minutes to read
    • Dark
      Light
    • PDF

    Secret Management Configuration

    • Dark
      Light
    • PDF

    Article summary

    About Secret Management

    The Secret Management Service is an optional PAA component responsible for integrating with Secret stores and providing Secrets to other PAA services. Currently, the service supports two key use cases:

    • Providing a private key to the PDP for signing Authorization Decisions returned as JWTs.
    • Supplying AWS ElastiCache for Redis passwords to the Agent, PIP-Operator, and PDP for authenticating ongoing connections to Redis.

    Supported Secret Stores

    The PlainID Secret Management Service currently supports the following integrations:

    • Private Keys used for PDP JWT signing:
      • HashiCorp Vault
      • Environment Variable
      • Text File

    • AWS ElastiCache for Redis Passwords
      • AWS Secrets Manager
    Additional Integrations

    If additional Secret store integrations or use cases are required, please contact PlainID to discuss available options.

    Configuring the Secret Management Service

    By default, the Secret Management Service is not included in the PAA deployment. To enable it, you are required to:

    • Add a secret-mgmt section to the values-custom.yaml file.
    • Set it as enabled: true and define additional configuration parameters, such as ports, logging, and most importantly, secretStore definitions.

    Example:

    secrets-mgmt:
      enabled: true
      replicaCount: 1
      command: []
      # Allows you to add any config files to /app/config
      plainIDConfig:
        config.yaml:
          server:
            port: 8080
          # Which IP is allowed to make requests to secret-mgmt
            bindIp: 0.0.0.0
          management:
            port: 8081
          log:
            level: "info" #debug,trace
            format: "json"
            logTo: "console" # rolling, file
          rabbitmq:
            enabled: false
          gin:
            mode: release  # debug, release
          # Secret Store configuration
          secretStore:
            - id: fileSecret
              type: File
              isDefault: true
              decoder: Base64
              details:
                path: /app/config/filename.txt
            # Examples below for HashiCorp Vault based Secret Store
    #       - id: vault
    #         type: Vault
    #         isDefault: true
    #         decoder: None # Base64, None
    #         details:
    #           PathPrefix: prefix
    #           defaultPath: vault_
    #           skipVerify: true
    #           url: vault
    #           timeout: 3s
    #           enginePath: custom
    #         
              # Authentication method should use a Token
    #         auth:
    #           method: Token # Token, Kubernetes
    #           tokenValue: token
    #           tokenFileName: ${ VAULT_TOKEN }  - instead of token value
    #         
              # OR Authentication method should use vault Kubernetes authentication
    #         auth:
    #           method: Kubernetes
    #           path: /var/path-to-file/file.txt
    #           role: secrets-mgmt
            
            # Examples below for Environment Variables based on the Secret Store
    #       - id: ENV_VAR_SECRET
    #          type: Environment
    #          isDefault: false
    #          decoder: Base64
    #          details:
    #            name: ENV_VAR_SECRET
    

    Optional Settings for the Secret Management Service

    SectionParameterValueDescription
    serverport8080The port number that the server listens to for incoming requests.
    serverbindIp0.0.0.0The IP address that is authorized to make requests to the Secret management service.
    Use 0.0.0.0 to allow connections from every IP address on the local machine
    managementport8081The port number that the management service listens to for incoming requests.
    loglevelinfoThe Application logging level.
    logformatjsonThe Application logging format.
    loglogToconsoleThe logging output destination.
    ginmodereleaseThe mode that Gin (the HTTP framework) should run in.

    Configuring a Secret Store

    The Secret Store retrieves relevant Secrets (e.g., Private Keys). The Secrets Management Service supports various store integrations, allowing multiple stores to be configured for different purposes. The id defined in this configuration identifies the store integration when setting up a Secret's usage.

    General Store Parameters

    secretStore:
     - id: vault
       type: Vault
       isDefault: true
       Decoder: None # Base64, None
       details:
         defaultPath: vault_
         skipVerify: true
         url: vault
         timeout: 3s
         enginePath: custom
         pathPrefix: prefix
       auth: #by Token or by K8s authentication
         method: Token # Token, Kubernetes
         tokenValue: token
    #    tokenFileName: ${ VAULT_TOKEN } - alternate for token value
    #         
    #  auth:
    #    method: Kubernetes
    #    path: /var/path-to-file/file.txt
    #    role: secrets-mgmt
            
    

    The below parameters are defined under the secretStore section.

    ParameterValueDescription
    idvaultAn identifier for the Secret store
    typeVaultThe type of Secret store, available types:
    - Vault
    - Environment
    - File
    - AWSSecretsManager
    isDefaulttrueIndicates whether this is the default Secret store. If multiple Secret stores are defined, the default store is used for operations that do not specify a specific store. If more than one Secret store is defined as default, or none are defined as default, the first Secret store is used as the default.
    DecodernoneThe decoder used for the Secrets stored in this store. Possible values are:
    - Base64
    - None
    detailsAdditional details related to the Secret store configuration - specific to each store type (see below)
    details.defaultPathvault\_The default path used for accessing the Secrets in this store

    Configuring a HashiCorp Vault secretStore

    The following parameters are required for configuring the HashiCorp Vault Secret Store:

    ParameterValueDescription
    details.skipVerifytrueIndicates whether to prefix the Secret Store URL with https:// or http://.
    details.urlThe Secret store URL.
    details.timeout3sThe timeout value for connecting to the Secret store.
    details.enginePathsecretSpecifies the root folder in the HashiCorp Vault Engine from which the Secret Management Service should access Secrets.

    Note: When configuring Vault information in the Scope in the Scope Details in the Environment Settings for JWT signing, do not include the enginePath in the Path to Key parameter.
    details.pathPrefixenvironments/productionThe prefix for all paths when accessing Secrets in this store.
    authAuthentication details for accessing the Secret store. Add relevant attributes based on the auth.method (Kubernetes or Token).
    auth.methodKubernetesThe authentication method used for accessing the Secret store. Available methods:
    - Kubernetes
    - Token
    auth.pathused for KubernetesFile path for the Kubernetes authorization value.
    auth.roleused for KubernetesVault authorization role name.
    auth.tokenValueused for tokenThe token value used for authentication when accessing the Secret store.
    auth.tokenFilePathused for tokenFile path holding the token.

    Configuring an Environment Variable secretStore

    The Secret Management Service can also obtain private keys from environment variables. To configure a Secret Store that reads private keys from environment variables, use the following configuration:

    ParameterValueDescription
    typeEnvironmentSet the type of the Secret store to environment variables
    details.nameENV_VAR_SECRETThe name of the environment variable from which to obtain the private key

    Sample Configuration

    secrets-mgmt:
      ... 
      plainIDConfig:
        config.yaml:
          server:
            port: 8080
          # Which IP is allowed to make requests to secret-mgmt
            bindIp: 0.0.0.0
          management:
            port: 8081
          log:
            level: "info" #debug,trace
            format: "json"
            logTo: "console" # rolling, file
          gin:
            mode: release  # debug, release
          # Secret Store configuration
          secretStore:
           - id: ENV_VAR_SECRET
              type: Environment
              isDefault: true
              decoder: Base64
              details:
                name: ENV_VAR_SECRET
    

    Configuring a Text File secretStore

    The Secret Management Service can also obtain private keys from text files, to configure a secretStore that reads the private keys from text files, use the following configuration:

    ParameterValueDescription
    typeFileSet the type of the Secret store to text file
    details.path/app/conf/filename.txtThe full path to the file containing the private key

    Sample Configuration

    secrets-mgmt:
      ... 
      plainIDConfig:
        config.yaml:
          server:
            port: ${APP_PORT:8072}
            bindIp: ${IP_BIND:127.0.0.1}
        management:
          port: ${MANAGEMENT_PORT:8077}
        log:
          level: "debug"
          format: "json"
          logTo: "console"
        gin:
          mode: ${GIN_MODE:release}  # debug, release
        # Secret Store configuration
    	secretStore:
    	 - id: fileSecret
    	   type: File
    	   isDefault: true
    	   Decoder: Base64
    	   details:
    	     path: /app/conf/filename.txt
    

    Using Secret Manager for PDP JWT Signing

    The PlainID PDP (Runtime service) supports obtaining the authorization response as a signed JWT. This functionality requires retrieving a private key via the Secret Management Service, which is used for JWT signing, along with additional setup described below.

    Scope Level Configuration

    In addition to configuring the Secret Management Service and Secret Store in the Policy Authorization Agent (PAA), additional settings relating to JWT need to be configured in the PlainID Policy Administration Point (PAP).

    JWT Sign In Settings

    JWT Sign In Setting.png

    This section includes the relevant configuration attributes for obtaining the private key used to sign the PlainID Policy Decision Point (PDP) JWT.

    AttributeDescriptionBehavior
    Secret StoreThe ID of the Secret Store to use (PAA can use multiple Secret Stores).If no Secret Store is defined, the default secret store (where default=true) will be used. If a value is specified, the specified Secret Store will be used.
    Path to KeyDefine the path to the key location in the vault (relevant only for Secret Stores of type vault, not applicable for File or Environment Secret Stores).If not specified, the details.defaultPath from the Secret Store configuration will be used.
    Key NameDefine the name of the key in the vault (relevant only for Secret Stores of type vault, not applicable for File or Environment Secret Stores).If not defined, the PlainID Scope ClientID is used as the Key Name.

    JWT Response Settings

    JWT Response Setting image.png

    JWT Response Attribute Settings

    AttributeDescription
    AudienceThe value to include in the aud claim.
    X509 CertificateThe public X.509 certificate that is published in the PDP JWKS URL. This allows the consumer of the PDP decisions to validate the PDP Signed JWT. The child of the X.509 Certificate in the JWKS is the ClientID of the PlainID ScopeSample JWKS URL.
    Token Lifetime - EXP (Seconds)The token lifetime expiration in seconds.

    Using Secret Manager with AWS Secrets Manager & AWS ElastiCache for Redis

    PlainID PAA services use Redis as a data store for different purposes, like PIP settings, Policy metadata and PDP caching. If you are using an AWS Managed Redis you can leverage PlainID's integration with AWS Secrets Manager to obtain AWS Redis passwords and serve them to the different services.
    This integration also supports seamless Secret rotation with no downtime or a need to restart the PAA pods, based on the AWS SM Secret rotation and password update in ElastiCache.

    Prerequisites

    Connecting to the AWS SM
    To access AWS Secrets Manager, IAM credentials with sufficient permissions are required. There are two available methods to grant these permissions:

    • Grant the Secrets Management Service an IAM role directly via EKS, as detailed here (recommended).

    • Configure an AWS key ID and Secret access key in the Secret Management service’s configuration file.

    AWS Roles

    The role assigned to the PAA's Secrets Management Service must grant permission to read the Secret containing the Redis password. The following resources and actions must be allowed in the Policy:

    • Action: secretsmanager:BatchGetSecretValue
      Resource: *

    • Action: secretsmanager:GetSecretValue
      Resource: the ARN of the Secret containing the Redis password.

    Note: the secretsmanager:BatchGetSecretValue permission must be granted on the * resource; It cannot be granted on specific Secret ARNs. See the BatchGetSecretValue row in this table for more information.

    Granting the above permission does not allow reading all Secrets. Each Secret returned by the BatchGetSecretValue API must be explicitly allowed by the secretsmanager:GetSecretValue permission. Therefore, the latter must be granted only on the specific ARN of the Redis password Secret.

    Check out an AWS example policy that explains the above principle here.

    Example JSON for an AWS Policy

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "fetchSpecificSecret",
          "Effect": "Allow",
          "Action": [
            "secretsmanager:GetSecretValue"
          ],
          "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:redis-rotated-pw"
        },
        {
          "Sid": "batchFetchSecrets",
          "Effect": "Allow",
          "Action": [
            "secretsmanager:BatchGetSecretValue"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
    }
    

    Configuring the AWS SM secretStore

    This capability is supported with the AWS Secrets Manager integration. Use the following parameters in your configuration:

    ParameterDescription
    typeSet the type of the secret store to AWS Secrets Manager: AWSSecretsManager.
    details.authSet the authentication for AWS Secrets Manager using the parameters below (see more details below).
    details.auth.regionSet the relevant AWS region. You can use an Env Var: ${AWS_AUTH_REGION}.
    details.auth.accessKeyIdSet the access key ID. You can use an Env Var: ${AWS_AUTH_ACCESS_KEY_ID}.
    details.auth.secretAccessKeySet the access key Secret. You can use an Env Var: ${AWS_AUTH_SECRET_ACCESS_KEY}.
    serviceAccount.annotations.eks.amazonaws.com/role-arnSet the AWS role defined for the PlainID Secret Management Service.
    externalRedisThis configuration is not under secretStore but in the general Redis config section.
    .hostHost for external Redis.
    .usernameThe username used for the connection with AWS Secrets Manager.
    .passwordThe password used for the connection with AWS Secrets Manager.

    Sample Configuration

    secretsMgmt: 
      enabled: true 
      ...
      
      secretStore: 
      - id: AWS_SECRETS_MANAGER_STORE 
        type: AWSSecretsManager 
        isDefault: false 
        details: 
          auth: 
            region: ${AWS_AUTH_REGION} 
            accessKeyId: ${AWS_AUTH_ACCESS_KEY_ID} 
            secretAccessKey: ${AWS_AUTH_SECRET_ACCESS_KEY} 
    
    serviceAccount: 
      annotations: 
        eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/redis-rotating-pw-fetch 
        
    redis: 
      enabled: false # this capability supported for external redis in AWS so the internal redis needs to be disabled
      
    externalRedis: 
      host: "master.redis-pw-rotation1.0wgi71l.use2.cache.amazonaws.com" 
    username: |
        "{{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.username}}"
    password: |
        "{{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.password}}"
      port: 6379 
      tls: true
    

    AWS Secrets Manager Rotation

    AWS SM supports rotating ElastiCache for Redis Secrets via Lambda, a service which Amazon provides. PlainID password rotation support assumes that AWS SM password rotation is implemented in the manner recommended by Amazon. See this guide for more information.

    PAA Service Configurations

    PAA services that use Redis get the credentials from the Secret Manager (SM) service do not require special configurations. Once an SM is configured and the externalRedis parameter is defined, all other services seamlessly utilize this configuration.

    Customers Managing Their Own K8s Deployment

    If you are not using the PlainID Helm deployment and instead build your own Kubernetes deployment procedures, you are required to configure the Redis user and password keys in the PAA services under the relevant Redis configuration entries.
    Use the following key pattern to integrate with the SM service and obtain the credentials:
    {{store=<store name>,key=<AWS SM key>,jsonpath=<AWS SM value jpath>}}

    Example:

    • Redis Connection Username: {{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.username}}
    • Redis Connection Password: {{store=AWS_SECRETS_MANAGER_STORE,key=redis-rotated-pw,jsonpath=$.password}}

    Optional Configurations in PDP, PIP, and Agent Services

    Secret Caching Duration & Refresh
    In the PAA services (PDP, PIP, and Agent) that use Redis connections, you can define the cache duration for the Redis Secret retrieved from SM. Configure this using the optional environment variable:
    SECRET_MGMT_CLIENT_CACHE_MANAGER_DURATION_SECONDS (default: 3600 seconds).

    Troubleshooting SM Integration
    Each PAA service connecting to Redis for key retrieval uses the password provided by the SM service. To enable detailed logging for troubleshooting, configure the following settings and set the log level to TRACE:

    • For the agent and pip-operator:

      secretMgmtClient.logging.detailed.enabled = true
      
    • For the runtime:

      "secretsProvider": { 
        "logging": { 
          "detailed": { 
            "enabled": true 
          } 
        } 
      }
      

      Alternatively, use can also use this environment variable:

      RT_secretsProvider__logging__detailed__enabled=true
      

    This logging configuration adds the following logs in the PAA services while establishing Redis connections:

    logger.detailed("Successfully parsed input: store=" + store);  
    logger.detailed("identified as not a pattern, treating as password...");
    



    By following this guide, you can effectively configure and troubleshoot Redis connections for PAA services, whether using PlainID's Helm deployment or managing custom Kubernetes setups. Leveraging the integration with the Secret Manager Service ensures secure and streamlined access to Redis credentials, while optional configurations and logging enable enhanced flexibility and visibility for your deployment.


    Was this article helpful?