Enrichment Agent

This service operates as part of the PlainID Edge enrichment flow and prepares vector metadata for downstream authorization and governance use cases.

Configuration

Configuration is loaded from a YAML file and merged with default values. All configuration values support Environment Variables substitution.

Configuration File and Loading

• Default path: config/config.yaml, or a path provided to the application at startup.

• Environment Variables substitution:

  • Use ${VAR} for required variables.
  • Use ${VAR:default} for optional variables with a default value, for example ${LOG_LEVEL:info}.

The following top-level keys are used by the application: server, log, http, management, jwt, and databases.

The framework consumes server and log. All other sections are consumed by the enrichment agent application.

Parameters

http Parameters

All parameters, except for port, follow the micro-infra HttpConfig. Default values may also be applied by the application at runtime.

Management Parameters

JWT Parameters

Databases Parameters

Each entry in the databases array defines a single enrichment target.

Classification Service Parameters

Pinecone Vendor Configuration

pinecone Object

collections Object

Controls which namespaces are processed. Matching is applied to the string indexName_namespaceName (for example my-index_users).

With include, only namespaces matching at least one pattern are processed.
With exclude, all namespaces except those matching are processed.

Filter Behavior

If no patterns are defined:

• include processes all namespaces.
• exclude processes no namespaces.

Patterns are compiled and evaluated as regular expressions.

Configuration Examples

Minimal Pinecone Configuration

Below is an example configuration with a single database, no schedule, and no namespace filtering:

server:
  name: enrichment-agent
log:
  level: info
jwt:
  jwksUrl: ${JWKS_URL:}
  enabled: false
databases:
  - id: pineconeDb
    type: PINECONE
    classificationService:
      serviceUrl: http://localhost:8000/classify
    metadataKey: category
    vendor:
      pinecone:
        apiKey: ${PINECONE_API_KEY}

Full Pinecone Configuration

Below is an example with scheduled enrichment, namespace filtering, JWT enabling, and Environment Variables substitution.

Key elements include:

• periodicStart: "0 * 20 * * ?" - Runs enrichment every day at 20:00 using a six-field Cron format with seconds.
• collections.mode and collections.patterns control namespace inclusion or exclusion.
• Environment Variables such as CLASSIFICATION_SERVICE_URL, PINECONE_API_KEY, JWKS_URL, JWT_VALIDATION_ENABLED, and LOG_LEVEL.

server:
  name: enrichment-agent
log:
  level: ${LOG_LEVEL:info}
jwt:
  jwksUrl: ${JWKS_URL:}
  enabled: ${JWT_VALIDATION_ENABLED:true}
databases:
  - id: pineconeDb
    type: PINECONE
    periodicStart: "0 * 20 * * ?"
    classificationService:
      serviceUrl: ${CLASSIFICATION_SERVICE_URL}
    metadataKey: category
    vendor:
      pinecone:
        apiKey: ${PINECONE_API_KEY}
      collections:
        mode: exclude
        patterns:
          - users
          - books_.*

Multiple Pinecone Databases

Multiple enrichment targets can be defined with different identifiers, API keys, classification services, filters, or metadata keys.

Example:

databases:
  - id: pineconeProd
    type: PINECONE
    periodicStart: "0 0 2 * * ?"
    classificationService:
      serviceUrl: https://classifier.example.com/classify
    metadataKey: category
    vendor:
      pinecone:
        apiKey: ${PINECONE_PROD_API_KEY}
      collections:
        mode: include
        patterns:
          - "index1_.*"
  - id: pineconeStaging
    type: PINECONE
    classificationService:
      serviceUrl: http://localhost:8000/classify
    metadataKey: category
    vendor:
      pinecone:
        apiKey: ${PINECONE_STAGING_API_KEY}

Cron Scheduling (periodicStart)

The periodicStart field uses a six-field Cron format, including seconds:

second minute hour day-of-month month day-of-week

Example:

"0 * 20 * * ?" runs every day at 20:00:00.

Leave this value empty to disable scheduled enrichment for a specific database.

Common Environment Variables

The following Environment Variables are commonly used.

Environment Variables can be defined in the Runtime environment, for example via Helm env or secret, or referenced directly in the configuration file using the ${VAR:default} syntax.