Vector DB Engine

The service ingests vectorized data, performs classification, and uses advanced machine learning models to discover and generate candidate metadata filters for downstream governance and authorization use cases.

Configuration

Required Environment Variables

This application requires specific Environment Variables to operate correctly.

• Non-Sensitive Variables (ConfigMap)

  • CATEGORY_EMBEDDINGS_PATH: Path to the category embeddings JSON file. Default is /data/category_embeddings.json.
  • EMBEDDING_CACHE_DIR: Directory used to cache embeddings. Default is /data/embedding_cache.
  • MODEL_DIR: Directory containing trained ML models. Default is /models.
  • CONFIDENCE_THRESHOLD: Minimum confidence threshold for classifications. Default is 0.7.
  • LOG_LEVEL: Application logging level. Default is INFO.

• Sensitive Variables (Secret)

  • OPENAI_API_KEY: Required. OpenAI API key used for embedding generation.

# Option 1: Using --set (creates a Secret via Helm)
helm install vector-db-classifier-engine ./vector-db-classifier-engine \
  --set secret.OPENAI_API_KEY="sk-your-actual-key"

# Option 2: Using a separate secrets file
helm install vector-db-classifier-engine ./vector-db-classifier-engine \
  -f secrets.yaml

# Option 3: Using an existing Kubernetes Secret
kubectl create secret generic my-openai-secret \
  --from-literal=OPENAI_API_KEY="sk-your-actual-key"

helm install vector-db-classifier-engine ./vector-db-classifier-engine \
  --set existingSecret.enabled=true \
  --set existingSecret.name="my-openai-secret"

Key Values

Global Values

When this chart is deployed as part of an umbrella chart, global values can be defined and applied across all sub-charts.

global:
  annotations:
    custom.io/annotation: "value"

  labels:
    app.kubernetes.io/part-of: "my-platform"

  imagePullSecrets:
    - name: my-registry-secret

  tolerations:
    - key: "special"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: disktype
                operator: In
                values:
                  - ssd

Health Checks

The Helm chart defines the following health probes:

• Liveness Probe: Verifies that the application process is running at /healthcheck/liveness.
• Readiness Probe: Verifies that the application is ready to receive traffic at /healthcheck/liveness.
• Startup Probe: Allows additional startup time before liveness checks are enforced.

Autoscaling

Horizontal Pod Autoscaling can be enabled using the following configuration:

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

Examples

Minimal Configuration

image:
  tag: "v1.0.0"

env:
  LOG_LEVEL: "INFO"
  CONFIDENCE_THRESHOLD: "0.7"

secret:
  OPENAI_API_KEY: "sk-your-openai-key"

Using an Existing Secret

When secrets are managed externally, reference an existing Kubernetes Secret.

image:
  tag: "v1.0.0"

env:
  LOG_LEVEL: "INFO"
  CONFIDENCE_THRESHOLD: "0.7"

existingSecret:
  enabled: true
  name: "my-openai-secret"

Create the Secret in advance:

kubectl create secret generic my-openai-secret \
  --from-literal=OPENAI_API_KEY="sk-your-actual-key" \
  --namespace your-namespace

Production Configuration

Below is a Product Configuration example:

replicaCount: 3

image:
  tag: "v1.0.0"
  pullPolicy: Always

imagePullSecrets:
  - name: registry-credentials

env:
  CONFIDENCE_THRESHOLD: "0.75"
  LOG_LEVEL: "INFO"
  ENVIRONMENT: "production"

secret:
  OPENAI_API_KEY: "sk-your-openai-key"
  DATABASE_PASSWORD: "secure-password"
  API_TOKEN: "secure-token"

resources:
  requests:
    memory: "1Gi"
    cpu: "500m"
  limits:
    memory: "4Gi"
    cpu: "2000m"

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20
  targetCPUUtilizationPercentage: 70

ingress:
  enabled: true
  className: "nginx"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
  hosts:
    - host: api.example.com
      paths:
        - path: /classifier
          pathType: Prefix
  tls:
    - secretName: api-tls
      hosts:
        - api.example.com

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - vector-db-classifier-engine
          topologyKey: kubernetes.io/hostname

Upgrades

Use the code below to upgrade the Engine:

helm upgrade vector-db-classifier-engine ./vector-db-classifier-engine \
  --namespace my-namespace \
  --values my-values.yaml

Uninstallation

Use the code below to uninstall the Vector DB Engine.

helm uninstall vector-db-classifier-engine --namespace my-namespace