PIP Operator Service

The PIP Operator Service is used for managing, deploying, and mapping data sources.

Installation Guide

• Setup environment variables:
  • REDIS_HOST - Redis host
  • REDIS_PORT - Redis port
  • REDIS_PASS - Redis password (default is with no password)

For environments like staging and production it is recommended to use an external and managed Redis instance.

Notice: Redis must enable notify-keyspace-events so either allow configuration changes to redis from the app, or change them yourself manually.
Additional settings can be found under Environmental Variables below.

Environment Variables

Management

• SERVER_PORT - Server Port
• MANAGEMENT_PORT - Port for the info and health checks
• APP_AUTH_ENABLED - Enable/disable jwt authentication (default is false)
• APP_AUTH_JWT - JWT for authentication
• APP_AUTH_SECRET - JWT secret key
• APP_AUTH_ISS - JWT Issuer
• TRANSACTION_MANAGER_ENABLED - Enable/disable transaction manager
• TRANSACTION_MANAGER_COORDINATOR_ENVIRONMENT_BEAN_DEFAULT_TIMEOUT - Transaction Manager time out (default is 300 seconds)
• REDIS_HOST - Redis host
• REDIS_PORT - Redis port
• REDIS_PASS - Redis password (default is with no password)
• REDIS_SSL_ENABLED - Enable/disable SSL (default is false)
• REDIS_TIMEOUT_MIL - Redis connection timeout in milliseconds (default value is 60000)
• REDIS_POOL_MAX_TOTAL - This setting controls the max number of connections that can be created at a given time.
• REDIS_POOL_MAX_IDLE - This is the max number of connections that can be idle in the pool without being immediately evicted (closed).
• REDIS_POOL_MIN_IDLE- This is the number of "warm" connections (e.g. ready for immediate use) that remain in the pool even when load has reduced
• REDIS_POOL_MIN_EVIC_MI - This specifies the minimum amount of time a connection may sit idle in the pool before it is eligible for eviction due to idle time.
• REDIS_POOL_TIME_BETWEEN_EVIC_MIL - The number of milliseconds to sleep between runs of the idle connection validation, abandoned cleaner and idle pool resizing.

• MV_STATUS_LOGGING_FREQUENCY - Determines whether Mat View status is retrieved and logged, and specifies the logging frequency. Default is 60 minutes.
• MV_STATUS_LOGGING_EXTERNAL_STATUS_TABLES - Specifies the name(s) of external materialized view status table(s) as defined in the materialized view property teiid_rel:MATVIEW_STATUS_TABLE, in the form of <model_name>.<status_table_name>. Use the MV_STATUS_LOGGING_FREQUENCY configuration key to set how often the status is fetched and logged.
• APP_MAX_THREADS (default: 64) - Sets the number of threads for query engine work (plans, transactions, source queries). Increase for high-concurrency workloads or systems with many processors.
• APP_MAX_ACTIVE_PLANS (default: 20) - Sets the number of active query plans. Increase for long-running queries or high-concurrency workloads. Adjust memory settings if issues arise. Note: This should be smaller than max-threads.
• APP_USER_REQUEST_SOURCE_CONCURRENCY (default: 0) - Sets concurrent source queries per request. 0 uses the default (2 * (max-threads / max-active-plans)), 1 forces serial execution, and values greater than a 1 limit concurrency. Adjust for workloads with many long-running queries. Note: This should be smaller than max-threads.
• QUERY_QUEUE_STATUS_LOGGING_FREQUENCY(default: 1 minute) - Controls how often PIP queries and logs the VDB query queue status. Set to 0 to disable or a value greater than 0 (in minutes) to log at the INFO level.
• ENABLE_VDB_HEALTH_METRICS: Enables the collection and exposure of 3 new Prometheus metric counters for PIP Request Queue and better VDB health visibility:
  • requests_view_requests_count
  • requests_view_requests_count_is_source_true
  • requests_view_requests_count_is_source_false

VDB

• VDB_USERNAME - Set the username for the jdbc connection (default is pip)
• VDB_PASSWORD - Set the username for the jdbc connection (default is pa33word)
• TRANSPORT_PROTOCOL - VDB transport protocol (default is teiid)
• TRANSPORT_BIND_ADDRESS - Transport bind address (default is 0.0.0.0)

Logging

The available log levels are: error, warn, info, trace, and debug

• LOGGING_FORMAT - Logging format - JSON or text (default is JSON)
• LOGGING_ROOT_LEVEL - Root logging level (default is error)
• LOGGING_TOMCAT_LEVEL - Tomcat logging level (default is error)
• LOGGING_WEB_LEVEL - Web logging level (default is error)
• LOGGING_PLAINID_LEVEL - PlainID components logging level (default is debug)
• LOGGING_TEIID_LEVEL - Teiid logging level (default is info)
• LOGGING_TEIID_RUNTIME_LEVEL - Teiid runtime logging level (default is info)
• LOGGING_TEIID_CONNECTOR_LEVEL - Teiid connector logging level (default is info)
• LOGGING_TEIID_COMMAND_LOG_LEVEL - Teiid command log level (default is info)

Proxy

• PROXY_SOCKET_TIMEOUT - How long should the proxy socket remain open after initial handshake (default is 2 hours)
• PROXY_SOCKET_BUFFER_SIZE - Buffer size of the proxy input stream (default is 8192 bytes)
• PROXY_SERVER_PORT - Which port the Tiied proxy server should listen to (default is 31350)

Agent

• AGENT_URL - URL of the agent on the client side (default is http://localhost:8761)
   

For users using their own deployment procedure (instead of Helm or Standalone), ensure that you add the PAA Version Number to the Agent Deployment.
Environment Variable Example: PAA_PACKAGE_VERSION: PAA VERSION NUMBER

Health-check

URL

http://<host>:<port>/actuator/health

OK Response

{
   "status": "UP"
}

Java

The PlainID Agent runs with Java 17 and above.

Command Line

    java -jar pip-operator-1.0.jar