--- title: "Installing and Configuring the Authorizer" slug: "access-file-authorizer" updated: 2026-06-10T09:50:30Z published: 2026-06-10T09:50:30Z canonical: "docs.plainid.io/access-file-authorizer" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.plainid.io/llms.txt > Use this file to discover all available pages before exploring further. # Installing and Configuring the Authorizer The Access File Authorizer generates access files based on policy evaluations and serves them to connected systems. It integrates with data sources and PDP flows to calculate access decisions and supports various configurations, including templates and webhooks, to align with specific organizational requirements. Proper installation, configuration, and integration ensure consistent and accurate access enforcement across environments. ## Prerequisites Users need these prerequisites before installing the Access File Authorizer: - PlainID Private Docker Hub access - An operational database: PostgreSQL with **read and write** permissions. - Access to the relevant **Subjects** Data source database. - Allocated POD resources: Minimum 2 CPU and 4GB of memory. - Access permission on the POD "/app/data" folder to write files. - Access to Storage for output files writing/copying. - PAA Load Requirements: Ability to accommodate extensive load by configuring the PDP and PIP replicas as required. ## How to Install and Upgrade The Access File Authorizer deploys as a K8s Stateful Set. It should run alongside a PAA deployment as it integrates with the PDP and PIP. **To install or upgrade**: 1. Obtain an Authorizer pack from PlainID Global Services that includes the Helm Chart and other config files. 2. Use the `values-custom.yaml` file to define the Helm Chart specifications as per your Environment's deployment requirements. - Configure the pull secret for accessing Docker images from **PlainID Docker Hub**. - Configure the Authorizer using a **Config Map** and **Environment Variables** (under the `extraEnv:` section): - **Database Connection**: Specify details for the database used in data processing (e.g., PostgreSQL). - **Data Source Configuration**: Provide connection details for the relevant data source. - **PDP Configuration**: Adjust settings according to your deployment requirements. - **Jobs & Flows Configuration**: Define parameters for processing jobs and flows. - Refer to the [example configuration](/docs/access-file-authorizer#basic-configuration) for more information. - Enable `rclone` support to use our rclone integration to copy generated files to other destinations using `rclone` capabilities such as SFTP transfer (optional). See detailed information about this integration [here](/v1/docs/access-file-authorizer#rclone). 3. Install the Helm Chart using the custom `custom-values.yaml` file. - Deploy the Access File POD within the same namespace as the PAA (recommended). ```bash helm install authz-access-file --values values-custom.yaml ``` 4. Post-deployment validation - Check that the Access File service is up and running. Review the logs for any significant errors. - The error `ERROR: rabbit component is null. skipping fetching schemas from tenant-mgmt` can be safely ignored. - During the initial deployment, the service startup should also create tables in the operational DB you configured. The tables that will be created are, `decisions` table, `goose_db_version` table and `jobs_history` table. You can validate that these tables created successfully in the DB. - In case the `db.tablesPrefix` is defined (see details below), these tables will be created with prefixed names. _Note: We recommend defining a test job in the configuration with a test flow, targeting a limited subject population and running this job as a post deployment validation._ 5. Upgrade the Authorizer version by adjusting the image tag number and relevant configuration. Ensure that you also upgrade the deployment using the Helm Upgrade: ```bash helm upgrade -i authz-access-file --values values-custom.yaml ``` ## Configuration ### Key Glossary | Term | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Jobs | The main working unit of the Authorizer.
The Authorizer triggers, executes, and manages operations at the job level.
A job can include multiple flows, resulting in different outputs from the same run. | | Flows | The basic working unit that handles the complete process of loading the subject population, managing access decisions, and generating output files.
Each Flow references specific configurations to set up the data pulled from a data source, define the PDP configuration to use, and convert authorization decisions into the output file. | | Data Sources | A set of definitions used to connect to a database and pull subject population data into the Flow process. | | PDPs | A set of definitions that manage authorization request execution during Flow processing, including the configuration of concurrency and timeouts. | | Locations | A set of definitions specifying where to generate output files during execution. | | Converters | A set of definitions that specify how to generate output files using Go Templates. | ### Structure 1. The Authorizer configuration structure is a hierarchical YAML with sets of keys and a referencing structure. 2. The configuration includes these sections which define the setup "building blocks" used by reference for Jobs: | Section | Functionality | |----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `datasources` | Specifies data sources for the subject's population used in Job processing. | | `pdps` | Configures PDP(s) for processing Jobs and handling additional Authorization request flags and definitions. | | `locations` | Specifies locations for files generated by Jobs. | | `converters` | Defines the converter template responsible for transforming PDP responses into the required file output of a Job. | | `webhooks` (optional) | Defines webhook templates for post-processing or notifications when a Job is completed.
For example, a webhook can trigger an SFTP file transfer of the generated files. | | `flows` | Outlines processing flows, including flow settings and references to Data Sources, PDPs, Converters, and Locations used in execution. | | `jobs` | Configures job processing by setting job-specific parameters, referencing flows, and optionally including special converters for aggregated files and webhooks at the job’s conclusion. | :::(Info) (YAML Config Structure) The full YAML config structure is defined in the `values-custom.yaml` under ```yaml plainIDConfig: config.yaml: ``` ::: 3. Use Environment Variables in the config.yaml to keep it readable and organized (_strongly recommended_). All configuration values support Environment Variable substitution using the following format: `${ENV_VAR_NAME:default_value}`. The Environment Variable is then defined under the `extraEnv:` section in the `values-custom.yaml` 4. In addition, the `values-custom.yaml` includes additional service configurations such as, Image, Resources, Ingress etc. 5. The `values-custom.yaml` should also be used for enabling the `rclone` integration if required. If enabled the deployment of the Authorizer also deploys an additional container running the `rclone` service, which can be used by the Authorizer using webhooks. See more details about `rclone` and webhooks [here](/v1/docs/access-file-authorizer#rclone). ## Secret Management Configuration The Access File Authorizer supports integration with the PlainID Secret Management service for resolving credentials and other sensitive values from configured secret stores. To enable Secret Management resolution in the Authorizer, add the following configuration under the root of the Authorizer `config.yaml`: ```yaml secretManager: useInConfiguration: ${SECRET_MANAGER_ENABLED:true} apiClient: url: ${SECRET_MANAGER_URL} timeout: ${SECRET_MANAGER_TIMEOUT:40s} cacheManager: ttl: ${SECRET_MANAGER_CACHE_TTL:1h} returnEmptyValueOnError: ${SECRET_MANAGER_RETURN_EMPTY_VALUE_ON_ERROR:true} retryTimeoutOnError: ${SECRET_MANAGER_RETRY_TIMEOUT_ON_ERROR:30s} ```` ### Parameters | Parameter | Description | | ------------------------- | ---------------------------------------------------------------------------- | | `useInConfiguration` | Enables resolving Secret Management references in configuration values | | `apiClient.url` | Secret Management service endpoint. The Secret Management Service used by the Authorizer. This URL should be exposed in the Secret Management instance, which is deployed as part of the PAA. | | `apiClient.timeout` | Timeout for Secret Management API calls | | `returnEmptyValueOnError` | Returns empty values if secret resolution fails instead of failing execution | | `retryTimeoutOnError` | Time to wait before retrying failed secret retrieval | | `cacheManager.ttl` | Cache duration for resolved secrets. The secret is refreshed after sending a request to the Secret Management service. | After Secret Management is configured, supported configuration values can use secret references with the standard syntax: ```text {{store=,key=,refreshInterval=}} ``` ## Redis The Access File Authorizer requires a Redis instance for internal runtime operations. Redis is used by the service for shared state management and coordination during job processing. Configure the Redis connection under the `redis` section of the Authorizer configuration. A typical configuration using environment variables: ```yaml redis: host: ${REDIS_HOST:localhost} port: ${REDIS_PORT:6379} password: ${REDIS_PASS} db: ${REDIS_DB:0} ssl: ${REDIS_SSL:false} ``` The environment variables can then be defined in the `extraEnv` section of `values-custom.yaml`: ```yaml extraEnv: REDIS_HOST: "my-plainid-paa-redis-master" REDIS_PASS: "" REDIS_DB: "0" ``` Refer to the **[Configuration Parameters](/v1/docs/access-file-authorizer#parameters)** table for a detailed description of the supported Redis configuration fields. #### Secret Manager Example Sensitive values such as the Redis password can also be retrieved from a Secret Manager instead of being stored directly in configuration files. ```yaml redis: host: ${REDIS_HOST} port: ${REDIS_PORT:6379} password: {{store=AZURE_KEY_VAULT_STORE,key=redis-password}} db: ${REDIS_DB:0} ``` Using Environment Variables or Secret Managers is recommended to avoid storing sensitive credentials directly in configuration files. Refer to [Access File Authorizer Connections](/v1/docs/access-file-authorizer-connections){target=`_blank`} for a detailed use case. ## Operational Database Credentials The Authorizer connects to an operational database used for runtime processing and job management. Database credentials can be configured directly or retrieved from a Secret Manager. **Standard configuration** ```yaml db: username: ${DB_USER} password: ${DB_PASS} ``` **Using Secret Management** ```yaml db: username: {{store=AZURE_KEY_VAULT_STORE,key=db-username}} password: {{store=AZURE_KEY_VAULT_STORE,key=db-password}} ``` Refer to the **[Configuration Parameters](/v1/docs/access-file-authorizer#parameters)** table for a detailed description of the supported fields. See **[Access File Authorizer Connections](/v1/docs/access-file-authorizer-connections)** for detailed examples of configuring Secret Management with the Authorizer. ### Parameters | Name | Required | Description | Value Examples | | --------------------------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `management.port` | Yes | Management API port | `9090` | | `http.port` | Yes | HTTP server port | `8080` | | `http.enableXSSValidator` | No | Enable XSS validation | `true`, `false` | | `http.jwt.jwtIgnoreVerification` | No | Ignore JWT verification | `true`, `false` | | `log.level` | No | Logging level | `trace`, `debug`, `info`, `warn`, `warning`, `error`, `fatal`, `panic` | | `log.format` | No | Logging format | `text`, `json` | | `redis.host` | Yes | Redis host | `"localhost"`, `"my-plainid-paa-redis-master"` | | `redis.port` | Yes | Redis port | `6379` | | `redis.password` | Yes | Redis password
Supports Secret Management references. See the [Redis](/v1/docs/access-file-authorizer#redis) section for more information. | `"secret123"` | | `redis.db` | No | Redis database number | `0` | | `db.enabled` | Yes | Enable database integration | `true` | | `db.username` | Yes | Database username
Supports Secret Management references. See the [Operational Database](/v1/docs/access-file-authorizer#operational-database-credentials) section for more information. | `"postgres"` | | `db.password` | Yes | Database password
Supports Secret Management references. See the [Operational Database](/v1/docs/access-file-authorizer#operational-database-credentials) section for more information. | `"secret123"` | | `db.host` | Yes | Database host | `"localhost"`, `"my-plainid-paa-postgresql"` | | `db.port` | Yes | Database port | `5432` | | `db.database` | Yes | Database name | `"authz_db"` | | `db.driver` | Yes | Database driver type | `"postgres"` | | `db.migrationSchemas` | No | Schemas to apply migrations | `["public"]` | | `db.migrationType` | No | Type of migrations to run | `"goose"` | | `db.migrationOnStart` | No | Run migrations on startup | `true`, `false` | | `db.schema` | No | Default database schema | `"public"` | | `db.ssl` | No | Enable SSL for database | `true`, `false` | | `db.maxIdleConns` | No | Max idle connections | `10` | | `db.maxOpenConns` | No | Max open connections | `10` | | `db.connMaxLifetime` | No | Connection max lifetime in seconds | `3600` | | `db.tablesPrefix` | No | Prefix applied to the Operational DB tables created during deployment to manage the Authorizer process. These tables include `decisions`, `jobs_history`, and `goose_db_version` tables, and if prefix param is defined they will be created with prefixed names. This enables table isolation in case the Operational DB is utilized for multiple instances of the Authorizer. | `"app1"` prefix results in`app1_decisions` table | | `datasources..username` | Yes | Data source username.
Supports Secret Management references. See the [Data Source Credentials](/v1/docs/access-file-authorizer#data-source-connections) section for more information. | `"pip"` | | `datasources..password` | Yes | Data source password
Supports Secret Management references. See the [Data Source Credentials](/v1/docs/access-file-authorizer#data-source-connections) section for more information. | `"pa33word"` | | `datasources..host` | Yes | Data source host | `"localhost"` | | `datasources..port` | Yes | Data source port | `30303` | | `datasources..database` | Yes | Data source database name | `"vdb"` | | `datasources..sslmode` | No | Enable SSL for data source | `true`, `false` | | `datasources..maxIdleConns` | No | Data source max idle connections | `10` | | `datasources..maxOpenConns` | No | Data source max open connections | `10` | | `datasources..connMaxLifetime` | No | Data source connection max lifetime in seconds | `3600` | | `datasources..connectTimeout` | No | Data source connection timeout in seconds | `10` | | `pdps..type` | Yes | PDP type | `"runtime"` | | `pdps..runtimeParameters.url` | Yes | PDP runtime URL | `"http://localhost:30040"` | | `pdps..runtimeParameters.timeout` | No | PDP request timeout | `"30s"` | | `pdps..runtimeParameters.maxConcurrentConnections` | No | Max concurrent PDP connections | `5` | | `pdps..runtimeParameters.ignoreSSL` | No | Ignore SSL verification for PDP | `true`, `false` | | `locations..path` | Yes | Output location path | `"/path/to/output"` | | `converters..type` | Yes | Converter type | `"goTemplate"` | | `converters..templateProperties.content` | Yes | Template content | Go template string | | `flows..mode` | No | Flow execution mode | `"Normal"`, `"Full"` | | `flows..retries` | No | Number of retry cycles for failed PDP requests.
Default `0`, maximum `5`. | `0`, `1`, `3` | | `flows..retryThreshold` | No | Maximum percentage of PDP failures allowed to trigger retry cycles. If exceeded, retries are skipped.
Default 0 | `10`, `25`, `50` | | `flows..skipFileGenerationOnError` | No | Skip file generation for the flow if PDP errors exist in the current run.
Default `false`. | `true`, `false` | | `flows..source.datasource` | Yes | Flow data source reference | `"ds1"` | | `flows..source.schema` | Yes | Flow source schema | `"public"` | | `flows..source.table` | Yes | Flow source table | `"users"` | | `flows..source.uid` | Yes | Flow population source unique identifier column name | `"id"` | | `flows..pdp.id` | Yes | Flow PDP reference | `"pdp1"` | | `flows..pdp.runtimeParameters.type` | Yes | Flow PDP runtime type | `"userList"`, `"resolution"`, `"userAccessToken"` | | `flows..pdp.runtimeParameters.userListParameters.resourceType` | Yes* | Resource type for user list | `"groups-new"` | | `flows..pdp.runtimeParameters.userAccessTokenParameters.entityTypeId` | Yes* | Entity type ID | `"entity-type-1"` | | `flows..pdp.runtimeParameters.path` | Yes | PDP API path | `"/api/runtime/userlist/v3"`, `"/api/runtime/resolution/v3"`, `"/api/runtime/token/v3"` | | `flows..pdp.runtimeParameters.clientId` | Yes | PDP client ID.
Supports Secret Management references. See the [PDP Authentication](/v1/docs/access-file-authorizer#pdp-authentication) section for more information. | `"client123"` | | `flows..pdp.runtimeParameters.clientSecret` | Yes | PDP client secret. PDP client ID.
Supports Secret Management references. See the [PDP Authentication](/v1/docs/access-file-authorizer#pdp-authentication) section for more information. | `"secret123"` | | `flows..pdp.runtimeParameters.includeAsset` | No | Include Asset in PDP response | `true`, `false` | | `flows..pdp.runtimeParameters.includeIdentity` | No | Include Identity in PDP response | `true`, `false` | | `flows..pdp.runtimeParameters.includeAccessPolicy` | No | Include Access Policy in PDP response | `true`, `false` | | `flows..convert.batchSize` | No | Processing batch size | `10` | | `flows..convert.converters..id` | Yes | Converter reference | `"t1"` | | `flows..convert.converters..output.transient` | No | Mark output as temporary | `true`, `false` | | `flows..convert.converters..output.location.id` | Yes | Output location reference | `"l1"` | | `flows..convert.converters..output.location.filenameTemplate` | Yes | Output filename template | `"users-{{nowWithFormat \"20060102\"}}.json"` | | `jobs..flows` | Yes | List of flows to execute (executed in order) | `["flow1", "flow2"]` | | `jobs..maxWorkers` | Yes | Max concurrent workers | `100` | | `jobs..mode` | No | Job execution mode | `"Normal"`, `"Full"` | | `jobs..schedule` | No | Cron schedule expression | `"0 * * * *"` | | `jobs..timeout` | No | Job execution timeout (default: 30m) | `"24h"` | | `jobs..skipFileGenerationOnError` | No | Skip job-level file generation if any flow skipped file generation due to PDP errors. Job status will be set to `FILE_GENERATION_SKIPPED`.
Default value is `False` | `true`, `false` | | `jobs..converters..id` | Yes* | Job converter reference | `"t1"` | | `jobs..converters..output.location.filenameTemplate` | Yes* | Job output filename template | `"output-{{nowWithFormat \"20060102\"}}.json"` | | `jobs..webhooks.template` | Yes (if used) | Reference to webhook template | `"copyFileWebhook"` | | `jobs..webhooks.template.variables` | No | Variables injected into webhook payload.
Supports Secret Management references. See the [Webhooks](/v1/docs/access-file-authorizer#webhooks) section for more information. | See example | ### Examples #### Basic Configuration ```yaml management: port: ${MANAGEMENT_PORT:8081} http: port: ${APP_PORT:8080} enableXSSValidator: true jwt: jwtIgnoreVerification: ${JWT_IGNORE_VERIFICATION:true} redis: host: ${REDIS_HOST:localhost} port: ${REDIS_PORT:6379} password: ${REDIS_PASS} db: ${REDIS_DB:0} db: enabled: true username: ${DB_USER:offline} password: ${DB_PASS:offline} host: ${DB_HOST:localhost} port: ${DB_PORT:5432} database: ${DB_DATABASE:offline} driver: postgres migrationSchemas: - ${DB_SCHEMA:public} migrationType: goose migrationOnStart: ${DB_MIGRATION_ON_START:true} schema: ${DB_SCHEMA:public} ssl: ${DB_SSL:false} maxIdleConns: ${DB_MAX_IDLE_CONNECTIONS:1} maxOpenConns: ${DB_MAX_OPEN_CONNECTIONS:10} connMaxLifetime: ${DB_CONNECTIONS_MAX_LIFE_TIME_SECONDS:3600} datasources: ds1: username: ${DS1_DB_USER:pip} password: ${DS1_DB_PASS:pa33word} host: ${DS1_DB_HOST:localhost} port: ${DS1_DB_PORT:30303} database: ${DS1_DB_DATABASE:vdb} sslmode: ${DS1_DB_SSL_MODE:false} connectTimeout: ${DS1_CONNECT_TIMEOUT:10} pdps: pdp1: type: ${PDP1_TYPE:runtime} runtimeParameters: url: ${PDP1_URL:http://localhost:30040} maxConcurrentConnections: ${PDP1_MAX_CONCURRENT_CONNECTIONS:5} timeout: ${PDP1_TIMEOUT:30s} ignoreSSL: ${PDP1_IGNORE_SSL:false} locations: l1: path: ${LOCATION1_PATH} converters: t1: type: goTemplate templateProperties: content: ${CONVERTER1_CONTENT} t2: type: goTemplate templateProperties: content: ${JOB1_CONVERTER1_CONTENT} flows: flow1: mode: Full source: datasource: ds1 schema: ${FLOW1_SOURCE_SCHEMA} table: ${FLOW1_SOURCE_TABLE} uid: ${FLOW1_SOURCE_UID} pdp: id: pdp1 runtimeParameters: type: ${FLOW1_PDP_RUNTIME_PARAMETERS_TYPE:userList} userListParameters: resourceType: ${FLOW1_PDP_RUNTIME_PARAMETERS_USER_LIST_PARAMETERS_RESOURCE_TYPE} userAccessTokenParameters: entityTypeId: ${FLOW1_PDP_RUNTIME_PARAMETERS_USER_ACCESS_TOKEN_PARAMETERS_ENTITY_TYPE_ID} resolutionParameters: entityTypeId: ${FLOW1_PDP_RUNTIME_PARAMETERS_RESOLUTION_PARAMETERS_ENTITY_TYPE_ID} path: ${FLOW1_PDP_RUNTIME_PARAMETERS_PATH} clientId: ${FLOW1_PDP_RUNTIME_PARAMETERS_CLIENT_ID} clientSecret: ${FLOW1_PDP_RUNTIME_PARAMETERS_CLIENT_SECRET} includeAsset: true includeIdentity: true includeAccessPolicy: false convert: batchSize: ${FLOW1_CONVERT_BATCH_SIZE:10} converters: c1: id: t1 output: transient: true location: id: l1 filenameTemplate: ${FLOW1_CONVERTER1_FILENAME_TEMPLATE:output-jb1-flow1-{{nowWithFormat "20060102-150405"}}.json} jobs: jb1: flows: - flow1 maxWorkers: ${JOB1_MAX_WORKERS:100} ``` #### Advanced Configuration with Multiple Flows and Converters ```yaml management: port: ${MANAGEMENT_PORT:8081} http: port: ${APP_PORT:8080} enableXSSValidator: true jwt: jwtIgnoreVerification: ${JWT_IGNORE_VERIFICATION:true} log: level: info format: text redis: host: ${REDIS_HOST:localhost} port: ${REDIS_PORT:6379} password: ${REDIS_PASS} db: ${REDIS_DB:0} db: enabled: true username: ${DB_USER:offline} password: ${DB_PASS:offline} host: ${DB_HOST:localhost} port: ${DB_PORT:5432} database: ${DB_DATABASE:offline} driver: postgres migrationSchemas: - ${DB_SCHEMA:public} migrationType: goose migrationOnStart: ${DB_MIGRATION_ON_START:true} schema: ${DB_SCHEMA:public} ssl: ${DB_SSL:false} maxIdleConns: ${DB_MAX_IDLE_CONNECTIONS:1} maxOpenConns: ${DB_MAX_OPEN_CONNECTIONS:10} connMaxLifetime: ${DB_CONNECTIONS_MAX_LIFE_TIME_SECONDS:3600} tablesPrefix: ${DB_TABLES_PREFIX} datasources: ds1: username: ${DS1_DB_USER:pip} password: ${DS1_DB_PASS:pa33word} host: ${DS1_DB_HOST:localhost} port: ${DS1_DB_PORT:30303} database: ${DS1_DB_DATABASE:vdb} sslmode: ${DS1_DB_SSL_MODE:false} connectTimeout: ${DS1_CONNECT_TIMEOUT:10} pdps: pdp1: type: ${PDP1_TYPE:runtime} runtimeParameters: url: ${PDP1_URL:http://localhost:30040} maxConcurrentConnections: ${PDP1_MAX_CONCURRENT_CONNECTIONS:5} timeout: ${PDP1_TIMEOUT:30s} ignoreSSL: ${PDP1_IGNORE_SSL:false} locations: l1: path: ${LOCATION1_PATH} converters: t1: type: goTemplate templateProperties: content: ${CONVERTER1_CONTENT} t2: type: goTemplate templateProperties: content: ${JOB1_CONVERTER1_CONTENT} flows: flow1: mode: Full source: datasource: ds1 schema: ${FLOW1_SOURCE_SCHEMA} table: ${FLOW1_SOURCE_TABLE} uid: ${FLOW1_SOURCE_UID} pdp: id: pdp1 runtimeParameters: type: ${FLOW1_PDP_RUNTIME_PARAMETERS_TYPE:userList} userListParameters: resourceType: ${FLOW1_PDP_RUNTIME_PARAMETERS_USER_LIST_PARAMETERS_RESOURCE_TYPE} userAccessTokenParameters: entityTypeId: ${FLOW1_PDP_RUNTIME_PARAMETERS_USER_ACCESS_TOKEN_PARAMETERS_ENTITY_TYPE_ID} resolutionParameters: entityTypeId: ${FLOW1_PDP_RUNTIME_PARAMETERS_RESOLUTION_PARAMETERS_ENTITY_TYPE_ID} path: ${FLOW1_PDP_RUNTIME_PARAMETERS_PATH} clientId: ${FLOW1_PDP_RUNTIME_PARAMETERS_CLIENT_ID} clientSecret: ${FLOW1_PDP_RUNTIME_PARAMETERS_CLIENT_SECRET} includeAsset: true includeIdentity: true includeAccessPolicy: false convert: batchSize: ${FLOW1_CONVERT_BATCH_SIZE:10} converters: c1: id: t1 output: transient: true location: id: l1 filenameTemplate: ${FLOW1_CONVERTER1_FILENAME_TEMPLATE:output-jb1-flow1-{{nowWithFormat "20060102-150405"}}.json} jobs: jb1: flows: - flow1 maxWorkers: ${JOB1_MAX_WORKERS:100} schedule: "00 18 * * *" converters: jc1: id: t2 output: location: id: l1 filenameTemplate: ${JOB1_CONVERTER1_FILENAME_TEMPLATE:output-aggregate1-jb1-{{nowWithFormat "20060102-150405"}}.json} ``` **Full Example** This example includes the **Basic** and **Advanced Configuration with Multiple Flows and Converters** examples. ```yaml # Should contain a configuration file matching usage detailed in the documentation. See examples/values-custom.yaml. imagePullSecrets: - name: "image-pull-secret" rclone: enabled: true image: repository: "docker.io/rclone/rclone" tag: "1.69" config: | [my-sftp] type = sftp host = sftp-service user = username pass = port = 22 ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx hosts: - host: access-file-authz.ps-cluster.plainid.net paths: [ "/" ] resources: requests: memory: "4000Mi" cpu: "2000m" limits: memory: "4000Mi" cpu: "2000m" podSecurityContext: runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 fsGroupChangePolicy: "Always" plainIDConfig: config.yaml: debug: profiling: enable: ${DEBUG_PROFILING_ENABLE:false} management: port: ${MANAGEMENT_PORT:8081} http: port: ${APP_PORT:8080} enableXSSValidator: true jwt: jwtIgnoreVerification: ${JWT_IGNORE_VERIFICATION:true} log: level: debug format: text redis: host: ${REDIS_HOST:localhost} port: ${REDIS_PORT:6379} password: ${REDIS_PASS} db: ${REDIS_DB:0} db: enabled: true username: ${DB_USER:offline} password: ${DB_PASS:offline} host: ${DB_HOST:localhost} port: ${DB_PORT:5432} database: ${DB_DATABASE:offline} driver: postgres migrationSchemas: - ${DB_SCHEMA:public} migrationType: goose migrationOnStart: ${DB_MIGRATION_ON_START:true} schema: ${DB_SCHEMA:public} ssl: ${DB_SSL:false} maxIdleConns: ${DB_MAX_IDLE_CONNECTIONS:1} maxOpenConns: ${DB_MAX_OPEN_CONNECTIONS:10} connMaxLifetime: ${DB_CONNECTIONS_MAX_LIFE_TIME_SECONDS:3600} tablesPrefix: ${DB_TABLES_PREFIX} datasources: ds1: username: ${DS1_DB_USER:pip} password: ${DS1_DB_PASS:password} host: ${DS1_DB_HOST:localhost} port: ${DS1_DB_PORT:30303} database: ${DS1_DB_DATABASE:vdb} sslmode: ${DS1_DB_SSL_MODE:false} connectTimeout: ${DS1_CONNECT_TIMEOUT:10} pdps: sample-pdp: type: runtime runtimeParameters: url: ${SAMPLE_PDP_URL:http://localhost:30040} maxConcurrentConnections: ${SAMPLE_PDP_MAX_CONCURRENT_CONNECTIONS:5} timeout: 30s ignoreSSL: false locations: location_sample: path: ${SAMPLE_JOB_LOCATION_PATH} converters: template1: type: goTemplate templateProperties: content: ${CONVERTER1_TEMPLATE} t2_1: type: goTemplate templateProperties: content: ${CONVERTER2_1_TEMPLATE} t2_2: type: goTemplate templateProperties: content: ${CONVERTER2_2_TEMPLATE} t2_3: type: goTemplate templateProperties: content: ${CONVERTER2_3_TEMPLATE} aggregated_file_sample: type: goTemplate templateProperties: content: | { "property-A": [ { {{ fileInput "flow-sample-2" "c2_1" }}{{ fileInput "flow-sample-2" "c2_2" }}{{ fileInput "flow-sample-2" "c2_3" }}] } flows: flow-sample-1: mode: Full retries: ${FLOW1_RETRIES:0} retryThreshold: ${FLOW1_RETRY_THRESHOLD:0} skipFileGenerationOnError: ${FLOW1_SKIP_FILE_GENERATION_ON_ERROR:false} source: datasource: ds1 schema: ${FLOW_SAMPLE_1_SOURCE_SCHEMA} table: ${FLOW_SAMPLE_1_SOURCE_TABLE} uid: ${FLOW_SAMPLE_1_SOURCE_UID} pdp: id: sample-pdp runtimeParameters: type: ${SAMPLE_PDP_RUNTIME_PARAMETERS_TYPE} userAccessTokenParameters: entityTypeId: ${FLOW_SAMPLE_1_PDP_RUNTIME_PARAMETERS_USER_ACCESS_TOKEN_PARAMETERS_ENTITY_TYPE_ID} path: ${SAMPLE_PDP_RUNTIME_PARAMETERS_PATH} clientId: ${SAMPLE_PDP_RUNTIME_PARAMETERS_CLIENT_ID} clientSecret: ${SAMPLE_PDP_RUNTIME_PARAMETERS_CLIENT_SECRET} includeAsset: true includeIdentity: true includeAccessPolicy: false convert: batchSize: 10 converters: c1: id: template1 output: transient: false location: id: location_sample filenameTemplate: sample-file-1-output-{{nowWithFormat "20060102-150405"}}.json flow-sample-2: mode: Full source: datasource: ds1 schema: ${FLOW_SAMPLE_2_SOURCE_SCHEMA} table: ${FLOW_SAMPLE_2_SOURCE_TABLE} uid: ${FLOW_SAMPLE_2_SOURCE_UID} pdp: id: sample-pdp runtimeParameters: type: ${SAMPLE_PDP_RUNTIME_PARAMETERS_TYPE:userList} userAccessTokenParameters: entityTypeId: ${FLOW_SAMPLE_2_PDP_RUNTIME_PARAMETERS_USER_ACCESS_TOKEN_PARAMETERS_ENTITY_TYPE_ID} path: ${SAMPLE_PDP_RUNTIME_PARAMETERS_PATH} clientId: ${SAMPLE_PDP_RUNTIME_PARAMETERS_CLIENT_ID} clientSecret: ${SAMPLE_PDP_RUNTIME_PARAMETERS_CLIENT_SECRET} includeAsset: true includeIdentity: true includeAccessPolicy: false convert: batchSize: 10 converters: c2_1: id: t2_1 output: transient: true location: id: location_sample filenameTemplate: temp-output-2_1-{{nowWithFormat "20060102-150405"}}.json c2_2: id: t2_2 output: transient: true location: id: location_sample filenameTemplate: temp-output-2_2-{{nowWithFormat "20060102-150405"}}.json c2_3: id: t2_3 output: transient: true location: id: location_sample filenameTemplate: temp-output-2_3-{{nowWithFormat "20060102-150405"}}.json webhooks: copyFileWebhook: method: "POST" headers: - name: Content-Type values: - "application/json" - name: Authorization values: - ${RCLONE_BASIC_BASE64} url: ${RCLONE_URL}/operations/copyfile payload: | { "srcFs": "{{ .SrcFs }}", "srcRemote": "{{ .SrcRemote }}", "dstFs": "{{ .DstFs }}", "dstRemote": "{{ .DstRemote }}", "_config": { "inplace": true } } timeout: 180 deleteFileWebhook: method: "POST" headers: - name: Content-Type values: - "application/json" - name: Authorization values: - ${RCLONE_BASIC_BASE64} url: ${RCLONE_URL}/operations/deletefile payload: | { "fs": "{{ .SrcFs }}", "remote": "{{ .SrcRemote }}" } timeout: 180 jobs: jb_sample: timeout: ${SAMPLE_JOB_TIMEOUT:30m} flows: - flow-sample-1 - flow-sample-2 maxWorkers: ${SAMPLE_JOB_MAX_WORKERS:100} skipFileGenerationOnError: ${JOB1_SKIP_FILE_GENERATION_ON_ERROR:false} schedule: "00 02 * * 1" converters: jb_sample_aggregated_converter: id: aggregated_file_sample output: location: id: location_sample filenameTemplate: sample-aggregated-file-output-{{nowWithFormat "20060102"}}.json webhooks: - template: "copyFileWebhook" variables: - name: SrcFs value: /app/data/ - name: SrcRemote value: '{{ fileName "flow" "flow-sample-1" "c1" }}' - name: DstFs value: "my-sftp:/upload" - name: DstRemote value: "sample-1-file-name.json" - template: "copyFileWebhook" variables: - name: SrcFs value: /app/data/ - name: SrcRemote value: '{{ fileName "job" "jb_sample" "jb_sample_aggregated_converter" }}' - name: DstFs value: "my-sftp:/upload" - name: DstRemote value: "sample-aggregated-file-name.json" appDataPersistence: enabled: true extraEnv: DEBUG_PROFILING_ENABLE: 'true' #GOMEMLIMIT: '3900MiB' #GOMAXPROCS: '2' SAMPLE_JOB_MAX_WORKERS: '50' # Processing DB DB_USER: 'postgres' DB_PASS: '' DB_DATABASE: 'accessfile' DB_HOST: 'alexb-persistent-mat-view.cmo3hixnsmm5.us-east-1.rds.amazonaws.com' DB_SSL: 'true' DB_SCHEMA: '' REDIS_HOST: '' REDIS_PASS: '' DB_TABLES_PREFIX: 'app1' # Data Source Params DS1_DB_USER: 'postgres' DS1_DB_PASS: '' DS1_DB_HOST: '.us-east-1.rds.amazonaws.com' DS1_DB_PORT: '5432' DS1_DB_DATABASE: '' DS1_DB_SSL_MODE: 'true' DS1_CONNECT_TIMEOUT: 10 # PDP Params SAMPLE_PDP_URL: 'http://plainid-paa-runtime' SAMPLE_PDP_RUNTIME_PARAMETERS_PATH: '/api/runtime/token/v3' SAMPLE_PDP_RUNTIME_PARAMETERS_CLIENT_ID: '' SAMPLE_PDP_RUNTIME_PARAMETERS_CLIENT_SECRET: '' SAMPLE_PDP_RUNTIME_PARAMETERS_TYPE: 'userAccessToken' SAMPLE_PDP_MAX_CONCURRENT_CONNECTIONS: '100' # Flow Sample 1 definitions FLOW_SAMPLE_1_PDP_RUNTIME_PARAMETERS_USER_ACCESS_TOKEN_PARAMETERS_ENTITY_TYPE_ID: '' #FLOW_SAMPLE_1_PDP_RUNTIME_PARAMETERS_USER_LIST_PARAMETERS_RESOURCE_TYPE: '' FLOW_SAMPLE_1_SOURCE_SCHEMA: 'public' FLOW_SAMPLE_1_SOURCE_TABLE: '' FLOW_SAMPLE_1_SOURCE_UID: '' # Flow Sample 2 definitions FLOW_SAMPLE_2_PDP_RUNTIME_PARAMETERS_USER_ACCESS_TOKEN_PARAMETERS_ENTITY_TYPE_ID: '' #FLOW_SAMPLE_2_PDP_RUNTIME_PARAMETERS_USER_LIST_PARAMETERS_RESOURCE_TYPE: '' FLOW_SAMPLE_2_SOURCE_SCHEMA: 'public' FLOW_SAMPLE_2_SOURCE_TABLE: '
' FLOW_SAMPLE_2_SOURCE_UID: '' # Converters Params CONVERTER1_TEMPLATE: '{{ "\r\n" }}"property-A": [{{ "\r\n" }}{{- range $index, $data := . }}{{ if gt $index 0 }},{{ "\r\n" }}{{ end }}{ {{ "\r\n" }}"UserID": [{{ "\r\n" }}{{- range $entityIndex, $entity := (index $data.response 0).access }}{{ if $entityIndex }},{{ "\r\n" }}{{ end }}"{{ $entity.path }}"{{- end }}{{ "\r\n" }}],{{ "\r\n" }}"details": { {{ "\r\n" }}"name": "{{ index $data.identity.attributes.Name 0 }}",{{ "\r\n" }}"ID": "{{ index $data.identity.attributes.ID 0 }}",{{ "\r\n" }}"lastUpdated": "{{ index $data.identity.attributes.LastUpdated 0 }}"{{ "\r\n" }} } {{ "\r\n" }} } {{- end }}{{ "\r\n" }}] {{ "\r\n" }}' CONVERTER2_1_TEMPLATE: '"Type1":[{{ "\r\n" }}{{- $first := true }}{{- range $i, $data := . }}{{- range $response := $data.response }}{{- range $access := $response.access }}{{- if eq (index $access.attributes.path 0) "TYPE-1" }}{{- range $userid := $access.attributes.UserID }}{{- if not $first }},{{ "\r\n" }}{{ end }}"{{ $userid }}"{{- $first = false }}{{- end }}{{- end }}{{- end }}{{- end }}{{- end }}{{ "\r\n" }}],' CONVERTER2_2_TEMPLATE: '"Type2":[{{ "\r\n" }}{{- $first := true }}{{- range $i, $data := . }}{{- range $response := $data.response }}{{- range $access := $response.access }}{{- if eq (index $access.attributes.path 0) "TYPE-2" }}{{- range $userid := $access.attributes.UserID }}{{- if not $first }},{{ "\r\n" }}{{ end }}"{{ $userid }}"{{- $first = false }}{{- end }}{{- end }}{{- end }}{{- end }}{{- end }}{{ "\r\n" }}],' CONVERTER2_3_TEMPLATE: '"Type3":[{{ "\r\n" }}{{- $first := true }}{{- range $i, $data := . }}{{- range $response := $data.response }}{{- range $access := $response.access }}{{- if eq (index $access.attributes.path 0) "TYPE-3" }}{{- range $userid := $access.attributes.UserID }}{{- if not $first }},{{ "\r\n" }}{{ end }}"{{ $userid }}"{{- $first = false }}{{- end }}{{- end }}{{- end }}{{- end }}{{- end }}{{ "\r\n" }}]' # Job Params SAMPLE_JOB_LOCATION_PATH: '/app/data' SAMPLE_JOB_TIMEOUT: '180m' # RCLONE RCLONE_URL: 'http://localhost:5572' ``` --- ## Subject Data Sources The Access File Authorizer processes the subject population, calculates access for each subject, and generates an access file representing Authorizations for the entire population. To support this process, the Authorizer loads a predefined subject population from a customer's data source and evaluates each entry using a predefined PDP calculation flow based on modeled Templates and Policies in the PlainID Platform. The Authorizer processes subjects that are either Identities or Assets (resources) requiring Authorization decisions. Data sources are defined in the Authorizer configuration in two parts: - **Data Source Connectivity** – Defined under the `datasources` set in the configuration, specifying connection details such as `host`, `port`, `user`, and `password`. - **Data Reference** – Defined under `flow: source`, referencing the connectivity setup and specifying the `schema`, `table`, and `uid` to identify the data source for subjects. The Data Source configuration can either connect directly to a customer database or use the PIP service via `postgres` transport. When using PIP, connectivity is configured to the `pip-operator` through the Postgres port, referencing a View name as the Source table. This enables support for multiple data source types (not limited to databases), the creation of virtual views for defining population subsets, and necessary data normalization using PlainID PIP capabilities. The `uid` should serve as a unique identifier for the subject, such as `UserID` for Identities or `AssetID` for resources. **Note:** The subject data source must be preordered by the `uid` field to prevent data inconsistencies and issues with multi-values. Ordering can also be enforced using PIP Views. ### Data Source Connections Data Source credentials can be configured directly in the configuration or retrieved from a Secret Manager. **Standard configuration** ```yaml datasources: ds1: username: ${DS1_DB_USER} password: ${DS1_DB_PASS} ``` **Using Secret Management** ```yaml datasources: ds1: username: {{store=AZURE_KEY_VAULT_STORE,key=ds1-db-username}} password: {{store=AZURE_KEY_VAULT_STORE,key=ds1-db-password}} ``` Refer to **[Access File Authorizer Connections](/v1/docs/access-file-authorizer-connections)** for more details on configuring Secret Management with the Authorizer. ## PDP Flows The **Authorizer** processes both **Identities** and **Assets** and supports several **PDP calculation flows**: `userAccessToken`, `resolution`, and `userList`. Before configuring a Flow in the Authorizer, validate the **PDP calculation** independently. This validation ensures that the **Policy model** and templates return the expected results. Each **Flow** references a predefined **PDP configuration** and defines **Runtime parameters** used when sending authorization requests to the **PDP Runtime**. These parameters specify: - The **Policy Scope** used for evaluation (`clientId`). - The **PDP Flow type** (`userList`, `resolution`, or `userAccessToken`). - The **PDP endpoint path**. - The **Authentication method** used when calling the **PDP Runtime**. - Additional **PDP Request** and **Response Flags**. ### PDP Authentication The **Access File Authorizer** supports several authentication mechanisms when sending requests to the **PDP Runtime** based on the supported PDP Authentication patterns. The Authentication method determines how credentials are transmitted with the request. This setup must be aligned with the Authentication defined for the Scope in the Platform. Supported authentication methods include: - **Credentials sent in Request Body** `ccBody`(Default) The `clientId` and `clientSecret` are sent as part of the request body. This method maintains backward compatibility with existing configurations. - **Credentials sent as Headers** `ccHeader` The `clientId` and `clientSecret` are sent using the HTTP headers `X-Client-Id` and `X-Client-Secret` instead of the request body. - **JWT Authorization Header** `token`/`idpToken` The Authorizer authenticates using a **JWT token** sent in the `Authorization` header. ```none Authorization: Bearer ``` When this method is used, the `clientSecret` is **not** included in the request and the `clientId` **is included** in the request body. **Configuration Snippet:** ```yaml flows: flow1: ... pdp: id: pdp1 runtimeParameters: ... clientId: authType: "ccBody" # possible values: "" (default to ccBody), ccBody, ccHeader, idpToken, token clientId: clientSecret: ##can support SM with {{store=,key=}} # alternative for secret token: # can support SM with {{store=,key=}} idpSourceId: # required for idpToken authType ``` #### JWT Authentication Options When **JWT authentication** is enabled, two configuration options are supported. - **Fixed Token** A predefined **JWT token** can be configured directly or retrieved from an external **Secret Manager**. This token is sent with every **PDP request**. - **IDP Generated Token** The Authorizer can retrieve a **JWT token** dynamically from an external **Identity Provider (IDP)**. The token is generated using the configured `idpSourceId` parameter and attached to the PDP request as an `Authorization` header. This approach allows organizations to avoid static secrets and supports automated **token generation** and **rotation**. This option requires an **`idpSource`** configuration with credentials, a token url and optionally other IdP parameters for token generation: ```yaml idpSources: - id: auth1 # this id will be referenced in the pdp setup above url: # https://.../openid-connect/token as an example clientId: # or with SM {{store=,key=}} clientSecret: # or with SM {{store=,key=}} asyncRefresh: true additionalParameters: grant_type: client_credentials ... ``` #### Secret Management for PDP Authentication Credentials used in **PDP authentication** can be defined in several ways: - Direct configuration values. - **Environment Variables**. - External **Secret Manager** integrations. Using **Environment Variables** or **Secret Managers** is recommended to avoid storing sensitive credentials directly in configuration files. Refer to [Access File Authorizer Connections](https://docs.plainid.io/v1/docs/access-file-authorizer-connections) for a detailed use case. #### Example PDP Requests - **ClientID and Secret in Request Body** ```http POST /api/runtime/token/v3 Content-Type: application/json { "clientId": "file-access-policy-scope", "clientSecret": "exampleSecret", "pdp payload..." } ``` - **ClientID and Secret in Header** ```http POST /api/runtime/token/v3 Content-Type: application/json X-Client-Secret: exampleSecret X-Client-Id: exampleClientId { "pdp payload..." } ``` - **JWT Authorization** ```http POST /api/runtime/token/v3 Content-Type: application/json Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9... { "clientId": "file-access-policy-scope", "pdp payload..." } ``` ### PDP Flow Configuration Each **Flow** references a **PDP definition** and configures the parameters required for the authorization request. #### Basic Runtime Parameters and Configuration | Parameter | Description | Required | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- | | `type` | Defines the **PDP Runtime Flow** used for evaluation. Supported values: `userList`, `resolution`, `userAccessToken`. | Yes | | `path` | The **PDP Runtime API endpoint path** used to send the authorization request. | Yes | | `authType` | The authentication type used for PDP request. Default `ccBody` | Yes
*But if omitted use default CC in the body* | | `clientId` | Identifies the **Policy Scope** used by the PDP during Policy evaluation. | Yes | | `clientSecret` | Credential used for authentication when using secret-based authentication methods. Not required when using JWT authentication. | Conditional | | `resourceType` | Defines the **Asset Type** template used for Policy evaluation when using the `userList` Flow. | Conditional | | `entityTypeId` | Defines the **Identity entity type** used during Policy evaluation when using the `userAccessToken` Flow. | Conditional | | `includeAsset` | When enabled, the PDP response includes **Asset information** associated with the authorization result. | Optional | | `includeIdentity` | When enabled, the PDP response includes **Identity details**. | Optional | | `includeAccessPolicy` | When enabled, the PDP response includes the **evaluated Policy** responsible for the authorization decision. | Optional | ```yaml runtimeParameters: type: ${FLOW1_PDP_RUNTIME_PARAMETERS_TYPE:userList} userListParameters: resourceType: userAccessTokenParameters: entityTypeId: path: authType: "ccBody" clientId: ${FLOW1_PDP_RUNTIME_PARAMETERS_CLIENT_ID} clientSecret: ${FLOW1_PDP_RUNTIME_PARAMETERS_CLIENT_SECRET} includeAsset: true includeIdentity: true includeAccessPolicy: false ``` #### Advanced Runtime Parameters and Configuration The following parameters extend the runtime configuration to support more granular PDP filtering and advanced runtime customization. | Parameter | Description | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **resourceTypes** | Defines specific resource types to include in the PDP request. Each entry can specify a resource `name`, an `attributeList`, and allowed `actions`, enabling the PDP to return a filtered response based on these criteria. See the [User List](https://docs.plainid.io/apidocs/user-list), [User Access Token](https://docs.plainid.io/apidocs/user-access-token), or [Resolution](https://docs.plainid.io/apidocs/policy-resolution) API `resourceType` in the parameter list for more details | | **includeAssetAttributes** | Controls the `includeAssetAttributes` flag that is sent to the runtime. When enabled, Asset Attributes are included in the runtime response, allowing them to be used for templating | | **runtimeFinetune** | Add custom request flags to the runtime request. Values defined here can override standard configuration behavior and allow advanced request customization.
- Parameters defined under this must follow the Runtime (PDP) request flags and structure (see our [Authorization APIs](/apidocs/authorization-apis){target=`_blank`} documentation for more information) and is passed to the PDP request as is.

Example:
- `combinedMultiValue`: Controls how multi-value identity attributes are evaluated when calculating access decisions.
- `operationalFilters`: Filters that affect the Runtime behavior and results by applying additional filtering. | ```yaml runtimeParameters: type: ${FLOW1_PDP_RUNTIME_PARAMETERS_TYPE:userList} userListParameters: resourceTypes: - name: "accounts" attributeList: ["balance"] actions: ["READ"] userAccessTokenParameters: entityTypeId: resourceTypes: - name: "accounts" attributeList: ["balance"] actions: ["READ"] path: authType: "ccBody" clientId: ${FLOW1_PDP_RUNTIME_PARAMETERS_CLIENT_ID} clientSecret: ${FLOW1_PDP_RUNTIME_PARAMETERS_CLIENT_SECRET} includeAsset: true includeIdentity: true includeAccessPolicy: false includeAssetAttributes: ${FLOW1_PDP_RUNTIME_PARAMETERS_INCLUDE_ASSET_ATTRIBUTES:true} runtimeFineTune: | { "combinedMultiValue": "false", "includeAssetAttributes":"true" } ``` ### PDP Retries Behavior The bulk processing of large populations in the PDP with underlying dependencies may face request errors, mainly due to connectivity issues or data sources availability. Therefore, a retry mechanism has been implemented and can be defined at the `flow` level to guide the Authorizer on retries behavior. As detailed in the configuration section, this includes two parameters: 1. `flows..retryThreshold` - Defines the level of PDP errors that initiate retries. Higher amount of errors usually point to an unhealthy system, hence retries will not be effective and the operator should check system health. 2. `flows..retries` - Defines the number of retry cycles that take place. When the population processing ends, all PDP responses containing errors are retried with new PDP requests. ## Templates The Authorizer uses Go Templates to define file structures and inject data elements while processing access files based on PDP authorization decisions retrieved and stored for subject population. The templating capabilities are based on native Go templates (https://pkg.go.dev/text/template) and the extension capabilities of the common Spring library http://masterminds.github.io/sprig/. PlainID recommends simulating PDP requests (e.g., using Postman) and working with authentic sample responses when building and defining Go Templates. For guidance on creating Go Templates, you can refer to resources like [gotemplate.io](https://gotemplate.io/) (that implement Go templating using the native and Sprig libraries). Additionally, the PlainID Professional Services team is available for further assistance. The Authorizer also provides a template validation endpoint to help verify your Templates. For more details, see [Authorizer API Endpoints](https://docs.plainid.io/docs/access-file-authorizer#authorizer-api-endpoints). **Note:** The Authorizer processes PDP responses as a range, meaning template processing should assume a JSON array as input. When validating your template using the validation API or an online tool, ensure that your input is a JSON array and that your template accesses the data using a range syntax, such as: ```go {{- range $i, $data := . }} ``` ### Supported Template Functions Templates defined with Go Templates can include custom functions executed as part of the file generation. In addition to the supported Sprig functions, the Authorizer currently supports the custom functions: - **padRight** - Can be used in your output file when padding any line with spaces. - Syntax - `padRight ` - Example usage for completing lines to 100 char length - `{{ padRight "some text" 100 -}}` - **subtract** - Allows you to subtract one number from another. Useful when you need to handle the last items in an array by subtracting 1 from the length. - Syntax - `subtract ` - Example usage with native lan function - `subtract (len $range) 1` cfc - **fileInput** - Allows you to inject a file into the template. See a detailed explanation under [Flow Results Aggregator](/docs/access-file-authorizer#flow-results-aggregator) on how this function can be used. - Syntax - `{{ fileInput "" "" }}` - Example usage of injecting a file output inside a template - `"some text... " {{ fileInput "flow4" "c4_1" }} " some other text..."` - **fileName** - Allows you to get file name of the generated file for a specified job/flow and converter IDs. - Syntax - `{{ fileName "flow" "" "" }}` or `{{ fileName "job" "" "" }}` - Example usage of getting file name of the file generated for flow4 with converter c4_1 `{{ fileName "flow" "flow4" "c4_1" }}` - Useful for setting up a webhook and inject a filename as a variable to its payload, as an example to trigger a file transfer using SFTP. **Note:** If using any online Go Template tools, be aware that the mentioned functions are PlainID custom functions, not native Go Template functions, and cannot be tested online. ### Additional Template Hints You can use these templating hints: - Use `{{ "\r\n" }}` in the template to mark a new line. - Use `If` statements for conditional placement of data in the template. Example: - `{{ if gt $index 0 }}` - Use `-` to trim whitespace before or after the template expression, depending on its placement. - Use `ranges` to iterate over data lists. Example: - `{{- range $entityIndex, $entity := (index $data.response 0).access }}` - The `range` iterates over the elements in `.access`, which assigns the: - **index** to `$entityIndex`. - **value** of the current element to `$entity`. - `index $data.response 0` retrieves the first element in the `$data.response` array. - `.access` retrieves the `access` field from the first element of `$data.response`. - For a **Property Existence Check**, use the syntax: `{{ if index $data.asset.attributes "property" }}...{{ else }}""{{ end }}` - For a **Value Existence Check** use the syntax `{{ with index $data.asset.attributes "property" 0 }}"{{ . }}"{{ else }}{{ end }}` **Special Cases Examples** These examples demonstrate usage of range processing, native and custom functions and protection from non-present input data: - **Avoid Last Comma in Arrays** - Range Index Handling - Avoids adding a comma after the last range item by subtracting 1. - Example: `{{ if lt (subtract (len (index $data.response 0).access) 1) $entityIndex }}"{{ $entity.path }}",{{else}}"` - Use this snippet to add array data elements with a comma and newline after, excluding a comma from the last item `{{- range $index, $data := . }}{{ if gt $index 0 }},{{ "\r\n" }}{{ end }}`. - **Correct Range Length Handling** - End of Range Handling - Ensures correct index calculation when processing ranges by subtracting - Example: `{{ if lt (subtract (len $range) 1) $rangeIndex }}` - **Protect from Missing Input Data** - Handling Non-Existing Properties or Values using If statements. - Example - `"Property Name":"{{ if index $access.attributes.property 0 }}{{ with index $access.attributes.property 0 }}{{ . }}{{ else }}{{ end }}{{ else }}{{ end }}"` - In this example, `if` statements are used to check whether the referenced property exists and contains a value. ### Filename Template The output file names are also configured and can be defined using Templates including a time stamping as part of the name like this `output-jb3-flow3-{{nowWithFormat "20060102-150405"}}.json`. The `nowWithFormat` template function gets a time stamp format and uses it with the current timestamp when creating the file name. ### Flow Results Aggregator The Flow Results Aggregator enables complex file generation by combining results from multiple flows. This is particularly useful for intermediate processing steps that require creating file structures, which can later be incorporated into a more complex template. At the job level, you can define an additional template converters that take one or more Flow output files as inputs and integrates them into the defined template along with other template structures and elements. This is done by using the `File Input` template function to reference other Flow files. The syntax is: `{{ fileInput "" "" }}` For example, this template definition aggregates outputs from multiple flows into a final job output file in JSON format: ``` { "ResourceTypes": [ { {{ fileInput "flow2" "c2" }}, {{ fileInput "flow3" "c3" }}, {{ fileInput "flow4" "c4" }} } ], "AdditionalAccess": {{ fileInput "flow1" "c1" }} // where flow1 and c1 are references to the flowId and its defined converter. } ``` At the Flow level, you can configure whether the generated Flow file is temporary by setting `output: transient: true` : - If set to `true`, the Flow file is saved temporarily until it is used by the aggregation template during job completion. It is automatically cleaned up after successful aggregation. - If set to `false`, the Flow file is saved as an output file at the designated location, and injected into the aggregation template. **Example Configuration** with multiple flows and file aggregation used as the job converter: ```yaml converters: t1: ... t2: ... t3: ... t4: ... aggregator_template: type: goTemplate templateProperties: content: | { "ResourcesTypes":[ { {{ fileInput "flow2" "c2" }}, {{ fileInput "flow3" "c3" }}, {{ fileInput "flow4" "c4" }} } ], "AdditionalAccess": {{ fileInput "flow1" "c1" }} } flows: # Main flow with permanent output flow1: source: datasource: ds1 schema: ${FLOW1_SCHEMA:public} table: ${FLOW1_TABLE:users} convert: converters: c1: id: t1 output: location: id: l1 filenameTemplate: ${FLOW1_OUTPUT:users-{{nowWithFormat "20060102"}}.json} # Flows with temporary outputs flow2: source: datasource: ds1 schema: ${FLOW2_SCHEMA:public} table: ${FLOW2_TABLE:locations} convert: converters: c2: id: t2 output: transient: true # Mark as temporary location: id: l1 filenameTemplate: ${FLOW2_OUTPUT:temp-loc-{{nowWithFormat "20060102"}}.json} flow3: source: datasource: ds1 schema: ${FLOW3_SCHEMA:public} table: ${FLOW3_TABLE:departments} convert: converters: c3: id: t3 output: transient: true # Mark as temporary location: id: l1 filenameTemplate: ${FLOW3_OUTPUT:temp-dept-{{nowWithFormat "20060102"}}.json} flow4: source: datasource: ds1 schema: ${FLOW4_SCHEMA:public} table: ${FLOW4_TABLE:offices} convert: converters: c4: id: t4 output: transient: true # Mark as temporary location: id: l1 filenameTemplate: ${FLOW4_OUTPUT:temp-office-{{nowWithFormat "20060102"}}.json} jobs: complex_job: timeout: ${JOB1_TIMEOUT:24h} flows: - flow1 - flow2 - flow3 - flow4 converters: final: id: aggregator_template output: location: id: l1 filenameTemplate: ${FINAL_OUTPUT:final-{{nowWithFormat "20060102"}}.json} ``` In this example: - `flow1` generates a permanent output with user data - `flow2`, `flow3`, and `flow4` generate temporary files (marked with `transient: true`) - The job's final converter uses the `fileInput` function to: - Combine related data from flows 2-4 - Include additional data from flow1 - Generate a single aggregated output file - All temporary files (2,3 and 4) are automatically cleaned up after successful processing - Final result of the job will be file1 and aggregated file ## Webhooks The Authorizer supports configuring webhooks that are triggered at the end of Job processing. Once a Job is completed, an **optional** set of webhooks can be invoked—for example, to trigger a file transfer of output files from the Authorizer’s configured file store to an external location. Similarly, this capability can be used to send a notification to an external system that exposes a supported webhook. To use this feature, you must define a Webhook Template, specifying parameters such as URL, Method, Headers, and Payload. The template can include variable declarations within the webhook payload. Once configured, the template is referenced in the Job configuration, ensuring it is triggered upon job completion, with variable values assigned as needed. See the list of configuration parameters in the **[Configuration](/v1/docs/access-file-authorizer#configuration)** section. A common use case for webhooks is transferring output files to an external destination at the end of Job processing. To support this, the Authorizer includes built-in integration with [`rclone`](https://rclone.org/rc/). ### Secret Manager Example Webhook headers support retrieving sensitive values from a Secret Manager instead of storing them directly in configuration files. For example, the `Authorization` header can reference a secret stored in Azure Key Vault: ```yaml webhooks: copyFileWebhook: method: "POST" headers: - name: Content-Type values: ["application/json"] - name: Authorization values: ["{{store=AZURE_KEY_VAULT_STORE,key=afarclone}}"] ``` Note that using Secret Managers is recommended to avoid storing sensitive credentials directly in configuration files. Refer to [Access File Authorizer Connections](/v1/docs/access-file-authorizer-connections){target=`_blank`} for a detailed use case. ### rclone The `rclone` integration is based on deploying `rclone` as an additional container with the Authorizer service, running the rclone http server, making webhook calls to it to trigger its remote operations. To use this integration you will need to enable the `rclone` container deployment by copying to the `values-custom.yaml` the rclone section from the `values.yaml`, uncomment it, and configure it. Using Secret Managers is recommended to avoid storing sensitive credentials directly in configuration files. ### Secret Manager Example When using the `rclone` integration, authorization headers used by webhook calls can reference secrets instead of embedding credentials directly in configuration. For example, the `Authorization` header used by the `rclone` HTTP server can be configured as: ``` Authorization: basic ${RCLONE_BASIC_BASE64} ``` The `${RCLONE_BASIC_BASE64}` environment variable can be populated from a Secret Manager as part of the deployment configuration. Using Secret Managers is recommended to avoid storing sensitive credentials directly in configuration files. Refer to [Access File Authorizer Connections](/v1/docs/access-file-authorizer-connections){target=`_blank`} for a detailed use case. **Configuration**: ```yaml rclone: enabled: true image: repository: "docker.io/rclone/rclone" tag: "1.69" config: | [my-sftp] type = sftp host = YOUR_HOST user = YOUR_USERNAME pass = YOUR_SFTP_PASSWORD port = 22 # Extra arguments to pass to the rclone remote control server extraArgs: >- --retries=3 --low-level-retries=10 --retries-sleep=10s ``` 1. Set `enabled` to `true`. 2. Ensure the `image` section is configured to pull the `rclone` Docker image. 3. Configure the `config` section: - For SFTP integration, a predefined commented configuration is available: - Define a name for your SFTP in brackets. - Set the `type` to `sftp`. - Configure your SFTP server details: `host`, `user`, `pass`, and `port`. - The password you set in this configuration will be encrypted as part of the deployment to meet `rclone` requirements. - **Note:** The password encryption required for SFTP may limit other out-of-the-box (`rclone`) integrations. Consult with PlainID Professional Services for alternative integrations. - For any special configurations or `rclone` integrations beyond SFTP (such as AWS S3), consult with PlainID Professional Services. 4. Use `extraArgs` to define any additional supported `rclone` switches as needed. Once this section is added and `rclone` deployment is enabled, its container will be deployed, and the `rclone` HTTP server will run alongside the Authorizer, allowing HTTP requests via the webhook definition. ### SFTP Using the built-in integration with `rclone`, you can set up an SFTP server and transfer the Authorizer's generated job files to a remote SFTP server for further use by your application or system. ## Steps for Integration 1. As part of the `rclone` setup, define your SFTP server details as mentioned. 2. When defining the webhook template: - Set the webhook URL to the `rclone` endpoint: `../operations/copyfile`. - Set the webhook Method to `POST`. 3. Configure the webhook headers: - Use the Environment Variable `${RCLONE_BASIC_BASE64}` for `rclone` Authorization, which is predefined as part of the `rclone` deployment. - Set `Content-Type` to `application/json`. 4. Define the webhook payload with the `copyfile` operation JSON structure, including source and destination properties: - `"srcFs"`, `"srcRemote"`, `"dstFs"`, and `"dstRemote"`. - The payload definition can use Go Template syntax with `{{ .varName }}`. - The payload can include an optional `_config` object with `rclone` config flags. - For SFTP, the example below includes the `inplace` flag to guide the file copy operation. Complete example for a webhook template section: ```yaml webhooks: copyFileWebhook: method: "POST" headers: - name: Content-Type values: - "application/json" - name: Authorization values: - ${RCLONE_BASIC_BASE64} url: ${RCLONE_URL}/operations/copyfile payload: | { "srcFs": "{{ .SrcFs }}", "srcRemote": "{{ .SrcRemote }}", "dstFs": "{{ .DstFs }}", "dstRemote": "{{ .DstRemote }}", "_config": { "inplace": true } } timeout: 180 ``` ## Configuring Webhook Triggers for File Transfer 1. In the job configuration, add webhook triggers by referencing the defined webhook template and injecting variable values. 2. Trigger a webhook call for each file you want to transfer to the SFTP server. Set these parameters under `webhooks`: - **Template** – Reference the webhook template name. - **Variables** – Define entries for each variable specified at the template level: `"SrcFs"`, `"SrcRemote"`, `"DstFs"`, and `"DstRemote"`. - `SrcFs` is set to the local folder where the Authorizer saves the generated file: `/app/data`. - `DstFs` refers to your SFTP server using the name defined in the `rclone` configuration (e.g., `my-sftp`). - File names can be set using the custom function `fileName`. Complete webhook trigger section **example** for a job: ```yaml webhooks: - template: "copyFileWebhook" variables: - name: SrcFs value: /app/data/ - name: SrcRemote value: '{{ fileName "flow" "flow1" "c1" }}' - name: DstFs value: "my-sftp:/upload" - name: DstRemote value: '{{ fileName "flow" "flow1" "c1" }}' - template: "copyFileWebhook" variables: - name: SrcFs value: /app/data/ - name: SrcRemote value: '{{ fileName "job" "jb1" "final" }}' - name: DstFs value: "my-sftp:/upload" - name: DstRemote value: '{{ fileName "job" "jb1" "final" }}' ```