REST Services Source

REST Services Source Introduction

Policy Information Point (PIP) Settings allow you to configure Data Sources that provide essential information for Policy decisions. In many cases, the Data Source you need to integrate with provides the data through a REST service that exposes different endpoints serving subsets of the data. In some cases, these services are your internal application’s self-developed services, where you have more control over the integration pattern and the available endpoints. In other cases, there are third-party REST services solutions you are using in your organization that manage some of your data.
This guide focuses on setting up REST Service Data Sources, enabling you to integrate external REST APIs into your Policy Authorization process.

Configuring the REST Service Data Source

Prerequisites

Before configuring a REST Service Data Source, ensure that you have:

• Appropriate permissions in your tenant
• Basic understanding of REST APIs and JSON
• Knowledge of SQL syntax for defining data structures

General

The configuration of a REST data source includes the settings required to make a valid REST request to your service. It also includes a guide on managing the response from your service and transforming it from JSON to a relational data structure.
The overall setup includes:

• A declaration of a virtual table
• A definition of Data Attributes fetched from the REST service response
• Optional OPTIONS clause for each Data Attribute
• The Base URL of the service
• Authentication to the service
• The service endpoint’s URI
• A REST Method
• The location of the relevant data in the service response
• OPTIONS clause with additional configurations influencing the REST request and response
• Additional translator properties configuring the translator behavior

To Configure a Data Source:

1. Open the Tenant or Environment Settings screen and select the PIP Settings tile.
2. In the Policy Authorization Group drop-down menu, select the relevant PAA. A list of Data Sources and Views are displayed.
3. In the Data Source List, click New Data Source.
4. In the Data Source Details section:
   • Enter a Display Name for the new Data Source.
   • In the Connection Adapter dropdown, select REST.
   • In the Connection Settings section under Policy Authorization Agent, select a PAA.
   • Input your REST service's base URL in the designated End Point field. For example: https://domain.com/data.
   • You also have the option to specify a mock server's base URL, which can assist in designing and testing an integration when developing a new use case.
   • In the Models section under the Model Name field, input a unique Model name.
   • Under the DDL section, write a valid DDL to fetch REST attributes according to the service response. Refer to the section on Configuring a DDL for more information.

Note: If using OAuth 2 as a Security Type in the Connection Settings, select None from the dropdown, and see the full OAuth 2 instructions on setting up this authentication method.

Configuring a DDL for the REST Service

The structure of a DDL includes the declaration of the table, declaration of all attributes, and an optional OPTION property. The DDL defines the data structure that the REST Service creates. It details Attributes and their options, and allows users to configure the REST request options. This configuration determines how a request is generated and executed in addition to how to handle and parse the response.

When inputting a DDL property, ensure that you specify the name of the attribute and the type. You can add a nameinsource option in the OPTIONS clause if indicating a specific location in the response structure using a JSONPath. You can configure the path by utilizing the relative path to the general Data Source path when you have a general path set and you want to specify the property relative to this base path. This is helpful when the base path can change, but the relative path to the property remains consistent.

E.g. If the general base path is $.data.user, then simply specify the name to get the user’s name.

Example Usage

{
  "data": {
    "user": {
      "id": 123,
      "name": "John Doe"
    }
  }
}

• Relative Path (with general path $.data.user): name

DDL Structure

The basic structure of a DDL for the REST Service is as follows:

OPTIONS Clause Structure

See available options here.

OPTIONS (nameinsource "<option>"<'option value'>, "<option>"<'option value'>)

Overall Structure

CREATE FOREIGN TABLE <Virtual_Table_Name>(
    <Attribute_Name> <Data_Type>,
    <Attribute_Name> <Data_Type> OPTIONS (nameinsource '<relative JsonPath to the location of the field in the response>')
) OPTIONS (nameinsource '<JsonPath to the root location in the response>', "plainid:method" 'GET', "plainid:URI" '<ENDPOINT_URI_AFTER_BASE_URL>', "plainid:unwrap" '<True/False>');

Available OPTIONS in the DDL

Below is a list of OPTIONS that can be used in the DDL. Refer to the table below for examples.

• plainid:URI: The URI of the service endpoint that is used for the REST request. This parameter is added to the base URI defined in the data source. In addition, users can add a variable to this relative URI, which is used as a dynamic parameter based on values from the WHERE clause of a query sent to the REST Source:
  Example:
  /users/all
  It is possible to add a variable to the URI. In this case, the query will be translated based on the WHERE clause.
  Example:
  /users/$(id)
Select * from groups where id = 101 will be translated to /users/101.

• plainid:method is used for the REST request and supports GET and POST (with parameters/payload) methods.

• plainid:bodyJsonContent property includes the JSON content used as the POST request body. Dynamic parameters can be employed within this payload, which dynamically adjust the JSON content for POST requests. In the DDL, JSON should be converted into a string format. When utilizing this property, a standard Content-Type header is applied to the request sent to the REST service with the value application/json.

• nameinsource property is used both at the Attribute and general level:

  • At the Attribute level, this property is optional and should be used when a different Attribute name than the name in the response is needed, or when a specific JSONPath for that Attribute in the response needs to be configured.
  • This property is required and must contain a JSONPath that specifies the location from which to extract data from the response.
  • The value ($) specifies the root location’s JSONPath in the response where all Attributes are retrieved from. Other variations can include $.results[*], groups[*], or any other valid JSONPath locations that guide where to parse the response from and fetch the relevant data retrieved from the service. Note that this JSONPath is the base for each Attribute’s JSONPath.

• plainid:unwrap property supports unwrapping complex REST responses while ensuring that the relationship between attributes remains intact.

The JSON returned needs to be converted into a relational database format for complex Views. The unwrap feature allows for JSON multi-values to "unwrap" into an array, creating multiple records from each value in the array.

If working with complex nested structures and/or large data sets and arrays that may create a very large set of permutations, expect that the unwrap calculation will take a longer time and may affect performance. Ensure that you reduce complex structures, sets, or arrays where possible and only map required Attributes to reduce permutations. Contact the PlainID Global Services team for more details.

Property Descriptions and Examples

Translator Properties

Translator properties direct the REST translator's behavior and functionalities. Refer to the supported translator properties for detailed explanations and examples:

Support Dynamic Filters

When using a dynamic parameter in the form of an $(Attribute), in the URI, in a dynamic header, or in a JSON option for POST payload, the supportsfilter property must be added with the value true. If not defined or set to false, the $(Attribute) is treated as a text and is not replaced dynamically with a value.

• true: Filters are sent to the REST Data Source according to the dynamic parameters in the REST definition. When the data is considered as authorized, it is sent back to the requesting service (i.e. the PDP).
  The advantage of the true setting is that users can leverage the Data Source filtering capabilities by applying dynamic parameters and reducing the overall performance and payload with a smaller and filtered dataset.
  The drawbacks of this setting is that filtering is limited to the REST endpoint functionality, so only supported filters are applied on the query. The REST Data Source needs to support all required filters to avoid fetching unauthorized data. Refer to the Partial Filtering section for more information.

• false: Filters are not transmitted to the REST Data Source. Instead, the data is fetched according to the REST definition without any dynamic filters. After the data is retrieved, the PIP settings apply filters to the dataset based on the incoming SQL query, and only then is the filtered data sent back to the requesting service (i.e. the PDP).
  The advantage of the false setting is that filters can vary and can be updated dynamically as the filtering takes place at the SQL level in the PIP service.
  The drawbacks of the false setting is that it fetches larger datasets (potentially all datasets) from the REST Data Source to the PIP service, which can cause big payloads and performance impact.

Note: If this property is not sent, the default is false.

Multi-value Parameters

When a multi-value Attribute passes in the query, all the values used in the dynamic parameters are injected into the REST request.

Examples:

POST {
  "paramA": [
    "a1",
    "a2",
    "a3"
  ]
}

GET ../../..?paramA=a1,a2,a3

To partially filter the REST Data, refer to the section on Partial Filtering. You can also check out the Data Source Filtering Parameters section for more information.

Headers

In cases when your REST services require headers, whether fixed or custom headers like content-type, x-request, tokens, etc., you can use the headers property to inject custom headers into the REST request that will be sent. The value is a JSON containing a list of headers, each containing the header name and an array of values in this form:

{"x-header1": ["value1", "value2"] ,"x-header2": ["value1", "value2"]}

Note that if you need to use X-Auth-Token to authenticate your REST endpoint, you can add the property headers with the value {"X-Auth-Token":["<token>"]}.

Dynamic Headers

When configuring headers in the translator property, users can optionally adjust dynamic header values denoted by $(<attribute_name>). For example, the structure of a header setup could be similar to the existing one: {"x-header-accountID":["$(accID)"]}. In this setup, $(accID) would be replaced with the value sent by runtime during the preparation of the REST request.

It is important to note that if the value of the dynamic parameter $(<attribute_name>) is null or empty (not sent in the runtime request), the corresponding header should be skipped instead of being sent empty.

Note: Make sure the supportsfilter is set to true when setting up dynamic headers.

Partial Filtering

The Partial Filtering function in the REST translator enhances data filtering from a REST data source. It enables filtering even when the REST service does not support the dynamic filters required for data integration and policy evaluation. This function leverages the supportsFilter property to push supported filters down to the REST service, while additional filtering is simultaneously applied to the retrieved subset of data. This approach balances performance with flexibility, accommodating a wide range of use cases.

To set up partial filtering on your Attributes:

1. Set the supportsFilter and supportsPartialFiltering properties to true. Both parameters are false by default.
2. Configure the Dynamic Parameters supported by the REST service so they can be passed down in the service call.
3. Add the option "teiid_rel:partial_filter" 'true' for each Attribute you want to filter further after receiving the response from the REST Data Source.

Partial Filter and Full Configuration Examples

Partial Filter Configuration Example:

attributeA string OPTIONS ("teiid_rel:partial_filter" 'true'),
attributeB string OPTIONS ("teiid_rel:partial_filter" 'true'),
attributeC string

In this example:

• AttributeA and AttributeB are set up for partial filtering, meaning they can be further filtered.
• AttributeC does not have partial filtering enabled. Depending on the data, filtering with or for AttributeC may result in either a full payload response or no response at all.

Full DDL examples with Partial Filtering:
Example 1:

CREATE FOREIGN TABLE REST_PIP_Table( 
attributeA string OPTIONS ("teiid_rel:partial_filter" 'true'),
attributeB string OPTIONS ("teiid_rel:partial_filter" 'true'),
attributeC string
) OPTIONS (nameinsource '$.data[*]', "plainid:unwrap" 'true', "plainid:method" 'GET', "plainid:URI" '/byAttributeA/$(AttributeA)')

Example 2: x

CREATE FOREIGN TABLE accountData_byUser(
    userId string OPTIONS (nameinsource 'user', "teiid_rel:partial_filter" 'true'),
    accountId string OPTIONS (nameinsource 'accountsData[*].accountName', "teiid_rel:partial_filter" 'true'),
    balance string OPTIONS (nameinsource 'accountsData[*].accountBalance', "teiid_rel:partial_filter" 'true')
) OPTIONS (nameinsource '$', "plainid:method" 'GET', "plainid:unwrap" 'true', "plainid:URI" '/accountsData/$(userId)');

Data Source Filtering Parameters

Set the supportsPartialFiltering property to true to use the optional property added to attributes.

REST Error Handling

Due to request misconfigurations, temporary network issues, or service issues, REST services might return errors in some cases. If an error occurs, ensure that the data source was created properly. You can also use the following translator properties to configure how errors are handled in the response: