Data Sources

The PIP Service supports many different adapter types for many types of external information sources including such common Data Sources as Oracle, MySQL and many others. For each of the Policy Authorization Agents (PAA) that you have defined, you can configure one or more Data Sources.

To access the Data Source List:

Data Source Connection Adapters are divided into the following groups to help you find the Adapter you need  more easily. The Connection Adapter groups include:

• Database (for example: Apache, HyperSQL, JDBC, Microsoft Access, MySQL, PostgreSQL, etc.)
• Web Application (Amazon, Google Sheets, Salesforce
• File (Excel, .csv format)
• Webservices (for example: OData, OpenAPI, SAP Gateway, Swagger, etc.)
• User Repositories (for example: LDAP)

To create a new Data Source: 

1. Open the Tenant or Environment Settings screen and select PIP Settings.
2. Select an existing PAA from the list of available PAAs. The Data Source List and Views List for that PAA is displayed.
3. Click New Data Source.
4. In the Data Source Details section:
   • Enter the Display Name for the new Data Source.
   • In the Connection Adapter field, select the appropriate Connection Adapter. Note that the Connection Adapters are organized in groups, as detailed above (and in alphabetical order within each group).
5. In the Connection Settings section, complete the settings that define the Connections Settings to the external information source. These fields vary depending on which Connection Adapter you chose. 
6. In the Models section, set a model name and complete other parameters for fetching relevant data objects from the external information source. Models offer a representation of the data structure of the foreign schema built based on the original Source. It could be identical to the structure of the original Data Source, same attributes as the original Source has, or a subset of the original Data Source. 
   • Note that the Model name must be unique to the model and cannot contain special characters.
   • In the Data Model Properties fields, there are possible key value pairs of configurations that determine the way metadata is fetched from the Data source. In some Data Source types, such as JDBC, it will determine how the DDL statement (see below) will be generated and which data objects from the external information source will be fetched into the model.
     • An example for setting a property that will set only TABLES (and not VIEWS) from a RDBMS external information source:

Note:  This functionality is not supported for all Adaptors.

After all settings are defined, save the new Data Source with the connected Models, and click Sync Data Sources. This will sync all Data Sources and all models by fetching the data structure from the external information sources (it does not fetch any data from the external sources at this point).

Generating the DDL Statement

• DDL, or Data Definition Language, is used to describe the Data Source table structure (columns and types of columns). The DDL field on the Data Source screen provides another option to set what data will be fetched from the external Data Source. It allows the user to write a DDL statement with a CREATE FOREIGN TABLE command followed by a declaration of the Data Attributes for that model and specific options according to the source type.
  • An example for such DDL (for a Microsoft Active Directory model) might declare a foreign table “LDAP_TEST” with 4 data Attributes from the Active Directory, and add to it options part with BaseDN for the data fetching and an option to fetch data from all the tree:

• Generate DDL– this is used to generate a DDL statement according to the Data Source details and import properties. The Generate DDL functionality reads schema metadata from the Data Source and creates DDL statements based on the imported schema structure. 
  • As an example, setting a Postgres Data Source and adding import properties of table name and schema name, will generate a DDL.
  • Another example would be to set import properties only with schema name and the generated DDL will include creation of multiple foreign tables found in that schema.
  • You can add multiple Models for the same Data Source, each will have its unique name and will use different settings. By that you can, as an example, set an Oracle Data Source and connect to it one model that will fetch tables with specific prefix in its name, and another model that will fetch a specific view from the oracle schema.
• Translator Properties – a list of possible key value pairs of configurations that can determine how and what data objects from the external information source will be fetched.

After all settings are defined, save the new Data Source with the connected Models, and click Sync Data Sources. This will sync all Data Sources and all models by fetching the data structure from the external information sources (it does not fetch any data from the external sources at this point).

Supported Data Sources

The Platform supports a large number of Data Sources. The full list is available in the Platform when you begin the process of defining a new Data Source.

Among the Data Sources you may wish to define are:

• JDBC Data Sources
• LDAP Data Sources
• Azure AD
• OAuth2 Authentication Support

For more information on Data Sources, see the Admin Portal Data Sources section.