Connectors supporting the exchange and maintenance of metadata
These connectors help to accelerate the rollout of your open metadata ecosystem since they can be used to automate the extraction and distribution of metadata to the third party technologies.
The integration connectors support the exchange of metadata with third party technologies. This exchange may be inbound, outbound, synchronous, polling or event-driven.
Figure 1: Integration Connectors
Details of how integration connectors work is described in this article.
The files integration connectors run in the Files Integrator Open Metadata Integration Service (OMIS) hosted in the Integration Daemon.
The Data Files Monitor Integration Connector maintains a DataFile asset for each file in the directory (or any subdirectory). When a new file is created, a new DataFile asset is created. If a file is modified, the lastModified property of the corresponding DataFile asset is updated. When a file is deleted, its corresponding DataFile asset is also deleted (or archived if it is still needed for lineage).
The DataFolderMonitorIntegrationConnector maintains a DataFolder asset for the directory. The files and directories underneath it are assumed to be elements/records in the DataFolder asset and so each time there is a change to the files and directories under the monitored directory, it results in an update to the lastModified property of the corresponding DataFolder asset.
The database integration connectors run in the Database Integrator Open Metadata Integration Service (OMIS) hosted in the Integration Daemon.
- The PostgreSQL database integration connector automatically maintains the open metadata instances for the databases hosted on a PostgreSQL server. This includes the database schemas, tables, columns, primary keys and foreign keys.
Security Enforcement Engines
The security integration connectors run in the Security Integrator Open Metadata Integration Service (OMIS) hosted in the Integration Daemon.
Repository and Event Mapper Connectors
The repository connectors provide the ability to integrate a third party metadata repository into an open metadata repository cohort.
Figure 2 shows the repository connector providing a native open metadata repository that uses a particular type of store within an Egeria Metadata Server.
Figure 2: Repository connector supporting a native open metadata repository
The In-memory OMRS Repository Connector provides a simple native repository implementation that “stores” metadata in hash maps within the JVM. It is used for testing, or for environments where metadata maintained in other repositories needs to be cached locally for performance/scalability reasons.
The Read-only OMRS Repository Connector provides a native repository implementation that does not support the interfaces for create, update, delete. However it does support the search interfaces and is able to cache metadata. This means it can be loaded with open metadata archives to provide standard metadata definitions.
Figure 3 shows the repository connector providing a native open metadata repository that uses a particular type of store within an Egeria Metadata Server.
Figure 3: Repository connector and optional event mapper supporting an adapter to a third party metadata catalog
The Apache Atlas OMRS Repository Connectors implements read-only connectivity to the Apache Atlas metadata repository.
The IBM Information Governance Catalog (IGC) OMRS Repository Connector implements read-only connectivity to the metadata repository within the IBM InfoSphere Information Server suite.
The definition of the connector interfaces for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore Java package.
Open Discovery Services
Open Discovery Services are connectors that analyze the content of resources in the digital landscape and create annotations that are attached to the resource’s Asset metadata element in the open metadata repositories in the form of an open discovery report.
Figure 4: Discovery Services
The interfaces used by a discovery service are defined in the Open Discovery Framework (ODF) along with a guide on how to write a discovery service.
CSV Discovery Service extracts the column names from the first line of the file, counts up the number of records in the file and extracts its last modified time.
Governance Action Services
Governance Action Services are connectors that perform monitoring of metadata changes, validation of metadata, triage of issues, assessment and/or remediation activities on request.
Figure 5: Governance Action Services
They run in the Governance Action Open Metadata Engine Service (OMES) hosted by the Engine Host OMAG Server.
Generic Element Watchdog Governance Action Service listens for changing metadata elements and initiates governance action processes when certain events occur.
Generic Folder Watchdog Governance Action Service listens for changing assets linked to a DataFolder element. This may be for DataFiles directly linked to the folder or in sub-folders. It initiates governance action processes when specific events occur.
Move/Copy File Provisioning Governance Action Service moves or copied files from one location to another and maintains the lineage of the action.
Origin Seeker Remediation Governance Action Service walks backwards through the lineage mappings to discover the origin of the data.
- Link to the OMAG Server Platform for information on Egeria’s principle runtime.
- Link to the Open Metadata Repository Services for information on how the different store connectors are used.
- Link to the Administration Guide for information on how to configure connectors into Egeria’s runtime.
License: CC BY 4.0, Copyright Contributors to the Egeria project.