OpenLineage (BETA)

This document provides a step-by-step guide to connecting with the OpenLineage connector and viewing lineage data from jobs using the OpenLineage framework.

Supported Capabilities

Capability
Supported

Metadata Extraction

Metadata Types Collected

Schema, Virtual Table, Virtual Column, Data Job, Data Task, Data Run

Data Profiling

Data Preview

Data Quality

Configurable Collection

External Table

View Table

Stored Procedure

Data Quality Support

Capability
Supported

Freshness

Volume

Field Health

Custom SQL

Schema Drift

Job Failure

Connection Requirements

  • Step 1: Go to My Account and click on the Data Sources tab

Data Sources
  • Step 2: Scroll down to the Connect a new data source section

  • Step 3: Click on the OpenLineage icon

OpenLineage
  • Step 4: Enter a name for the data source and click Submit

OpenLineage
  • Step 5: A Webhook UUID and an API Key will be provided. Copy them into your connector’s configuration settings.

Provided Webhook UUID and API Key

Webhook Endpoint

Payload must submitted to the following endpoint:

https://integrations.<region>.decube.io/integrations/openlineage/webhook/<webhook-uuid>

Submitting Payload to OpenLineage Webhook

If you're using these tools, please follow the respective documentation in the OpenLineage website.

Custom Integration

If you want to create your own integration for your tools, follow these steps:

  • Submit the webhook payload to the above endpoint.

  • Use the Bearer token system for authentication.

Example request:

curl -X POST \
   -H "Authorization: Bearer <api-key>" \
   -H "Content-Type: application/json" \
   --data '{}' \
   https://integrations.<region>.decube.io/integrations/openlineage/webhook/<webhook-uuid>

Last updated