OpenLineage (BETA)

This document provides a step-by-step guide to connecting with the OpenLineage connector and viewing lineage data from jobs using the OpenLineage framework.

Supported Capabilities

Supported Capabilities

General

• Metadata — metadata extraction and display of asset information (tables, columns, schemas). Types collected: Schema, Virtual Table, Virtual Column, Data Job, Data Task, Data Run

Data Quality Monitors

• Job Failure

Not Supported

Connection Requirements

• Step 1: Go to My Account and click on the Integrations tab

• Step 2: Go to the Connect a new data source section

• Step 3: Click on the OpenLineage icon.

• Step 4: Enter a name for the data source and click Submit.

The exclusion filters can be added to exclude specific tables and lineage paths from being ingested. See more here.

Step 5: A Webhook UUID and an API Key will be provided. Copy them into your connector’s configuration settings.

Webhook Endpoint

Payload must submitted to the following endpoint:

https://integrations.<region>.decube.io/integrations/openlineage/webhook/<webhook-uuid>

Submitting Payload to OpenLineage Webhook

If you're using these tools, please follow the respective documentation in the OpenLineage website.

Tool	Documentation
Airflow	openlineage.io/docs/integrations/airflow/usage
Apache Spark	openlineage.io/docs/integrations/spark/configuration/usage
Apache Flink	openlineage.io/docs/integrations/flink/configuration

Custom Integration

If you want to create your own integration for your tools, follow these steps:

• Submit the webhook payload to the above endpoint.

• Use the Bearer token system for authentication.

Example request:

curl -X POST \
   -H "Authorization: Bearer <api-key>" \
   -H "Content-Type: application/json" \
   --data '{}' \
   https://integrations.<region>.decube.io/integrations/openlineage/webhook/<webhook-uuid>

Exclusion Filters

Exclusion filters let you exclude specific tables and lineage paths from being ingested by Decube. This is useful when your OpenLineage jobs produce metadata for staging tables, temporary paths, or other assets you do not want tracked in your catalog.

You configure exclusion filters directly in the Decube UI on your OpenLineage data source settings page. Each filter expects a Python RegEx-compliant regular expression for its fields.

Supported Filter Types

ADLS Gen2 Path

Matches tables with an ADLS Gen2 URI in the format:

abfss://<container-name>@<service-name>.dfs.core.windows.net/<path>

Field	Description
container-name	The ADLS container name
service-name	The ADLS storage account name
path	The file path within the container

Example — exclude everything under the discard/ path in the decube container across all storage accounts:

Field	Value
container-name	decube
service-name	.*
path	discard/.*

This excludes abfss://[e-mail]/discard/test/file but not abfss://[e-mail]/nodiscard/test/file.

Snowflake

S3

Generic Regex

You can add multiple exclusion filters of different types on the same data source. Each filter is evaluated independently — a table is excluded if it matches any filter.