Airflow

Adding Airflow to your decube connections helps your team to discover, document and monitor the quality of your pipelines.

Supported Capabilities

Metadata Types Collected:

Data Job, Data Run, Data Task

Data Quality

Capability
Supported

Freshness

Volume

Schema Drift

Field Health

Custom SQL

Job Failure

Catalog

Capability
Supported

Data Profiling

Data Preview

Minimum Requirement

  • Airflow Username

  • Airflow User Password

  • Airflow API host address

  • Airflow API enabled and set to Basic Auth. See Airflow Documentation for this.

  • Airflow Version 2.3.0 and above (Version >= 2.0.0 and < 2.3.0 may not work fully)

Airflow

Creating an Airflow User for Decube

  1. Go to Security > List Users

  1. Click on "+" to Add User

  1. Insert information for new decube user

    1. Username - suggested value: decube

    2. Role - Minimum required Op (which is a default role from Airflow)

    3. Password - Use a strong password

Airflow API is not Publicly Accessible

  • For decube to monitor your Airflow service, we will require that the Airflow API be publicly accessible or privately accessible to a SSH bastion host. Instruction on setting up a bastion host can be found here SSH Tunneling

Last updated