Airflow

Adding Airflow to your decube connections helps your team to discover, document and monitor the quality of your pipelines.

Supported Capabilities

Capability	Supported
Metadata Extraction	✅
Metadata Types Collected	Data Job, Data Run, Data Task
Data Profiling	❌
Data Preview	❌
Data Quality	❌
Configurable Collection	❌
External Table	❌
View Table	❌
Stored Procedure	❌

Data Quality Support

Capability	Supported
Freshness	❌
Volume	❌
Field Health	❌
Custom SQL	❌
Schema Drift	❌
Job Failure	✅

Connection Requirements

• Airflow Username

• Airflow User Password

• Airflow API host address

• Airflow API enabled and set to Basic Auth. See Airflow Documentation for this.

• Airflow Version 2.3.0 and above (Version >= 2.0.0 and < 2.3.0 may not work fully)

Airflow

Creating an Airflow User for Decube

1. Go to Security > List Users

1. Click on "+" to Add User

1. Insert information for new decube user

   1. Username - suggested value: decube

   2. Email - [email protected]

   3. Role - Minimum required Op (which is a default role from Airflow)

   4. Password - Use a strong password

Airflow API is not Publicly Accessible

• For decube to monitor your Airflow service, we will require that the Airflow API be publicly accessible or privately accessible to a SSH bastion host. Instruction on setting up a bastion host can be found here SSH Tunneling