Airflow
Adding Airflow to your decube connections helps your team to discover, document and monitor the quality of your pipelines.
Supported Capabilities
Metadata Extraction
✅
Metadata Types Collected
Data Job, Data Run, Data Task
Data Profiling
❌
Data Preview
❌
Data Quality
❌
Configurable Collection
❌
External Table
❌
View Table
❌
Stored Procedure
❌
Data Quality Support
Freshness
❌
Volume
❌
Field Health
❌
Custom SQL
❌
Schema Drift
❌
Job Failure
✅
Connection Requirements
Airflow Username
Airflow User Password
Airflow API host address
Airflow API enabled and set to Basic Auth. See Airflow Documentation for this.
Airflow Version 2.3.0 and above (Version >= 2.0.0 and < 2.3.0 may not work fully)
Creating an Airflow User for Decube
Go to Security > List Users
Click on
"+"to Add User
Insert information for new decube user
Username - suggested value:
decubeEmail - [email protected]
Role - Minimum required
Op(which is a default role from Airflow)Password - Use a strong password
Airflow API is not Publicly Accessible
For decube to monitor your Airflow service, we will require that the Airflow API be publicly accessible or privately accessible to a SSH bastion host. Instruction on setting up a bastion host can be found here SSH Tunneling
Last updated