Available Monitor Types

Comprehensive guide to all monitor types available in Decube's data quality platform.

Decube offers comprehensive monitoring capabilities to ensure data quality across your entire data infrastructure. Choose the right monitor type based on your specific data quality requirements.

Quick Selection Guide

Monitor Type	Best For	Setup Time	Technical Level
Schema Drift	Preventing pipeline breaks	Auto-enabled	Beginner
Freshness	Data dependencies	5 minutes	Beginner
Volume	Data load validation	5 minutes	Beginner
Field Health	Column-level data quality	10 minutes	Intermediate
Custom SQL	Complex business rules	15 minutes	Advanced
Job Failure	ETL pipeline monitoring	Auto-enabled	Beginner

---

Table-Level Monitors

Schema Drift ⚡ Auto-Enabled

Purpose: Automatically detects structural changes to your tables and columns Value: Prevents downstream application failures and pipeline breaks

This monitor is enabled automatically for all tables when you connect a data source. It detects:

• Table or column additions/deletions

• Data type changes

• Schema modifications that may cause compatibility issues

Use Cases:

• 🚨 Alert when critical table structures change

• 🛡️ Protect downstream applications from schema breaks

• 📊 Track data warehouse evolution over time

Modify Schema Drift Monitors

Freshness 🕒

Purpose: Tracks when tables were last updated to ensure data timeliness Value: Ensures dashboards and reports have current data

Our ML-powered freshness monitors learn your table update patterns and alert when data becomes stale based on historical patterns.

Use Cases:

• ⏰ Critical for real-time dashboards

• 📈 Essential for daily business reports

• 🔄 Monitor ETL pipeline completion

Set Up Freshness & Volume Monitors

Volume 📊

Purpose: Monitors row count changes and detects data load anomalies Value: Catches missing or incomplete data loads before they impact business

Volume monitors establish baselines for expected row counts and alert when insertions fall below or exceed normal ranges.

Use Cases:

• 📉 Detect missing data loads

• 📈 Identify unexpected data spikes

• 🔍 Monitor ETL pipeline data volumes

Set Up Freshness & Volume Monitors

---

Column-Level Monitors

Field Health 🔍

Purpose: Validates data quality at the column level with comprehensive tests Value: Ensures clean, consistent data for analytics and ML models

Available tests depend on the data type of the field selected. For example, min/max tests are only supported on numeric columns.

Available Field Health Tests:

Data Completeness Tests

Null Checks

• Null: Monitors null values with configurable thresholds

  • Absolute mode: Set threshold by row count (e.g., "fail when > 100 null rows")

  • Percentage mode: Set threshold by percentage (e.g., "fail when > 5% nulls")

  • Auto mode: Let ML learn acceptable null patterns (scheduled only)

• Not Null: Validates no null values exist (fails if any nulls found)

Use Case: Essential for mandatory fields and columns with dependencies

Uniqueness Validation

• Unique: Monitors duplicate values with configurable thresholds

  • Absolute mode: Set threshold by row count (e.g., "fail when > 10 duplicate rows")

  • Percentage mode: Set threshold by percentage (e.g., "fail when < 95% unique")

  • Auto mode: Let ML learn acceptable uniqueness patterns (scheduled only)

• Is Unique: Ensures all values are distinct (fails if any duplicates found)

Use Case: Critical for primary keys and unique constraints

Data Range & Distribution Tests

Statistical Validation

• Average: Monitors average values against expected ranges

• Min/Max: Validates minimum and maximum value boundaries

• Cardinality: Tracks distinct value counts (high/medium/low classification)

Use Case: Detect outliers and validate business rule compliance

String Validation

• String length: Validates minimum/maximum string lengths using range thresholds

• Email: Monitors invalid email addresses with configurable thresholds

  • Absolute mode: Set threshold by row count (e.g., "fail when > 50 invalid emails")

  • Percentage mode: Set threshold by percentage (e.g., "fail when > 2% invalid")

  • Auto mode: Let ML learn acceptable patterns (scheduled only)

• UUID: Monitors invalid UUIDs with configurable thresholds (supports Absolute, Percentage, Auto modes)

• Regex Match: Custom pattern matching validation with configurable thresholds

  • Pattern is specified separately from threshold

  • Threshold controls how many non-matching values trigger an incident

  • Supports Absolute, Percentage, and Auto modes

Microsoft SQL Server Note: For SQL Server and Azure SQL data sources, regex patterns use wildcard syntax instead of standard regex. See Microsoft Wildcard Characters for details.

Set Up Field Health Monitors

---

Advanced Monitoring

Custom SQL 🔧

Purpose: Create custom validation logic for complex business rules Value: Monitor sophisticated data relationships and business-specific requirements

Write custom SQL queries to validate complex business logic. Incidents trigger when your query returns any rows (row_count > 0).

Common Use Cases:

• 💰 Revenue reconciliation between systems

• 🔗 Cross-table data consistency checks

• 📋 Complex business rule validation

• ⚖️ Data quality SLA monitoring

Example Applications:

-- Detect revenue discrepancies
SELECT * FROM sales
WHERE total_amount != (price * quantity * tax_rate)

-- Monitor data freshness SLAs
SELECT * FROM critical_tables
WHERE last_updated < NOW() - INTERVAL '2 hours'

Set Up Custom SQL Monitors

Job Failure Monitoring ⚙️ Auto-Enabled

Purpose: Automatically monitors ETL pipeline and data job execution Value: Ensures data transformation processes complete successfully

Job Failure monitors are automatically created when you connect ETL-type sources. They track failed jobs to identify processing issues.

Key Features:

• 🔄 Auto-configured for dbt, Airflow, and other ETL tools

• 📧 Customizable alert channels

• 📊 Historical failure tracking

• ⚡ Real-time failure detection

Modify Job Failure Monitors (Data Job)

---

Enhanced Monitoring Features

Grouped-By Monitoring 🎯

Purpose: Segment monitoring by dimension values for granular insights Value: Monitor data quality at the business logic level

Track data quality metrics segmented by specific column values (e.g., by region, customer type, or product category).

Applications:

• 📍 Monitor data quality by geographic region

• 👥 Track customer data completeness by segment

• 📦 Validate product data by category

• 🏢 Ensure compliance by business unit

Grouped-by Monitors

---

Getting Started Recommendations

For Data Engineers 👨‍💻

1. Start with Freshness & Volume for critical tables

2. Add Custom SQL for complex validation logic

3. Use Field Health for data pipeline validation

For Governance Teams 🛡️

1. Begin with Schema Drift monitoring (auto-enabled)

2. Implement Field Health for compliance requirements

3. Set up Grouped-By monitoring for business segments

For Platform Admins 👤

1. Configure Job Failure monitoring for all ETL processes

2. Set up Volume monitoring for data load validation

3. Establish Freshness SLAs for business-critical data

---

Next Steps: Ready to get started? Visit our Enable Asset Monitoring guide to begin setting up your first monitors.