Examples of Monitor Creation

Practical examples for creating data quality monitors using the Decube Public API. These examples demonstrate proper payload structure and follow all validation rules.

This page provides comprehensive examples for creating monitors using the Decube Public API. All examples follow the validation rules and constraints documented in the Create, Manage & Delete guide.

Before using these examples, make sure you understand the Monitor Validation & Constraints to ensure your monitors are configured correctly.

Prerequisites

Authentication

All API requests require authentication via the X-Decube-Api-Key header:

# Set your API key as an environment variable
export DECUBE_API_KEY="your-api-key-here"

# Include in curl requests
curl -H "X-Decube-Api-Key: $DECUBE_API_KEY" ...

Base URL

Replace <REGION> with your Decube region (e.g., us, eu):

https://connect.<REGION>.decube.io/api/v1/data/monitors

Basic Examples

Example 1: Minimal Scheduled Volume Monitor

A basic scheduled monitor for dataset-level volume monitoring with hourly frequency.

curl -X POST \
  "https://connect.<REGION>.decube.io/api/v1/data/monitors" \
  -H "Content-Type: application/json" \
  -H "X-Decube-Api-Key: $DECUBE_API_KEY" \
  -d '{
    "asset": { "type": "dataset", "id": 123 },
    "test_type": "volume",
    "mode": "scheduled",
    "name": "hourly-volume-monitor",
    "description": "Hourly volume monitoring for dataset 123",
    "notify": true,
    "incident_level": "warning",
    "frequency": { "frequency": "1H" },
    "row_creation": {
      "filter_mode": "all_records"
    }
  }'

Expected Response:

{ "monitor_id": 1 }

Key Points:

row_creation is required for volume tests
Hourly frequency doesn't require timezone or time_of_day
all_records row creation mode processes the entire dataset

Example 2: Daily Freshness Monitor with Timezone

A scheduled freshness monitor running daily at a specific time with timestamp-based row creation.

curl -X POST \
  "https://connect.<REGION>.decube.io/api/v1/data/monitors" \
  -H "Content-Type: application/json" \
  -H "X-Decube-Api-Key: $DECUBE_API_KEY" \
  -d '{
    "asset": { "type": "dataset", "id": 456 },
    "test_type": "freshness",
    "mode": "scheduled",
    "name": "daily-freshness-monitor",
    "description": "Daily freshness monitor running at 3:00 UTC",
    "notify": true,
    "custom_alerts": {
      "email": ["[email protected]"],
      "slack": [],
      "teams_webhook": []
    },
    "incident_level": "critical",
    "frequency": {
      "frequency": "1D",
      "time_of_day": 3,
      "timezone": "UTC"
    },
    "row_creation": {
      "filter_mode": "timestamp",
      "metric_column_id": 789,
      "enable_smart_training": true
    }
  }'

Key Points:

Daily frequency requires time_of_day (0-23) and timezone
metric_column_id must reference an existing timestamp/datetime column
enable_smart_training enables automatic threshold learning

Example 3: On-Demand Field Health Monitor

An on-demand monitor for validating email format using regex matching.

curl -X POST \
  "https://connect.<REGION>.decube.io/api/v1/data/monitors" \
  -H "Content-Type: application/json" \
  -H "X-Decube-Api-Key: $DECUBE_API_KEY" \
  -d '{
    "asset": { "type": "property", "id": 987 },
    "test_type": "regex_match",
    "mode": "on_demand",
    "name": "email-format-validation",
    "description": "Validate email field format using regex",
    "notify": false,
    "incident_level": "info",
    "row_creation": {
      "filter_mode": "all_records"
    },
    "threshold": {
      "type": "value",
      "value": "^[\\w.-]+@[\\w.-]+\\.[a-zA-Z]{2,}$"
    },
    "lookback_period": 7
  }'

Key Points:

frequency is omitted for on-demand monitors
regex_match uses value threshold type with regex pattern
lookback_period is required for on-demand monitors with all_records row creation

Advanced Examples

Example 4: Custom SQL Monitor

A scheduled monitor using custom SQL that defines its own pass/fail logic.

curl -X POST \
  "https://connect.<REGION>.decube.io/api/v1/data/monitors" \
  -H "Content-Type: application/json" \
  -H "X-Decube-Api-Key: $DECUBE_API_KEY" \
  -d '{
    "asset": { "type": "source", "id": 1 },
    "test_type": "custom_sql",
    "mode": "scheduled",
    "name": "custom-sql-quality-check",
    "description": "Custom SQL monitor for data quality validation",
    "notify": true,
    "custom_alerts": {
      "email": ["[email protected]"],
      "slack": [],
      "teams_webhook": []
    },
    "incident_level": "warning",
    "frequency": {
      "frequency": "1D",
      "time_of_day": 2,
      "timezone": "UTC"
    },
    "custom_sql": "SELECT CASE WHEN COUNT(*) > 100 THEN 1 ELSE 0 END as has_issues FROM {{table}} WHERE status NOT IN ('"'"'ok'"'"','"'"'pending'"'"')"
  }'

Key Points:

Custom SQL monitors don't use row_creation or threshold
The query itself contains the validation logic and thresholds
When submitting the custom_sql payload, ensure that the SQL is valid. If the SQL is invalid, the API will return an error.

Common Errors and Solutions

Missing Required Fields

❌ Incorrect: Missing row_creation for volume test

{
  "asset": { "type": "dataset", "id": 123 },
  "test_type": "volume",
  "mode": "scheduled",
  "name": "invalid-monitor",
  "notify": true,
  "incident_level": "warning",
  "frequency": { "frequency": "1H" }
}

✅ Correct: Include required row_creation

{
  "asset": { "type": "dataset", "id": 123 },
  "test_type": "volume",
  "mode": "scheduled",
  "name": "valid-monitor",
  "notify": true,
  "incident_level": "warning",
  "frequency": { "frequency": "1H" },
  "row_creation": { "filter_mode": "all_records" }
}

Incorrect Frequency Format

❌ Incorrect: Daily frequency missing required fields

{
  "frequency": { "frequency": "1D" }
}

✅ Correct: Include time_of_day and timezone

{
  "frequency": {
    "frequency": "1D",
    "time_of_day": 9,
    "timezone": "UTC"
  }
}

Invalid Threshold Types

❌ Incorrect: Wrong threshold type for range-based test

{
  "test_type": "average",
  "threshold": { "type": "value", "value": "100" }
}

✅ Correct: Use range threshold for numeric tests

{
  "test_type": "average",
  "threshold": { "type": "range", "min": 50, "max": 150 }
}

Quick Reference

Required Fields for All Monitors

asset (with valid type and id)
test_type
mode ("scheduled" or "on_demand")
name (max 100 characters)
notify (boolean)
incident_level

Additional Required Fields by Mode

Scheduled: frequency
On-demand: lookback_period (for timestamp/sql_expression row creation)

Row Creation Requirement by Test Type

Required: All field health tests, volume, freshness
Not applicable: Custom SQL, schema drift, job failure

Frequency Requirements

Hourly: Just frequency field
Daily: frequency, time_of_day, timezone
Weekly: frequency, time_of_day, day_of_week, timezone
Monthly: frequency, time_of_day, day_of_month, timezone

For more detailed validation rules, see the Monitor Validation & Constraints section.

PreviousCreate, Manage & Delete NextACL

Last updated 23 days ago