> For the complete documentation index, see [llms.txt](https://docs.decube.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.decube.io/public-api/overview/index/data-quality-v2/examples.md).

# Examples

End-to-end recipes for the Data Quality V2 API: pipeline gates, monitors as code, case syncing, and maintenance windows

These recipes combine the Data Quality V2 endpoints into complete workflows. Each one is written in Python with `requests` and uses the same setup, so you can copy the pieces you need into your own scripts, Airflow tasks, or CI jobs.

## Setup

Every recipe reads your API key and base URL from environment variables and shares two helpers: one for requests, and one for paging through search results.

```bash
export DECUBE_API_KEY="your-api-key"
export DECUBE_BASE="https://connect.<REGION>.decube.io/api/v1/data/data_quality"
```

```python
import os, time, requests

BASE = os.environ["DECUBE_BASE"]
HEADERS = {"X-Decube-Api-Key": os.environ["DECUBE_API_KEY"]}

def call(method, path, **kwargs):
    response = requests.request(method, f"{BASE}{path}", headers=HEADERS, **kwargs)
    if response.status_code == 400:
        raise RuntimeError(f"{method} {path} failed: {response.text}")
    response.raise_for_status()
    return response.json() if response.content else None

def search_all(path, body):
    """Page through a search endpoint and return every item."""
    items, page = [], 1
    while True:
        result = call("POST", path, json={**body, "page_size": 100, "page_number": page})
        items += result["items"]
        if len(result["items"]) < 100:
            return items
        page += 1
```

## Gate a pipeline on a data quality check

Run a check as soon as your pipeline loads a table, and stop the pipeline if the check fails. This keeps bad data from reaching the models and dashboards downstream.

The monitor must use a `manual` schedule — see [Trigger a run](https://docs.decube.io/public-api/overview/index/data-quality-v2/monitor-runs.md#trigger-a-run) for the requirements.

```python
import sys

def run_check(monitor_id, interval=20, limit=1800):
    run_id = call("POST", "/monitors/trigger", json={"monitor_id": monitor_id})["run_id"]

    deadline = time.time() + limit
    while time.time() < deadline:
        result = call("GET", "/monitors/runs/details", params={"run_id": run_id})["result"]
        if result["state"] == "errored":
            raise RuntimeError(f"Run {run_id} errored: {result['reason']}")
        if result["state"] == "completed":
            return result["evaluation"]["outcome"]
        time.sleep(interval)
    raise TimeoutError(f"Run {run_id} did not finish in {limit} seconds")

outcome = run_check(monitor_id=456)
if outcome == "failed":
    sys.exit("Data quality check failed. See the case in Decube for evidence.")
```

To use this in Airflow, call `run_check` from a `PythonOperator` placed after the load task. The task fails when the check fails, and the downstream tasks don't run. In dbt or any other CI step, run the script after the build and let the non-zero exit code stop the job.

{% hint style="success" %}
Decide up front how to treat a skipped run. `skipped_no_data` means the scanned rows were empty, which may itself be a problem after a load.
{% endhint %}

## Create monitors as code

Keep monitor definitions in a file under version control, and apply them with a script that creates only the monitors that don't exist yet. The script matches on monitor name, so run it as often as you like.

```python
MONITORS = [
    {
        "name": "orders | loaded within 2 hours",
        "dq_dimension": "timeliness",
        "configuration": {
            "type": "freshness_sla",
            "primary_asset": {"type": "dataset", "id": 123},
            "schedule": {"frequency": "every_hour"},
            "threshold": {"type": "duration", "value": 2, "unit": "hours"},
            "detection_method": {"type": "metadata"},
        },
    },
    {
        "name": "orders | daily row count within -20% to +50%",
        "dq_dimension": "completeness",
        "configuration": {
            "type": "volume",
            "primary_asset": {"type": "dataset", "id": 123},
            "schedule": {
                "frequency": "daily",
                "start_time": "07:00:00",
                "timezone": {"type": "zone", "name": "Asia/Singapore"},
            },
            "detection_method": {
                "type": "full_table_count",
                "threshold": {"type": "percentage_delta", "min": -0.2, "max": 0.5},
            },
        },
    },
    {
        "name": "orders.email | valid email format",
        "dq_dimension": "validity",
        "configuration": {
            "type": "regex_match",
            "primary_asset": {"type": "property", "id": 4522},
            "schedule": {"frequency": "manual"},
            "pattern": {"type": "preset", "pattern_name": "email"},
            "tolerance": {"type": "percentage", "value": 0.01},
            "scan_method": {"type": "full_table_scan"},
        },
    },
    {
        "name": "orders | every order has a customer",
        "dq_dimension": "consistency",
        "configuration": {
            "type": "custom_sql",
            "primary_asset": {"type": "source", "id": 7},
            "attribution_assets": [{"type": "dataset", "id": 123}],
            "schedule": {"frequency": "manual"},
            "sql_query": "SELECT o.* FROM sales.orders o LEFT JOIN sales.customers c "
                         "ON o.customer_id = c.id WHERE c.id IS NULL",
            "search_space_query": None,
            "tolerance": {"type": "absolute", "value": 0},
        },
    },
]

existing = {m["name"] for m in search_all("/monitors/search", {})}

for monitor in MONITORS:
    if monitor["name"] in existing:
        continue
    created = call("POST", "/monitors", json=monitor)
    print(f"Created monitor {created['id']}: {created['name']}")
```

{% hint style="info" %}
The UI asks you to run **Test Value** before saving a Custom SQL monitor. Run your `sql_query` on the source yourself first to confirm it returns the failing rows you expect.
{% endhint %}

To apply changes to monitors that already exist, read each one with [Get a monitor](https://docs.decube.io/public-api/overview/index/data-quality-v2/monitors.md#get-a-monitor), compare it with your definition, and send the difference through the [update endpoints](https://docs.decube.io/public-api/overview/index/data-quality-v2/monitors.md#update-a-monitor). Keep in mind that most configuration changes create a new version.

## Sync new cases to your ticketing tool

Pull the cases that received incidents since your last sync, open a ticket for each one that doesn't have an owner yet, and assign the case so your team sees who's on it in Decube as well.

```python
from datetime import date, timedelta

def create_ticket(case, incident):
    """Replace with a call to Jira, ServiceNow, Linear, or your own tool."""
    print(f"Ticket for case {case['id']}: {case['monitor']['name']}")

since = (date.today() - timedelta(days=1)).isoformat()
cases = search_all("/cases/search", {
    "filters": {"states": ["open"], "assignees": [], "incident_from": since}
})

for case in cases:
    latest = call("GET", "/cases/incidents",
                  params={"case_id": case["id"], "page_size": 1})["items"][0]
    incident = call("GET", "/incidents/details", params={"incident_id": latest["id"]})
    create_ticket(case, incident)
    call("PATCH", "/cases", params={"case_id": case["id"]},
         json={"assignee_id": {"value": "<triage-user-id>"}})
```

The incident's `evidence` and `debug_query` give the ticket enough context to start the investigation without opening Decube. When the ticket is resolved, close the case with `{"action": {"value": {"type": "close"}}}`.

## Pause monitors during maintenance

Disable every active monitor on a table before a backfill or migration, then turn the same monitors back on afterwards. Disabling stops scheduled runs, so the maintenance doesn't raise incidents.

```python
def pause_monitors(dataset_id):
    monitors = search_all("/monitors/search", {
        "filters": {"assets": [{"type": "dataset", "id": dataset_id}], "statuses": ["active"]}
    })
    for monitor in monitors:
        call("PATCH", "/monitors", json={"monitor_id": monitor["id"], "action": {"type": "disable"}})
    return [m["id"] for m in monitors]

def resume_monitors(monitor_ids):
    for monitor_id in monitor_ids:
        call("PATCH", "/monitors", json={"monitor_id": monitor_id, "action": {"type": "enable"}})

paused = pause_monitors(dataset_id=123)
# ... run your maintenance ...
resume_monitors(paused)
```

Store the returned IDs so that you re-enable only the monitors you paused, and leave monitors that were already disabled as they were.

## Report on monitor health by team

Use custom attributes to tag monitors by owning team, then pull a summary of which monitors are failing for each team. The same pattern works for any attribute you use, such as domain or regulatory classification.

```python
from collections import Counter

monitors = search_all("/monitors/search", {"filters": {"statuses": ["active"]}})
states = Counter(m["configuration"]["last_run_state"] or "not_evaluated" for m in monitors)
print(states)  # e.g. Counter({'passed': 182, 'failed': 9, 'errored': 2, 'not_evaluated': 4})
```

Add a `custom_attributes` filter to the search body to narrow the report to one team.
