Set Up Volume Monitors

Set up Volume monitors to detect when the number of new rows arriving in a table is abnormally high or low.

A Volume monitor tracks how many rows are added to a table in each scan window and alerts you when that increment falls outside the expected range. It is the right tool when you care about how much data arrived — not just whether it arrived.

Freshness vs Volume: which to use

Goal
Use

Detect that data arrived (or didn't)

Freshness

Detect that the right amount of data arrived

Volume

Table receives data on a known schedule

Freshness

Table grows by a predictable row count per period

Volume

If you need both signals, create one monitor of each type on the same table.

How the Volume monitor works

The Volume monitor measures the increment — the number of new rows added since the previous scan — not the total row count of the table. The ML model trains on historical increments to establish a normal range, and raises an incident when a new increment falls outside it.

Backfill warning

If a large historical backfill is loaded into a monitored table, the increment for that scan period will be far above normal. The model will flag this as an anomaly. If you plan a backfill, either mute the monitor during the load or acknowledge the resulting incident as expected behaviour.

Group by not supported

Volume monitors do not support Group By. If you need volume tracking broken down by a column value, use a Custom SQL monitor with a GROUP BY clause.

Low-volume tables

On tables with very few rows per period, the Volume monitor may hit the sparse-data threshold (fewer than 5 valid points in 30 observations) and silently skip scans. For low-volume tables, a manual threshold (Absolute or Percentage) is more reliable than Smart Training.


Before you begin

  • You need at least one data source connected and a table available under that source.

  • To use Smart Training, the table must have a timestamp column (or you must provide an SQL expression that produces one).


Step 1: Set up

  1. In the Data Quality module, go to the Config tab and select Create.

  2. Select the Volume monitor card.

  3. In the Create a New Monitor form, select your Source (Schema is optional) and Dataset.

  4. Choose Monitor mode: Scheduled or On-Demand.

  5. Click Proceed to Monitor Setup.


Step 2: Configure — Scheduled monitor

Field
Description

Monitor Name

A descriptive name for this monitor.

Monitor Description

Optional.

Row Creation

How Decube identifies new rows: Timestamp (select a timestamp column), SQL Expression (provide an expression that produces a timestamp), or All Records.

Smart Training

Toggle on to train the model on historical increments. Requires Timestamp or SQL Expression.

Frequency

How often the monitor scans. See Custom Scheduling for Monitors.

Incident Level

Severity assigned to incidents this monitor opens.

SQL Expression

Use an SQL Expression when your table stores timestamps in a non-standard format. See the SQL Expression examples on the Freshness setup page for common conversion patterns. Validating the expression before saving is required.

Notifications

Turn on Notify default channel to route incidents to a specific email or Slack channel. Click Submit to create the monitor.


Step 2: Configure — On-Demand monitor

On-Demand monitors do not use Smart Training, Auto Threshold, or frequency scheduling.

Field
Description

Monitor Name

A descriptive name.

Monitor Description

Optional.

Row Creation

Timestamp or SQL Expression only.

Lookback Period

The time window to check when the monitor runs.

Incident Level

Severity assigned to incidents this monitor opens.

To finish:

  • Click Save to create the monitor without running it immediately.

  • Click Save and Run to create and run the monitor straight away.

After creation, you can run the monitor again from All Monitors by clicking the ellipsis (︙) and selecting View Monitor, then Run once.


Set Up Freshness MonitorsHow Anomaly Detection WorksRetraining Monitors

Last updated