Monitor Behaviour and Design Considerations

How the Data Quality module is designed to behave in specific scenarios, and how to configure monitors to get the most out of each.

The Data Quality module makes a number of deliberate design choices about how monitors train, alert, and respond to configuration changes. Understanding these upfront helps you set up monitors that work reliably from day one.

---

Sensitivity adapts through use

Rather than asking you to set a sensitivity value at monitor creation — when you have no data to reason from — Decube starts every monitor at the default sensitivity (0) and lets you adjust it based on what you observe in practice.

Once a monitor has been running and producing incidents, you can tune its confidence interval through the incident feedback mechanism. Marking incidents as false positives shifts the model toward fewer alerts; marking missed anomalies shifts it toward more. This means sensitivity improves as your monitors accumulate real operational history.

Incident model feedback

---

The ML model waits for sufficient signal before alerting

To avoid producing unreliable alerts on insufficient data, the ML model requires a minimum of 5 valid data points in the last 30 observations before it begins raising incidents. On a newly created monitor or a low-volume table, scans may run without producing incidents until this threshold is reached — this is by design.

Once the signal threshold is met, the model activates and begins producing confidence-interval-based alerts.

If you need immediate coverage on a low-volume or sparsely-updated table, use On-Demand mode with a manual threshold. Manual thresholds evaluate on every scan without requiring prior observations.

How Anomaly Detection Works

---

Custom SQL gives you full control over thresholds

Custom SQL monitors evaluate threshold conditions you define explicitly, rather than using the ML model. This makes them well-suited to complex business rules and cross-table assertions where the right threshold is deterministic — for example, "zero rows should have a negative price" or "these two tables must always agree."

Set Up Custom SQL Monitors

---

Retraining gives you a clean, accurate baseline

When you make a significant change to a monitor's configuration — such as changing the scan frequency, row filtering mode, or timestamp column — Decube starts the training process fresh. This ensures the model is trained on data that reflects the monitor's new configuration, rather than producing a baseline that mixes two different measurement approaches.

Because the previous history no longer reflects the new configuration, it is cleared as part of the retrain. Before making any change that triggers a retrain, note or resolve any open incidents you want to preserve.

Retraining Monitors