Set Up Freshness & Volume Monitors
Here's how you set up Freshness & Volume Scheduled and On Demand Monitors
Last updated
Here's how you set up Freshness & Volume Scheduled and On Demand Monitors
Last updated
Our Freshness monitors observe the time elapsed since the last update or insertion in a table, learning from the update frequency and alerting if delays arise. Similarly, Volume monitors gauge the number of rows added to a table. If the row count differs significantly from expected patterns based on past data, an incident is triggered.
The process to set up monitoring for Freshness and Volume monitors (both Scheduled and On Demand) is alike. To illustrate, let's walk through setting up monitoring using Freshness as our example.
Begin by selecting "Freshness" card from our config module landing page.
Upon selecting the "Freshness" card, it's compulsory to first choose the data source. Only after making the data source selection, you then are able to pick the desired Schema (Optional).
Upon choosing your data source, a set of tables from the selected data source will be loaded enabling you to set up monitoring both for Scheduled and On Demand monitoring mode. Concurrently, a list showcasing both active and inactive monitors (both On Demand and Scheduled) that was previously configured for that specific chosen table will also be displayed.
On the list of tables loaded from the selected data source, users are able to set up/modify both On-Demand and Scheduled monitors.
For Scheduled Monitors: i. For tables that have never had Scheduled monitoring configured, a "Scheduled Monitoring" button will be displayed when initiating setup for the first time.
ii. For tables previously set up for Scheduled monitoring, you will notice that the "Scheduled Monitoring" button will be changed into "Modify Scheduled" state with a green indication.
For On-Demand Monitors: i. For tables that have never had On-Demand monitoring configured, a "On-demand monitoring" button will be displayed when initiating setup for the first time.
ii. For tables previously set up for On-Demand monitoring, you will notice that the "On-demand monitoring" button will be changed into "Modify On-Demand" state with a green indication.
Upon setting up monitoring for the first time on the table asset you aim to monitor, a popup will appear upon clicking the "Scheduled Monitoring" option.
Set up monitoring pop up for scheduled monitors is the same for both Volume and Freshness Monitors.
Within the "Set Up Monitoring" popup for scheduled monitors, users must complete the necessary fields to save their preferences and successfully set up their field health monitor. These required fields include:
Row Creation
Timestamp (Select Timestamp from the dropdown column)
Validation for SQL Expression (when SQL Expression is chosen)
Frequency (Learn more about Custom Scheduling below)
Incident Levels.
To set custom alerts, you must first turn on the "Notify" toggle. Activating this will allow users to specify their desired alert channels, be it Emails or Slack Channels.
When using SQL Expression, validating your query is compulsory. Ensure that your query is written in the dialect compatible with your linked data source, as illustrated below:
Google BigQuery - CAST(your_string_column AS DATETIME)
PostgreSQL - your_timestamp_column::timestamp
When working with Google BigQuery, you can review the provided documentation for further details here.
Smart Training is applicable for both Row Creation and Threshold settings.
To activate Smart Training in Row Creation:
Users should initially select the timestamp.
If SQL Expression is chosen for row creation, users are required to validate the SQL Expression.
When Smart Training is activated for Row Creation, it enables the option to choose the Lookback Period. It's important to note that the Lookback Period becomes selectable only when Smart Training is enabled.
Upon completing the required fields, users can click on "Save Preferences." Subsequently, the freshly configured monitor will be displayed in the list of table properties for the chosen table.
The Freshness Set Up Monitoring pop up for On-Demand Monitoring typically differ slightly from those used in scheduled monitoring. Users must complete the following required fields in order to successfully create an On Demand Freshness monitor.
Row Creation ('All Records' option is neglected)
Timestamp (Select Timestamp from the dropdown column)
Validation for SQL Expression (when SQL Expression is chosen)
Incident Levels.
Lookback Period
The Volume Set Up Monitoring pop up for On-Demand Monitoring typically differ slightly from those used in scheduled monitoring. Users must complete the following required fields in order to successfully create an On Demand Volume monitor.
Row Creation
Timestamp (Select Timestamp from the dropdown column)
Validation for SQL Expression (when SQL Expression is chosen)
Incident Levels.
Lookback Period
Set Threshold
Note:
i. The "Frequency" field is not relevant for configuring any On Demand monitors and is therefore neglected.
ii. "Smart Training" and "Auto Threshold" options is neglected when it comes to setting up any On Demand Monitor.
Upon completing the required fields to create a Freshness/Volume On Demand monitor, users can complete the set up process by choosing either of the following confirmation buttons based on their preferred use case:
Save Preferences: This is an applicable option for users who wish to set up an on demand monitor without running the monitor scan immediately after creation.
Save & Run: This option is applicable for users who wish to run the On Demand Monitor immediately upon creation.
The newly created Freshness/Volume monitor (both Scheduled and On Demand) will appear on the list of loaded tables upon selecting a data source, and they can also be located in the "All Monitor's Tab".
This provides a streamlined approach for users who prefer not to reselect the data source, test type, and table when revisiting their recently created monitor.
If you have any modification to be made on your Scheduled or On Demand Freshness/Volume monitor, simply click "Modify Scheduled" or "Modify On Demand". Monitors modification can also be made at All Monitor's Tab.
