Set Up Freshness & Volume Monitors
Here's how you set up Freshness & Volume Scheduled and On Demand Monitors
Last updated
Here's how you set up Freshness & Volume Scheduled and On Demand Monitors
Last updated
Our Freshness monitors observe the time elapsed since the last update or insertion in a table, learning from the update frequency and alerting if delays arise. Similarly, Volume monitors gauge the number of rows added to a table. If the row count differs significantly from expected patterns based on past data, an incident is triggered.
The process to set up monitoring for Freshness and Volume monitors (both Scheduled and On Demand) is alike. To illustrate, let's walk through setting up monitoring using Freshness as our example.
Begin by selecting "Freshness" card from Create subtab on Config
Once selected, you’ll be redirected to the “Create a New Monitor” form.
The “Create a New Monitor” form consists of two steps:
Set up
Configure
Select the Source and Schema selection is optional
Select the Dataset
Choose Monitor mode: Scheduled or On-Demand
Enable “Grouped By” (if applicable) by toggling the switch.
Select the column for grouping and click Validate.
A success message (“Column is valid to be grouped by”) confirms validation.
Click “Proceed to Monitor Setup” to move to the next step.
Once you proceed to setup, you’ll reach the “Configure” page, where you can review your previous selections.
In the “Configure” popup, users must complete the required fields to save their preferences and set up their Freshness/Volume monitor. These required fields include:
Monitor Name
Monitor Description is Optional.
Row Creation: Select the Row creation from given options:
Timestamp (Select Timestamp from the dropdown column)
Validation for SQL Expression (when SQL Expression is chosen)
All Records
Enable Smart Training(Optional): Train your monitor on historical data to reduce the training period
SQL Expressions allow you to convert your non-standard timestamp formats into timestamp when setting up scheduled or on-demand monitors. This allows you to utilize the incremental monitor scanning functionality based on a given timestamp column.
Common Use Cases & Examples
Converting a String Column to a Timestamp
Some datasets store timestamps as strings, requiring conversion.
For example:
BigQuery- CAST(your_string_column AS DATETIME)
PostgreSQL- your_string_column::timestamp
Converting Unix Timestamp to a Datetime
If timestamps are stored as Unix epoch time (seconds or milliseconds), conversion is required.
For example:
BigQuery- TIMESTAMP_SECONDS(your_unix_column)
PostgreSQL- TO_TIMESTAMP(your_unix_column)
Combining Separate Date and Time Columns
Some databases store date and time in separate columns, requiring concatenation.
For example:
(use date_column || ' ' || time_column)
When Should You Use SQL Expressions?
Use SQL Expressions when:
Your timestamps are stored as strings instead of actual timestamp values.
Your dataset uses Unix timestamps instead of standard date/time formats.
You have separate date and time columns that need to be combined.
If your data already contains a proper timestamp column, you can simply select it without needing an SQL Expression.
Smart Training automatically learns patterns from historical data to enhance row creation and threshold settings. It helps users define dynamic monitoring parameters by analyzing past trends, improving anomaly detection and data quality insights.
To set custom alerts, first, turn on the "Notify default channel" toggle. This enables users to specify their preferred alert channels, such as email or Slack.
Select the desired alert channels from the dropdown.
Enter the email address or Slack channel name in the provided field.
Finally, specify the Incident Level.
Click on Submit to create your monitor successfully.
Once the monitor is created, you will be redirected to the ALL MONITORS tab.
Select On-Demand as the Monitor Mode and click “Proceed to Monitor Setup”.
Within the 'Configure' popup, users must complete the required fields to save their preferences and successfully set up their Freshness/Volume Monitor. The required fields include:
Monitor Name
Monitor Description is Optional.
Row Creation: Select the Row creation from given options:
Timestamp (Select a timestamp column from the dropdown)
Validation for SQL Expression (when SQL Expression is chosen)
Lookback Period
Incident Levels
Finalizing On-Demand Monitor Setup
Upon completing the required fields to create an On Demand monitor for Freshness/Volume, users can complete the setup process by choosing one of the following confirmation buttons based on their preferred use case:
Save: Creates the monitor without running it. This is an applicable option for users who wish to set up an On-Demand monitor without running the monitor scan immediately after creation.
Save and run : Runs the On-Demand Monitor immediately upon creation. To run the monitor again later, users can navigate to All Monitors.
After selecting the above option you will be redirected to ALL MONITORS tab.
Modify Monitoring
To modify an existing monitor:
Click the ellipsis (︙) and select View Monitor.
Click on Run once to run the monitor manually.
For a detailed understanding of monitor modes, check out
For a detailed understanding of Grouped By, check out
Frequency (Learn more about )
When working with Google BigQuery, you can review the provided documentation for further details .
Go to
