Comment on page
Set Up Freshness & Volume Monitors
Here's how you set up Freshness & Volume monitors.
Set up a Volume monitor.
Our Freshness monitors observe the time elapsed since the last update or insertion in a table, learning from the update frequency and alerting if delays arise. Similarly, Volume monitors gauge the number of rows added to a table. If the row count differs significantly from expected patterns based on past data, an incident is triggered.
The process to set up monitoring for Freshness and Volume monitors is alike. To illustrate, let's walk through setting up monitoring using Freshness as our example.
Begin by selecting "Freshness" card from our config module landing page.
Upon selecting the "Freshness" card, it's compulsory to first choose the data source. Only after making the data source selection, you then are able to pick the desired Schema (Optional).
Upon choosing your data source, a set of columns from the selected table will be loaded enabling you to set up monitoring. Concurrently, a list showcasing both active and inactive monitors previously configured for that specific chosen table will also be displayed.
On the list of properties loaded from the selected table:
i. For tables that have never had monitoring configured, a "Set up monitoring" button will be displayed when initiating setup for the first time.
ii. For tables previously set up for monitoring, you will notice a notification toggle. Additionally, an ellipsis (︙) menu will be present, offering options to edit, enable/disable, or delete the existing monitor configuration.
Upon configuring monitoring for the first time on the table asset you aim to monitor, a popup will appear upon clicking the "Set up monitoring" option.
Within the "Set Up Monitoring" popup, users must complete the necessary fields to save their preferences and successfully set up their field health monitor. These required fields include:
- Row Creation
- Timestamp (Select Timestamp from the dropdown column)
- Validation for SQL Expression (when SQL Expression is chosen)
- Frequency
- Incident Levels.
To set custom alerts, you must first turn on the "Notify" toggle. Activating this will allow users to specify their desired alert channels, be it Emails or Slack Channels.
When using SQL Expression, validating your query is compulsory. Ensure that your query is written in the dialect compatible with your linked data source, as illustrated below:
Google BigQuery -
CAST(your_string_column AS DATETIME)PostgreSQL -
your_timestamp_column::timestampWhen working with Google BigQuery, you can review the provided documentation for further details here.
Smart Training is applicable for both Row Creation and Threshold settings.
To activate Smart Training in Row Creation:
- Users should initially select the timestamp.
- If SQL Expression is chosen for row creation, users are required to validate the SQL Expression.
When Smart Training is activated for Row Creation, it enables the option to choose the Lookback Period. It's important to note that the Lookback Period becomes selectable only when Smart Training is enabled.
Upon completing the required fields, users can click on "Save Preferences." Subsequently, the freshly configured monitor will be displayed in the list of table properties for the chosen table.
The newly created Freshness/Volume monitor can also be located in the "All Monitor's Tab". This provides a streamlined approach for users who prefer not to reselect the data source, test type, and table when revisiting their recently created monitor.
If you have any modification to be made on your field health monitor, simply click on the ellipsis (︙) menu and select "Modify monitor". Monitors modification can also be made at All Monitor's Tab.
In the Modify Monitor's pop up, users are not able to configure the following fields:
- Smart training.
- Lookback period.
- Row creation.