Set Up Field Health Monitors
Here's how you set up monitors for specific field tests.
You can enable monitoring for field health by selecting the "Field Health" card in the Config landing page.
You can also activate field health monitoring within the "Asset Details" section of the Data Catalog Module. This can be done via the "Monitors" tab by selecting the "Configure Monitors" option.
Upon selecting the "Field Health" card, it's compulsory to first choose the data source. Only after making the data source selection, you then are able to pick the desired Schema (Optional) and the table whose assets you wish to monitor.
Upon choosing your table, a set of properties from the selected table will be loaded enabling you to set up monitoring. Concurrently, a list showcasing both active and inactive monitors previously configured for that specific chosen table will also be displayed.
On the list of properties loaded from the selected table:
i. For tables that have never had monitoring configured, a "Set up monitoring" button will be displayed when initiating setup for the first time.
ii. For tables previously set up for monitoring, you will notice a notification toggle. Additionally, an ellipsis (︙) menu will be present, offering options to edit, enable/disable, or delete the existing monitor configuration.
You can choose which test types to run. Our system provides several test types including null%, regex_match, is_email and more. On the Set Up Monitoring pop up, can also toggle the notification settings as well as custom alerts for each monitor.
If the current preset monitors is not sufficient to run the specific test you require, you can also create a test via custom SQL script.
Upon configuring monitoring for the first time on the table asset you aim to monitor, a popup will appear upon clicking the "Set up monitoring" option.
Within the "Set Up Monitoring" popup, users must complete the necessary fields to save their preferences and successfully set up their field health monitor. These required fields include:
- Row Creation
- Timestamp (Select Timestamp from the dropdown column)
- Validation for SQL Expression (when SQL Expression is chosen)
- Frequency
- Threshold (if Enable Machine Learning is deactivated)
- Incident Levels.
To set custom alerts, you must first turn on the "Notify" toggle. Activating this will allow users to specify their desired alert channels, be it via emails or Slack channels.
Smart Training requires Row Creation to be selected.
To activate Smart Training in Row Creation:
- Users should initially select the timestamp.
- If SQL Expression is chosen for row creation, users are required to validate the SQL Expression.
When Smart Training is activated for Row Creation, it enables the option to choose the Lookback Period. It's important to note that the Lookback Period becomes selectable only when Smart Training is enabled.
Upon completing the required fields, users can click on "Save Preferences." Subsequently, the freshly configured monitor will be displayed in the list of table properties for the chosen table.
The newly created Field Health monitor can also be located in the "All Monitor's Tab". This provides a streamlined approach for users who prefer not to reselect the data source, test type, and table when revisiting their recently created monitor.
If you have any modification to be made on your field health monitor, simply click on the ellipsis (︙) menu and select "Modify monitor". Monitors modification can also be made at All Monitor's Tab.
In the Modify Monitor's pop up, users are not able to modify the following fields:
- Smart training.
- Lookback period.
- Row creation.