Changelog
Here's where we let you know what's new on decube.
Multifactor Authentication with authenticator apps is now supported.
- MFA can now be enforced in your organization by an Owner or a user with
Manage users for access management. - Users can set up MFA on their account to ensure security for the account.
Release 1.7.0
New module - Data Governance. One place to manage classification policies in your organization.
Users are able to store all governance policies in the platform to be managed as well as link actual data assets from the Catalog to the policy.
- Classifications are now customisable - users can now create custom classifications by adding a custom policy in the Classification Policies tab.
- Users can now use our auto-classification workflow to set a rule in connected data sources to automatically tag columns that match a keyword or regex input. Example: classify all first_name columns with PII.
- All classified assets under a policy (eg. list of columns classified as PII) shows up in the Managed assets of the policy. Users can export the entire list in csv format.
Module-based policy in Group Management has been updated with new selection in dropdown that is Data Governance. Users will need to give access to the Data governance module by adding the required access through the module-based policy form.Revamped Config - Centralised control to create, update all monitors.
Our config module landing page simplifies the process of setting up various monitoring methods for your assets. You can navigate here by the Config panel in the sidebar.
- All monitors in Config now shows all available monitors that your organization has added, for easier modification and enable/disable. Job Failures, Schema Drift, Custom SQL monitors all are now shown in one page based on your selected data source.
- Incident Levels (Info / Warning / Critical): You can now set individual incident details
- For Field Health, bulk monitoring is now supported. Instead of adding a single test to each column at a time, this has been improved to add tests to multiple columns at once.
- You can control Schema Drift monitoring in a more granular manner by controlling which schema change type to monitor, and which schema to be monitored.
- Smart training & Lookback period: You can now shorten the period of training for each monitor by enabling smart training. Adding a lookback period also allows you to backfill your monitor with historical data, so you can see any anomalous data from before the time the monitor was set up.
- Custom alert channels: You can now set up specific alert channels for each monitor. For example, link Airflow Job failures monitoring to a specific Slack channel #job-failures. This setting will override the default alert setting in your organization.
- With the addition of custom alerting, we have upgraded the notification system to send out individual alerts instead of grouping the alerts based on test type. With this change to individual alerts, each alerts will now contain more information for each incident, and clicking on the provided link will send you straight to the specific incident details page instead of landing on the Data Quality page, saving you time.
- Row Creation selection for monitoring: “Metric Time” has now been renamed to “Row Creation” for more clarity. There are several important changes here:
- All Records: Adding a monitor to scan the entire table is now supported.
- SQL Expression: If there’s no datetime/date column in your table to be selected as a timestamp column (eg. you store it as a string type column), you can now convert it by adding a valid SQL expression to be used as a timestamp column on our platform.
Incidents Details
- More informative graphs for each incident type, with new chart types and added axis titles.
- The ability to assign the incident to one of the users.
- See the owners for the affected asset in the incident details on the side panel.
- An incident history where user actions of changing incident status of muting, closing the incident is tracked.
- For Field Health incidents, a sample of the affected rows can be found directly in the Preview tab, where dynamic masking of PII information is applied.
- See all scans from your monitors with datetime and actual with expected values in the History tab. Users can click into any failed incidents to see the specific incident description.
- See all assets impacted downstream of the asset with incident in the Impacted Areas tab, users can also download the list as a csv.
We added a new connector: AWS Glue. You can now see your transformations as a Data Job in the Catalog.
- UPS error ("invalid asset") is resolved during group creation without any sourced-based policy added.
- Missing data sources during Sourced-Based Policy creation now appears.
- Clicking on back button on policy forms sends user back to Policy selection form instead of closing the modal.
- Source Modal Policies creation using template (Use as template) fixed.
- When user wants to add more than 3 tags, there is now an error prompt that is shown. Users will also be shown an error when attempting to create a tag with special characters.
- "Database" word removed from forms with non-database connectors.
- Search in our Catalog is now powered by Elasticsearch, allowing for a better search experience with natural language search. Searching by path has also been enabled, such as searching
schema_name.table_name. - Clicking on specific search suggestions from your search input now leads you directly to the asset. Pressing on “Enter” or clicking on the Search button searches your Catalog with your search input.
- Narrow down your search by quickly filtering assets by type, schema, tags, classifications, owners, and count of incidents, ensuring you find exactly what you need with minimal effort.
- Introduced a sort option to sort by relevance, or alphabetical order.
- Glossary items such as Glossary, Categories and Terms are also shown in search results. Clicking on these will bring you to the location of it in the Glossary directly.
- Classification not showing in the Glossary Terms has been fixed.
Page crash experienced on Asset Details > Preview for Table assets has been fixed.
Added patches for security vulnerabilities identified.
- Fixed the issue of the password field on the organization invite form not allowing special characters. Added a list of special characters in the UI.
- The issue of schema selection in BigQuery form not being able to be submitted has been fixed.
- 1.Our navbar became thinner! This gives you more space for your Lineage, Incident Details to take up more real estate on your screen.
- 2.Our lineage also now can collapse once you've expanded it, by clicking on the "-" sign.
Bug fix
Some of our tooltips were hard to read because of the sizing, so it is not fixed to fit perfect on each component.
What's new
What's new
- 1.Group Management: Manage user access by creating groups and granting these groups permissions. Check out the summary here.
- If you're an existing customer, we will migrate you seamlessly to the new system. Admins are now in the Owners group, and Members will be transitioned to the Members (Legacy) group. All data sources access will be remained. Read more.
- 2.Approval Workflow: Now you need to go through an approval process to make changes to the Catalog or Glossary. Check it out here.
- 1.New connector: Azure Synapse Analytics! We are continuously adding integrations to Azure products. Check out our Public Roadmap here.
- 2.Made our Schema Drift incidents to show description of the event that happened, eg.
addition,table, ortype_change.
- 1.
- 2.Upgraded our Lineage with our SQL parsing engine, you may see some new lineage relationships shown in the Lineage tab of your Asset details.
- 3.You can now disconnect your Slack organization on the Configure Alerts page.
- 1.Issue where the decube Catalog does not show the right assets from your added Databricks Catalog is now fixed.
- 1.Data Recon UI has been completely refreshed! This includes:
- Adding new recon is now a more focused experience, taking up a whole page instead of being a sidebar.
- You can compare datasets from two separate data sources.
- Set up conditions to filter out rows to be checked in the recon. This functions as a WHERE clause. Read more here.
- Selection of datetime field is now optional.
- You can now export your completed recon in .csv. This downloads the unmatched rows found during the data recon.
- 2.Preview tab has been added for Asset Details.
- We show a sample of 10 rows from your table.
- If there are columns that are marked as PII or Sensitive, this will show up masked in the Preview.
- 1.UI clashing issue with the sidebar on small window sizes has been fixed.
- 2.Fonts in graphs and charts have been updated to Plus Jakarta Sans.
- 3.Notify toggle on Modify monitor modal has been fixed. It now shows the correct state, whether it's enabled or disabled.
New look on our app! We've updated our font for better readability across all our pages.
- 1.Lineage graph not re-appearing when navigating tabs has been fixed.
- 2.When viewed on small screen sizes, the UI on Table Overview had clashing issues, which has been fixed. Screen sizes down to 720p on normal scaling are now supported without issues.
- 1.On Safari 15.6, page crash when viewing the field health monitors tab has been fixed. (Safari 16.1 and other browsers are not affected by this bug).
- 2.Job failure monitors: Toggle for Notification on Data Job monitors now work as intended.
- You can now see the lineage from your transformations in your Lineage now! Just go to the Asset Details and head to the Lineage tab. You'll need to map your Lineage via the Additional Config tab in the Modify Data Source under My Account.
- Support for Azure SQL has been added.
- Documentation editor icons have been fixed. The editor also looks better on wider screens now.
- Data Recon now shows the empty state when there are no new data recons added yet.
Clicking on a table in Dashboard was sending users to a broken link. It's not been fixed and should lead to the Asset Details now.
- Added new integration: SQL Server! You can now connect your SQL Server to decube to monitor your data health and see all your tables in the catalog. Read more about it here.
- Business glossary: Introducing a new feature where users can add their own metrics, add owners and documentation. You can also link them to data assets. The business glossary will allow users to create a common language for metrics and data assets, improving communication and collaboration across teams.
- New integration with Fivetran
- New integration with Databricks
- New integration with dbt, supporting new asset types such as data jobs, data runs, virtual tables etc.
- New incident type, job failures has been introduced.
- Schema tab added, users can now add tags and classifications to fields.
- Field stats added, users can now query their datasets and find out some statistics on each field. Also integrated the data quality incidents into this view.
- A new look to the Data Quality and Dashboard page, with new asset icons.
- Fixed a bug where Intercom is still recognizing the user when user is logged out automatically.
- User is now directly correctly to the filtered incident types in the Data Quality page from notifications.
- Incident graph in Data Quality main page are now bar charts!
- For Google Big Query and Snowflake connections, you can now add Volume and Freshness monitoring without a metric time selection.
- For all sources, you can now use
datecolumn as a metric time column. Adding adatetype column would restrict your scan frequency selection to24 hoursonly. - Monitoring for schema changes is now enabled on all tables by default. You will just need to let us know if you'd like to receive notifications by toggling the notification within the Asset Details.
- UI improvements on the Bulk Table Config and Asset Table Config for better user experience, such as updated icons for metric time selection and descriptive placeholder texts. Also added clearer tooltip for users to be aware of the pre-requisite to select the metric time before a field monitor can be added.
- Lots of bug fixes and improvements made on the Data Quality engine for better reliablity and flexibility.
If user becomes inactive, after some time the cookie will expire and user will be redirected to the log-in page. Logged in user who is actively using the application will have user session automatically extended.
What’s New
- Sign-up and sign-in pages have been updated to be more responsive and make the sign-in process smoother.
- We are now supporting your organization via Intercom! Drop us a message via the bubble on the bottom right if you need any help.
- The UI for adding and modifying monitors has been updated.
Fixed new data recon: Scheduled run period not submitting when user did not interact with the dropdown.
What’s New
- For Google Big Query connections, you can now specify which schemas are to be scanned into decube.
- If you previously had connected GBQ, you can now go to My Account and modify the connection and unselect the schemas that you would not like to include. These schemas will be removed on our next collector run.
- You can also opt to not automatically add new schemas in the future when they are detected by our collector by selecting your preferred option.
- Don’t like the name that you used while signing up? You can now update it via the Update Profile tab on My Account.
- You are now able to add Owners (data owner and business owner) to an asset. For now, tags on tables will be unavailable as we are improving this functionality to add the tags directly on the field level such as PII, Sensitive tagging which is coming very soon.
- You can now save time by updating your table-level tests (Freshness, Volume, Schema drift) from the Data Quality page directly.
- The incident details graph has been improved to show the min/max of the Expected vs the Actual for Volume graphs.
- Our docs has been updated for easier navigation to specific connections and features.
- Fixed the count of data sources on filter selection when data sources were disabled.
- Fixed behavior of custom monitors lists so that it does not show lists of other tables.
What’s New
User will need to accept the disclaimer on the Table Overview (which is an opt-in only feature) before the profiler is ran.
Bug Fixes
- Bug fixes on the UI on the Dashboard and the bulk selection.
- Terms and Privacy Policy on the sign up page have been updated with new links.
There is now a sorting logic on the Bulk Table Configuration so that monitored tables appear on the top of the list followed by selections with metric time. This helps you to be able to see configure your tables easily.
- Plans & Billings page under My Account now shows the usage limit according to the plan the organization is subscribed to.
- UI fixes across the app such as truncating long asset names and rephrasing for a better user experience.
Creating a new data recon now shows loader when operation is running. If a recon job is not successful, a failure state will be shown.
A new connector to Snowflake has been added. Users can now add Snowflake as a new data source under the My Account page.
- When creating a new Data Recon, you can now search for your tables that you would like to use.
- When there are no incidents yet to be shown on the dashboard, we now have a more descriptive text on the UI.
The Lineage UI has been enhanced with a cleaner look and new icons.
- Disabled data sources now do not show inside data sources selection.
- Incidents list within Table details have been fixed.
Fixed issue in Bulk Table Configuration where selections were not able to be viewed when there are many metric time selections.
- Catalog has been enhanced with new icons and filtering for search functionality.
- User can now add new monitor directly from the Data Quality page.
- Get support via live chat on the bottom right of our page.
- Custom SQL monitors can now be added:
- User will be able to add Custom SQL scripts to write specific tests cases as monitors.
- Monitors page within Table details has been updated to support custom monitors which has user-defined names and descriptions.
- Fixed issue where more tables and incidents were not loaded when scrolling to the bottom of the page in the Dashboard and Data Quality.
- Fixed issues on Data Recon page.
- User are now able to use Bulk Table config feature to do bulk selection on multiple metric time columns. This new feature can be accessed via the Catalog.
- Users will be required to configure their tables’ metric time columns and monitoring before enabling table-level and field-level tests. Switching on monitoring for a table enables all table-level tests to be on by default.
- Schema/dataset name has been added to new data recon selection.
- Incident debug messages have been improved.
- Adding tags to assets, schema drift and table config overlay issue with metric time selection have been fixed.
- Removed error logs on console to prevent info leaks on logs.
- Ensure users are not able to log in after deactivation of account.
- User management
- Admin able to change user roles of other users (
membertoadminand vice versa). - Admin able to revoke invites of users who are
pending sign up. - Admin able to deactivate/re-activate other users.
Safari users facing issues with UI freezes on certain pages with loading functionality which was fixed with his hotfix.
Last modified 6h ago