Automated Lineage
Visualize your data assets from source to consumption.
Automated lineage visually represents your data's journey, showing your upstream sources and downstream consumers via relationships between associated columns. Lineage in Decube is automatically generated based on the metadata ingestion completed on your connected data sources.
For data teams and governance stewards, automated lineage acts as an essential map. It helps you quickly understand the downstream impact of data incidents, trace the root cause of reporting errors, and ensure overall data trust without needing to manually dig through database code.
You can access the lineage of your table by navigating to the Lineage tab under each table's Asset Details.
To view the different lineage types available for each data source, check the Supported Capabilities section in each connector page. Example: Supported Capabilities
The lineage is a great way for your teams to see the downstream impact of incidents that may have been triggered by our Data Quality module.
Navigating the Lineage Canvas
When you first open the Lineage page, you will see your Entry Point Asset (the asset you have navigated from). The lineage canvas is a highly interactive workspace designed for smooth exploration.
Panning and Zooming: Click and drag anywhere on the empty canvas to pan your view. You can zoom in and out using your mouse wheel or trackpad pinch gesture.
Expanding Lineage: If there is lineage data available, a Plus (+) icon will appear on the left (upstream) or right (downstream) of a node. Clicking this icon loads the next layer of connected assets.
Canvas Controls
Use the controls in the top-right corner to manage your view:
Refocus: Focuses on the Entry Point Asset or the node which you interacted with to select a column.
Reset All: Clears all expanded pathways and column selections, returning the canvas to its initial state showing only the Entry Point Asset.
Fullscreen: Maximizes the canvas area for viewing complex graphs.
Minimap: Located in the corner, this provides an overview of the entire graph and allows you to quickly jump to different areas by clicking.
Table-Level vs. Column-Level Lineage
Table-Level Lineage (Default)
By default, the lineage graph displays at the table/dataset level. Lines connecting the nodes show the general flow of data from one asset to another.
Column-Level Lineage
To investigate specific data points (like tracing a PII column), you can use the dropdown on any node to select specific columns.
Filtered View: When you select a column, the graph automatically filters out (prunes) any nodes that are not connected to that specific column.
Single Active Selection: You can only actively select columns on one node at a time.
Changing Selections: If you try to select a column on a different node, a prompt will ask if you want to clear your previous selection to start a new trace.
Viewing Lineage Details & Data Jobs
Connections between assets are represented by curved lines.
Clicking: Click on any line to open a side panel on the right. This panel displays exactly how the lineage was derived (e.g., ETL job, SQL query, Foreign Key, BI lineage, External Lineage, or view lineage).
For more details on the types of relationships, refer to Lineage Relationship.
Lineage mapping for ETL and BI tools
If you have connected transformation tools like dbt or Fivetran or BI tools like PowerBI, you will be able to see the lineage from different sources to destinations within Decube directly.
You will need to configure these connections first to map the namespaces to their respective Decube sources. For more details, refer to this link Additional configurations
Frequently Asked Questions
Why is my lineage missing or incomplete?
There are a few reasons why lineage might not be automatically picked up. Generally, we can categorize missing lineage into two main types:
Missing lineage within a single source (e.g., Snowflake table to Snowflake table)
Stale Query History: Decube reads from your database's query history. If the query that moves or transforms the data has not run recently, the lineage cannot be generated.
Complex SQL or Views: Decube parses SQL queries and view definitions to build lineage. Highly complex query structures may not be supported by the automated parser today.
Unmapped dbt Namespaces: For tables orchestrated by dbt, you may need to map the namespaces correctly through your dbt source connection settings.
Missing lineage between different sources (e.g., AWS S3 to Snowflake, or Snowflake to PowerBI)
OpenLineage Jobs Not Running: If you are using the OpenLineage framework, verify that your data jobs are actively running and successfully emitting lineage information to the webhook listener.
ETL or BI Namespace Mapping: For lineage expected from ETL pipelines or BI reporting tools, you may need to map the namespaces within the respective tool's source connection settings in Decube.
As noted above, there are multiple reasons why lineage might not appear. Because data environments are incredibly complex, no automated solution can guarantee 100% lineage coverage across every possible scenario. However, the Decube team is committed to providing support and extending our parser to handle new customer use cases wherever possible.
Please see the Need Further Assistance? section below on how to reach out to our team.
Why can I see table-level lineage, but my column-level lineage is missing?
Table-to-table lineage is generally easier to map than column-to-column lineage. This is because tracking an entire dataset moving from Point A to Point B is straightforward.
However, individual columns often undergo heavy transformations—like merging first and last names, running calculations, or filtering data. These are handled with Decube's lineage parsing engine, which determines the relationships based on the SQL that is harvested from your processes.
If there are expected column lineages that are missing, it may fall under a complex case that is not supported on the column lineage parsing engine yet. See Need Further Assistance? on how to reach out to the Support team to troubleshoot these cases.
Does Decube track lineage all the way to my BI dashboards?
Yes, Decube can track lineage from your data warehouse tables directly into reporting tools like Tableau, Looker, and PowerBI.
The level of detail you see (whether it stops at the dashboard level or goes all the way down to a specific chart) depends entirely on what information the BI platform shares with Decube. For example, Tableau shares highly detailed information, allowing Decube to trace lineage down to the exact columns powering a specific chart.
You will need to map the namespaces for these tools to the Decube sources. Read more here: Additional configurations
Why is my dbt (or transformation tool) lineage showing incomplete data?
Decube integrates seamlessly with transformation tools like dbt. However, even these specialized tools do not guarantee full lineage coverage for highly complex, custom code. If the transformation tool cannot determine the exact origin of a data point during its own processes, Decube's automated parser will also reflect that missing connection.
What should I do if the automated lineage cannot read my specific data workflow?
If you have unique data relationships that cannot be automatically traced, you can easily bridge the gap using Manual Lineage. This feature allows your data stewards to manually draw connections between assets, ensuring your visual map remains a complete and trusted source of truth. Read about it here: Add lineage relationships manually.
What types of lineage does Decube automatically discover?
Depending on your connected data sources, Decube automatically maps several types of relationships. You can read about the lineage relationship types available in this doc: Lineage Relationship.
If there are multiple relationship types that can be attributed to a single lineage type, it will be consolidated and shown in the Lineage Connection Details.
Need Further Assistance?
If you have reviewed the common limitations above and believe a lineage connection is incorrectly missing from your environment, our support team can help investigate the automated parser's logs.
Please reach out to [email protected] and provide the following information so we can dive straight into troubleshooting:
Fully Qualified Table Name: Provide the exact name of the table you are investigating.
Pro Tip: The easiest way to get this is to navigate to the Asset Details page of the table in Decube and copy it directly from your browser's URL bar.
Expected Connection: Tell us exactly what is missing. Please provide the name of the specific source table (where the data should be coming from) or target table (where the data should be going).
Data Flow Details (Optional but highly recommended): If you know how the data moves between these tables, please describe the workflow. For example, let us know if the missing connection is built using a specific Snowflake view, processed via a complex dbt transformation, or loaded via a Python script. The more context you provide, the faster we can identify why the parser missed it.
Last updated