Ctrlk
Try for free
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Apache Spark

See lineages from spark jobs in Decube Catalog.

Supported Capabilities

General

  • Metadata — metadata extraction and display of asset information (tables, columns, schemas). Types collected: Schema, Virtual Table, Virtual Column, Data Job, Data Run, Data Task

Data Quality Monitors

  • Job Failure

General

  • Profiling

  • Preview

  • Data Quality

  • Configurable Collection

  • External Table

  • View Table

  • Stored Procedure

Data Quality Monitors

  • Freshness

  • Volume

  • Field Health

  • Custom SQL

  • Schema Drift

Apache Spark can map lineage relationships to upstream and downstream objects from the following connectors:

  • Upstream Connectors: postgresql, adls

  • Downstream Connectors: postgresql, adls

Connection Requirements

Please see the instructions and minimum requirements for configuration in each data source below:

PreviousAzure Data FactoryNextApache Spark in Azure Synapse

Last updated