Databricks

Adding Databricks to your decube connections helps your team to find relevant datasets, understand their quality via incident monitoring and apply governance policies via our data catalog.

Adding a Databricks connection on decube.

From the My Account page, select the Databricks tile to be brought to the Databricks connection form.

You will need to generate an access token and create a service principal to connect to Databricks.

Required Information

1. Workspace URL

2. Personal Access Token

3. SQL Warehouse HTTP Path

4. Catalog to scan

Getting a Personal Access Token

The full documentation from Databricks can be found here.

1. Navigate to your Users Settings page.

2. Click on Generate New Token after navigating to the Access Tokens tab.

3. Give your token a name and specify the lifetime of the token. We suggest not specifying a lifetime to ensure uninterrupted service.

4. Once you click Generate ensure that you note down the token somewhere as you cannot retrieve it again. This is the Access Token for decube's form.

Getting the Workspace URL

Go to your workspace and copy the url in the browser bar.

It should look like <some_values>.cloud.databricks.com . The entire url highlighted in the screenshot is the Workspace URL to be added to decube's form.

Getting the SQL Warehouse HTTP Path

1. Go to your SQL Workspace as shown and then Navigate to the SQL Warehouses section.

2. Either create a new SQL warehouse (recommended) or choose an existing SQL Warehouse to be used with decube and navigate to the Connection Details tab. You will see both the Server hostname which should match your Workspace URL and the HTTP Path of the SQL Warehouse

We recommend creating a serverless Databricks SQL Warehouse. Other solutions may cause failure during metadata ingestion or data monitoring due to warehouse warm-up time.

If there is any issues or questions related with connecting your data sources, please reach out to us via our Live chat for support.