Google Bigquery

Adding Google Big Query to your decube connections helps your team to find relevant datasets, understand their quality via incident monitoring and apply governance policies via our data catalog.

Connecting to Big Query requires following the steps below to completed.

Connect Big Query

Prerequisite

To ensure a smooth experience configuring the connection.

1. Having access to the Project that contains the intended BigQuery.

2. An account that can view and create Service Accounts

Creating decube Data Observability Role

A custom role for the decube service account is required to ensure we have correct access to the resources we need.

1. From the dashboard, click the hamburger (three lines indicating a menu), the top left of the browser next to the Google Cloud logo, and Search for IAM and Admin.

2. Find Roles and at the top, below the search bar, click + Create Role.

3. Fill in Roles as below

• Title: decube data observability

• ID: decube

• Role launch Stage: General Availability

1. Click on Add permission, a popup will be displayed. Then on Enter property name or value type each of the permissions below, press enter and click on the checkbox to assign that permission to the decube data observability role.

bigquery.datasets.get
bigquery.datasets.getIamPolicy
bigquery.jobs.create
bigquery.jobs.get
bigquery.jobs.list
bigquery.jobs.listAll
bigquery.readsessions.create
bigquery.readsessions.getData
bigquery.readsessions.update
bigquery.routines.get
bigquery.routines.list
bigquery.tables.get
bigquery.tables.getData
bigquery.tables.list
resourcemanager.projects.get
storage.buckets.get
storage.buckets.list
storage.objects.get
storage.objects.list

1. When you are done assigning permissions, click Add to exit the popup, on Assigned Permission table, review the permissions granted to match above requirements.

2. Click Create.

Enabling Reporting Functionality

Our reporting module requires additional permission and Google Cloud Asset API to be enabled. We suggest creating a new role and attaching it to the existing service account above.

cloudasset.assets.analyzeIamPolicy
cloudasset.assets.searchAllIamPolicies
cloudasset.assets.searchAllResources

For full report generation especially with multiple GCP projects, the service account will need additional role on the Organization level. Without these additional roles, the generated report will only contain queries from the current project

bigquery.jobs.list
bigquery.jobs.listAll

1. Go to GCP API and Services and click Library.

2. Search for “cloud asset api”.

3. Click Enable

4. Go to IAM and admin and Roles section click Create Role.

5. Grant the role with these permissions and click Create

cloudasset.assets.analyzeIamPolicy
cloudasset.assets.searchAllIamPolicies
cloudasset.assets.searchAllResources

6. Grant the Role to the service account attached with the Bigquery source.

Getting a Service Account JSON Key File

For decube to connect with BigQuery, a JSON key file is required from a Service Account with the proper Roles assigned to it.

1. From the dashboard, click the hamburger (three lines indicating a menu), the top left of the browser next to the Google Cloud logo, and Search for API's and Services.

2. On that page, click on Credentials, at the top, click + Create Credentials and then select Service account.

3. Fill in the Service account details form, we recommend

• Service account name: decube,

1. Click Create and Continue.

2. On Grant this service account access to project, click on Select a role, the on the _Filter _ search, type decube data observability created previously and click to assign this role to the service account. Click Continue.

3. Skip Grant users access to this service account form and click Done.

4. On the Service Account page you will see a Service Account named decube, click on it.

5. On the options below the Service Account name, click on Keys

6. Click Add Key and then select Create new key from the drop-down.

7. For the Key Type, choose JSON and click Create.

8. You will automatically download the JSON file and can refer to the location of the file on your browser, save it somewhere easily accessible like your Desktop.