🧱 Databricks Setup Guide

Digital Tap connects to a Databricks workspace as a service principal, signing in with OAuth machine-to-machine credentials: a client ID and an OAuth secret. It does not accept a personal access token. It checks the credentials work before it stores them.

What You'll Need

Step-by-Step

1

Create the service principal

In your Databricks workspace, open Settings, then Identity and access, then Service principals, and add one, named for example digital-tap. You can also create it in the account console and add it to the workspace.

2

Generate an OAuth secret

Open the service principal, go to its Secrets tab and click Generate secret. Copy the Client ID, which is the service principal's application ID, and the secret. The secret is shown only once.

3

Grant what Digital Tap needs

Generated from the same list the connect check tests against. Digital Tap asks for nothing else: not CAN_MANAGE on a cluster, and no admin role.

  1. Entitlements. A workspace admin adds the service principal to the workspace with the Workspace access and Databricks SQL access entitlements.
  2. Clusters. On each cluster Digital Tap may hibernate and resume, grant the service principal CAN_RESTART. Clusters it should only monitor need CAN_ATTACH_TO. CAN_MANAGE is not needed. A grant to a group it belongs to works too.
  3. Jobs. On each job that runs on those clusters, grant it CAN_VIEW, so the check before a hibernate can see the job's runs: a run it can see blocks the hibernate. Nothing can show that it sees every job, so by design every hibernate needs a person's approval. A hibernate Digital Tap proposes waits on the Approvals page until an owner or admin of your Digital Tap organization approves it.
  4. SQL warehouse. For system-table analysis, grant it CAN_USE on one SQL warehouse. You enter that warehouse's ID in the connect form.
  5. System tables, granted by a metastore admin. A workspace admin is not enough: a workspace admin's GRANT on the system catalog is refused. A metastore admin runs the following, replacing <client-id> with the service principal's client ID: 
    GRANT USE CATALOG ON CATALOG system TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA system.compute TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.clusters TO `<client-id>`;
GRANT SELECT ON TABLE system.compute.node_timeline TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA system.billing TO `<client-id>`;
GRANT SELECT ON TABLE system.billing.usage TO `<client-id>`;
GRANT SELECT ON TABLE system.billing.list_prices TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<client-id>`;
GRANT SELECT ON TABLE system.lakeflow.job_run_timeline TO `<client-id>`;
GRANT SELECT ON TABLE system.lakeflow.jobs TO `<client-id>`;
GRANT USE SCHEMA ON SCHEMA system.access TO `<client-id>`;
GRANT SELECT ON TABLE system.access.audit TO `<client-id>`;
    system.compute, system.lakeflow, system.access may also need enabling for the metastore by an account or metastore admin.
4

Paste the client ID and secret into the connect form

In the dashboard, open Settings and click + Connect Platform. Choose Databricks, enter the Workspace URL, Client ID, Client secret and, for system-table analysis, the SQL warehouse ID, then click Connect & Scan.

Verify Connection

Digital Tap exchanges the client ID and secret for a short-lived token and lists the workspace's clusters before it stores anything. If either step fails, nothing is stored and the form shows Databricks' reason. A successful connection reports:

Connecting grants no automatic actions. A hibernate Digital Tap proposes appears on the dashboard's Approvals page for an org owner or admin to decide.

What Gets Monitored

Digital Tap reads, through Databricks' official APIs and system tables: