Connect Databricks to Digital Tap AI

Digital Tap signs in to Databricks as a service principal, with OAuth machine-to-machine credentials: a client ID and an OAuth secret. It does not accept a personal access token. It checks the credentials work before it stores them.

1 Before you start

• Your workspace URL, from the browser's address bar, e.g. https://dbc-abc123.cloud.databricks.com or https://adb-1234567890.1.azuredatabricks.net
• A workspace admin, to create the service principal and give it its entitlements
• A metastore admin, for the system-table grants in step 4
• A Digital Tap account: sign up if you haven't

2 Create the service principal

In your Databricks workspace, open Settings, then Identity and access, then Service principals, and add one, named for example digital-tap. You can also create it in the account console and add it to the workspace.

3 Generate an OAuth secret

Open the service principal, go to its Secrets tab and click Generate secret. Copy the Client ID, which is the service principal's application ID, and the secret. The secret is shown only once.

4 Grant what Digital Tap needs

These steps are generated from the same list the connect check tests against. Digital Tap asks for nothing else: not CAN_MANAGE on a cluster, and no admin role.

1. Entitlements. A workspace admin adds the service principal to the workspace with the Workspace access and Databricks SQL access entitlements.
2. Clusters. On each cluster Digital Tap may hibernate and resume, grant the service principal CAN_RESTART. Clusters it should only monitor need CAN_ATTACH_TO. CAN_MANAGE is not needed. A grant to a group it belongs to works too.
3. Jobs. On each job that runs on those clusters, grant it CAN_VIEW, so the check before a hibernate can see the job's runs: a run it can see blocks the hibernate. Nothing can show that it sees every job, so by design every hibernate needs a person's approval. A hibernate Digital Tap proposes waits on the Approvals page until an owner or admin of your Digital Tap organization approves it.
4. SQL warehouse. For system-table analysis, grant it CAN_USE on one SQL warehouse. You enter that warehouse's ID in the connect form.
5. System tables, granted by a metastore admin. A workspace admin is not enough: a workspace admin's GRANT on the system catalog is refused. A metastore admin runs the following, replacing <client-id> with the service principal's client ID:

   GRANT USE CATALOG ON CATALOG system TO `<client-id>`;
   GRANT USE SCHEMA ON SCHEMA system.compute TO `<client-id>`;
   GRANT SELECT ON TABLE system.compute.clusters TO `<client-id>`;
   GRANT SELECT ON TABLE system.compute.node_timeline TO `<client-id>`;
   GRANT USE SCHEMA ON SCHEMA system.billing TO `<client-id>`;
   GRANT SELECT ON TABLE system.billing.usage TO `<client-id>`;
   GRANT SELECT ON TABLE system.billing.list_prices TO `<client-id>`;
   GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `<client-id>`;
   GRANT SELECT ON TABLE system.lakeflow.job_run_timeline TO `<client-id>`;
   GRANT SELECT ON TABLE system.lakeflow.jobs TO `<client-id>`;
   GRANT USE SCHEMA ON SCHEMA system.access TO `<client-id>`;
   GRANT SELECT ON TABLE system.access.audit TO `<client-id>`;

   system.compute, system.lakeflow, system.access may also need enabling for the metastore by an account or metastore admin.

5 Paste the client ID and secret into the connect form

1. In the Digital Tap dashboard, open Settings and click + Connect Platform. A new account sees the form as soon as it signs up.
2. Choose Databricks, then enter the Workspace URL, the Client ID, the Client secret and, for system-table analysis, the SQL warehouse ID.
3. Click Connect & Scan. Digital Tap exchanges the client ID and secret for a short-lived token and lists the workspace's clusters before it stores anything. If either step fails, nothing is stored and the form shows Databricks' reason.

6 Read the connect report

A successful connection shows what Digital Tap can read:

• each system table as readable, not readable (with the grant that fixes it), or not checked (with the reason);
• job visibility as all, partial or unknown. Unless it is all, Digital Tap cannot rule out a job running on a cluster, so every hibernate goes to a person for approval.

Connecting grants Digital Tap no automatic actions. A hibernate it proposes appears on the dashboard's Approvals page for an org owner or admin to decide.

7 Troubleshooting

The connection is refused

• Check the client ID and secret. If the secret was lost or has expired, generate a new one on the service principal's Secrets tab.
• Check that the service principal has the Workspace access entitlement. Without it the workspace refuses the first call, which lists clusters.
• Check that the workspace URL is the workspace's own address, beginning https://.

No clusters, or fewer than expected

• In a workspace with cluster visibility control, a service principal sees only clusters it holds a permission on. Grant CAN_ATTACH_TO or CAN_RESTART on each one.

System tables "not checked" or "not readable"

• Not checked: enter a SQL warehouse ID, grant CAN_USE on that warehouse, and enable the Databricks SQL access entitlement.
• Not readable: a metastore admin runs the grants in step 4. A workspace admin's grant on the system catalog is refused. A schema that is not enabled for the metastore reports SCHEMA_NOT_FOUND until it is.