databricks_workspace

Manage clusters, jobs, notebooks, MLflow and other Databricks workspace resources.

info

For Databricks account operations use the databricks_account provider.

Provider Summary

total services: 36
total resources: 372
source project: stackql-provider-databricks

Installation

REGISTRY PULL databricks_workspace;

Authentication

To use the databricks_workspace provider, you can authenticate using one of the following methods:

OAuth2 (Service Principal) [Default]

Set the following environment variables:

DATABRICKS_ACCOUNT_ID - a uuid representing your Databricks account id, you can get this from the Databricks UI (see Locate your account id)
DATABRICKS_CLIENT_ID - obtained after creating a service principal through the Databricks UI (see Authenticate access to Databricks with a service principal using OAuth)
DATABRICKS_CLIENT_SECRET - obtained after creating a service principal secret through the Databricks UI, using the "Generate Secret" function (see Authenticate access to Databricks with a service principal using OAuth)

These are the same variables that Terraform, the Databricks SDKs, and CLI use.

Cloud selection (AWS, Azure, GCP)

The provider works against Databricks workspaces on AWS, Azure and Google Cloud with the same statements and the same three credential variables. It defaults to Databricks on AWS; one more variable selects another cloud, and both the OAuth token endpoint and the workspace host domain follow it:

DATABRICKS_CLOUD - aws (default), azure or gcp
DATABRICKS_ACCOUNT_HOST - optional, a bare account console host (no scheme) that issues the tokens, for accounts that are not on the standard hosts; overrides DATABRICKS_CLOUD
DATABRICKS_DEPLOYMENT_NAME - optional, defaults the deployment_name predicate for the session

Cloud	`DATABRICKS_CLOUD`	Workspace URL	`deployment_name`	Token endpoint host
AWS	`aws` (default)	`https://dbc-abcd0123-a1bc.cloud.databricks.com`	`dbc-abcd0123-a1bc`	`accounts.cloud.databricks.com`
Azure	`azure`	`https://adb-1234567890123456.7.azuredatabricks.net`	`adb-1234567890123456.7`	`accounts.azuredatabricks.net`
GCP	`gcp`	`https://1234567890123456.7.gcp.databricks.com`	`1234567890123456.7`	`accounts.gcp.databricks.com`

deployment_name is the host prefix of the workspace URL on every cloud (everything before .cloud.databricks.com, .azuredatabricks.net or .gcp.databricks.com). The databricks_account provider lists it for every workspace in the account:

SELECT workspace_name, deployment_name, workspace_status
FROM databricks_account.provisioning.workspaces
WHERE account_id = '<account_id>';

Cloud selection needs stackql v1.0.888 or later. One cloud per process: the token endpoint follows the environment, so run a separate stackql process (or shell) for each cloud you work with.

# Azure Databricks workspace
export DATABRICKS_CLOUD=azure
export DATABRICKS_ACCOUNT_ID=...
export DATABRICKS_CLIENT_ID=...
export DATABRICKS_CLIENT_SECRET=...
export DATABRICKS_DEPLOYMENT_NAME=adb-1234567890123456.7
stackql exec "SELECT id, name, state, cluster_size, auto_stop_mins FROM databricks_workspace.sql.warehouses"

# Databricks on Google Cloud workspace (PowerShell)
$env:DATABRICKS_CLOUD = "gcp"
$env:DATABRICKS_ACCOUNT_ID = "..."
$env:DATABRICKS_CLIENT_ID = "..."
$env:DATABRICKS_CLIENT_SECRET = "..."
stackql.exe exec "SELECT * FROM databricks_workspace.iam.current_user WHERE deployment_name = '1234567890123456.7'"

With no cloud variables set, queries behave exactly as before: AWS hosts and the AWS token endpoint.

Service principal by cloud

On every cloud the credential is a service principal with a Databricks OAuth secret, created in the account console under User management > Service principals (add the principal, then Generate secret). Cloud identity tokens (AWS IAM, Microsoft Entra ID, Google service accounts) are not used by the provider.

Cloud	Account console	Notes
AWS	accounts.cloud.databricks.com	Databricks-managed service principal (see OAuth M2M on AWS)
Azure	accounts.azuredatabricks.net	Sign in with the Microsoft Entra ID identity that administers the Databricks account. A Databricks-managed service principal, or an Entra ID application registered as a service principal, both work when the OAuth secret is generated in the Databricks account console (see OAuth M2M on Azure)
GCP	accounts.gcp.databricks.com	Databricks-managed service principal; the Google service account that Databricks itself uses is not the principal (see OAuth M2M on GCP)

The service principal needs access to each workspace it queries. Add it to the workspace from the account console (Workspaces > Permissions), or with the databricks_account provider, as a user, or as an admin for admin-only resources such as workspace configuration (settings.vw_all_settings) and other users' query history:

REPLACE databricks_account.iam.workspace_assignment
SET permissions = '["USER"]'
WHERE account_id = '<account_id>'
AND workspace_id = '<workspace_id>'
AND principal_id = '<service_principal_id>';

What differs by cloud

The workspace APIs are the same on all three clouds, so resources, columns and statements do not change. Resources tied to one cloud's primitives exist only there (for example AWS instance profiles under compute.instance_profiles, or the AWS availability zones from compute.clusters list_zones); node types and Spark versions are cloud-specific values. Where a workspace's cloud or region does not offer a feature (serverless, Lakebase, some AI services), Databricks answers with an error rather than an empty list, and the provider surfaces that error as is.
Workspace-level settings and compute policies that reference cloud storage or identities take the cloud's own identifiers (S3 paths and IAM roles on AWS, ABFSS paths and managed identities on Azure, GCS paths and service accounts on GCP); the provider passes them through verbatim.

Personal Access Token (Bearer)

Alternatively, set DATABRICKS_TOKEN to a Databricks personal access token (see Databricks personal access tokens), then supply the auth config when starting the shell:

export DATABRICKS_TOKEN=xxx

# Linux/Mac
AUTH='{ "databricks_workspace": { "type": "bearer", "credentialsenvvar": "DATABRICKS_TOKEN" }}'
./stackql shell --auth="${AUTH}"

# PowerShell
$Auth = "{ 'databricks_workspace': { 'type': 'bearer', 'credentialsenvvar': 'DATABRICKS_TOKEN' }}"
stackql.exe shell --auth=$Auth

Example Queries

Try the following queries using stackql shell, or run them from a script or CI pipeline with stackql exec.

Unity Catalog inventory

Catalogs in a workspace, with ownership and audit fields:

SELECT
  full_name,
  catalog_type,
  owner,
  comment,
  datetime(created_at/1000, 'unixepoch') AS created,
  created_by
FROM databricks_workspace.catalog.catalogs
WHERE deployment_name = '<deployment_name>';

Walk the catalog hierarchy - schemas and tables in a catalog:

SELECT full_name, table_type, data_source_format, owner
FROM databricks_workspace.catalog.tables
WHERE catalog_name = 'main'
AND schema_name = 'default'
AND deployment_name = '<deployment_name>';

SQL statement execution - the full lifecycle

The Databricks SQL Statement Execution API maps naturally to SQL verbs - INSERT submits a statement, SELECT polls it, DELETE cancels it:

/* submit a statement to a SQL warehouse */
INSERT INTO databricks_workspace.sql.statement_execution (
  statement,
  warehouse_id,
  wait_timeout,
  deployment_name
)
SELECT
  'SELECT * FROM samples.nyctaxi.trips LIMIT 100',
  '<warehouse_id>',
  '0s',
  '<deployment_name>';

/* poll for status and results */
SELECT status, manifest, result
FROM databricks_workspace.sql.statement_execution
WHERE statement_id = '<statement_id>'
AND deployment_name = '<deployment_name>';

/* cancel a running statement */
DELETE FROM databricks_workspace.sql.statement_execution
WHERE statement_id = '<statement_id>'
AND deployment_name = '<deployment_name>';

Recent query history across SQL warehouses and serverless compute:

SELECT query_id, status, query_text, user_name
FROM databricks_workspace.sql.query_history
WHERE deployment_name = '<deployment_name>';

Workspace identity

Users, their groups and entitlements, flattened with built-in views:

SELECT * FROM databricks_workspace.iam.vw_users WHERE deployment_name = '<deployment_name>';
SELECT * FROM databricks_workspace.iam.vw_user_groups WHERE deployment_name = '<deployment_name>';
SELECT * FROM databricks_workspace.iam.vw_user_entitlements WHERE deployment_name = '<deployment_name>';
SELECT * FROM databricks_workspace.iam.vw_group_members WHERE deployment_name = '<deployment_name>';

Compute estate

Clusters with their state, node types and autotermination settings:

SELECT
  cluster_id,
  cluster_name,
  state,
  node_type_id,
  autotermination_minutes,
  spark_version
FROM databricks_workspace.compute.clusters
WHERE deployment_name = '<deployment_name>';

SQL warehouses, sized and by state:

SELECT id, name, state, cluster_size, num_clusters, auto_stop_mins
FROM databricks_workspace.sql.warehouses
WHERE deployment_name = '<deployment_name>';

Workspace settings

All workspace admin settings as key/value pairs, using a built-in view:

SELECT key, value
FROM databricks_workspace.settings.vw_all_settings
WHERE deployment_name = '<deployment_name>';

Installation​

Authentication​

OAuth2 (Service Principal) [Default]​

Cloud selection (AWS, Azure, GCP)​

Service principal by cloud​

What differs by cloud​

Personal Access Token (Bearer)​

Example Queries​

Unity Catalog inventory​

SQL statement execution - the full lifecycle​

Workspace identity​

Compute estate​

Workspace settings​

Services​