databricks_workspace
Manage clusters, jobs, notebooks, MLflow and other Databricks workspace resources.
For Databricks account operations use the databricks_account provider.
total services: 29 total resources: 342
See also: [SHOW] [DESCRIBE] [REGISTRY]
Installation
REGISTRY PULL databricks_workspace;
Authentication
To use the databricks_workspace provider, you can authenticate using one of the following methods:
OAuth2 (Service Principal) [Default]
Set the following environment variables:
DATABRICKS_ACCOUNT_ID- a uuid representing your Databricks account id, you can get this from the Databricks UI (see Locate your account id)DATABRICKS_CLIENT_ID- obtained after creating a service principal through the Databricks UI (see Authenticate access to Databricks with a service principal using OAuth)DATABRICKS_CLIENT_SECRET- obtained after creating a service principal secret through the Databricks UI, using the "Generate Secret" function (see Authenticate access to Databricks with a service principal using OAuth)
These are the same variables that Terraform, the Databricks SDKs, and CLI use.
Personal Access Token (Bearer)
Alternatively, set DATABRICKS_TOKEN to a Databricks personal access token (see Databricks personal access tokens), then supply the auth config when starting the shell:
export DATABRICKS_TOKEN=xxx
# Linux/Mac
AUTH='{ "databricks_workspace": { "type": "bearer", "credentialsenvvar": "DATABRICKS_TOKEN" }}'
./stackql shell --auth="${AUTH}"
# PowerShell
$Auth = "{ 'databricks_workspace': { 'type': 'bearer', 'credentialsenvvar': 'DATABRICKS_TOKEN' }}"
stackql.exe shell --auth=$Auth
Unity Catalog inventory
Catalogs in a workspace, with ownership and audit fields:
SELECT
full_name,
catalog_type,
owner,
comment,
datetime(created_at/1000, 'unixepoch') AS created,
created_by
FROM databricks_workspace.catalog.catalogs
WHERE deployment_name = '<deployment_name>';
Walk the catalog hierarchy - schemas and tables in a catalog:
SELECT full_name, table_type, data_source_format, owner
FROM databricks_workspace.catalog.tables
WHERE catalog_name = 'main'
AND schema_name = 'default'
AND deployment_name = '<deployment_name>';
SQL statement execution - the full lifecycle
The Databricks SQL Statement Execution API maps naturally to SQL verbs - INSERT submits a statement, SELECT polls it, DELETE cancels it:
/* submit a statement to a SQL warehouse */
INSERT INTO databricks_workspace.sql.statement_execution (
statement,
warehouse_id,
wait_timeout,
deployment_name
)
SELECT
'SELECT * FROM samples.nyctaxi.trips LIMIT 100',
'<warehouse_id>',
'0s',
'<deployment_name>';
/* poll for status and results */
SELECT status, manifest, result
FROM databricks_workspace.sql.statement_execution
WHERE statement_id = '<statement_id>'
AND deployment_name = '<deployment_name>';
/* cancel a running statement */
DELETE FROM databricks_workspace.sql.statement_execution
WHERE statement_id = '<statement_id>'
AND deployment_name = '<deployment_name>';
Recent query history across SQL warehouses and serverless compute:
SELECT query_id, status, query_text, user_name
FROM databricks_workspace.sql.query_history
WHERE deployment_name = '<deployment_name>';
Workspace identity
Users, their groups and entitlements, flattened with built-in views:
SELECT * FROM databricks_workspace.iam.vw_users WHERE deployment_name = '<deployment_name>';
SELECT * FROM databricks_workspace.iam.vw_user_groups WHERE deployment_name = '<deployment_name>';
SELECT * FROM databricks_workspace.iam.vw_user_entitlements WHERE deployment_name = '<deployment_name>';
SELECT * FROM databricks_workspace.iam.vw_group_members WHERE deployment_name = '<deployment_name>';
Compute estate
Clusters with their state, node types and autotermination settings:
SELECT
cluster_id,
cluster_name,
state,
node_type_id,
autotermination_minutes,
spark_version
FROM databricks_workspace.compute.clusters
WHERE deployment_name = '<deployment_name>';
SQL warehouses, sized and by state:
SELECT id, name, state, cluster_size, num_clusters, auto_stop_mins
FROM databricks_workspace.sql.warehouses
WHERE deployment_name = '<deployment_name>';
Workspace settings
All workspace admin settings as key/value pairs, using a built-in view:
SELECT key, value
FROM databricks_workspace.settings.vw_all_settings
WHERE deployment_name = '<deployment_name>';