Skip to main content

clusters

Creates, updates, deletes, gets or lists a clusters resource.

Overview​

Nameclusters
TypeResource
Iddatabricks_workspace.compute.clusters

Fields​

The following fields are returned by SELECT queries:

NameDatatypeDescription
▶checksarrayList of individual checks (maps to UI rows).
diagnostics_statusstringOverall run status (PASSED / FAILED / NOT_RUN). FAILED if any individual check fails. (DIAGNOSTICS_STATUS_FAILED, DIAGNOSTICS_STATUS_NOT_RUN, DIAGNOSTICS_STATUS_PASSED)

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
clusters_get_diagnosticselectname, deployment_nameReturns the most recent cluster diagnostics result for a cluster.
getselectcluster_id, deployment_nameRetrieves the information for a cluster given its identifier. Clusters can be described while they are
listselectdeployment_namefilter_by, page_size, page_token, sort_byReturn information about all pinned and active clusters, and all clusters terminated within the last
createinsertdeployment_name, spark_versionCreates a new Spark cluster. This method will acquire new instances from the cloud provider if
change_ownerexecdeployment_name, cluster_id, owner_usernameChange the owner of the cluster. You must be an admin and the cluster must be terminated to perform
deleteexecdeployment_name, cluster_idTerminates the Spark cluster with the specified ID. The cluster is removed asynchronously. Once the
editexecdeployment_name, cluster_id, spark_versionUpdates the configuration of a cluster to match the provided attributes and size. A cluster can be
eventsexecdeployment_name, cluster_idRetrieves a list of events about the activity of a cluster. This API is paginated. If there are more
permanent_deleteexecdeployment_name, cluster_idPermanently deletes a Spark cluster. This cluster is terminated and resources are asynchronously
pinexecdeployment_name, cluster_idPinning a cluster ensures that the cluster will always be returned by the ListClusters API. Pinning a
resizeexecdeployment_name, cluster_idResizes a cluster to have a desired number of workers. This will fail unless the cluster is in a
restartexecdeployment_name, cluster_idRestarts a Spark cluster with the supplied ID. If the cluster is not currently in a RUNNING state,
startexecdeployment_name, cluster_idStarts a terminated Spark cluster with the supplied ID. This works similar to createCluster
unpinexecdeployment_name, cluster_idUnpinning a cluster will allow the cluster to eventually be removed from the ListClusters API.
updateexecdeployment_name, cluster_id, update_maskUpdates the configuration of a cluster to match the partial set of attributes and size. Denote which

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
cluster_idstringThe cluster about which to retrieve information.
deployment_namestringThe Databricks Workspace Deployment Name (default: dbc-abcd0123-a1bc)
namestringThe resource name of the cluster whose diagnostics to retrieve. Format: clusters/{cluster_id}
filter_byobjectFilters to apply to the list of clusters.
page_sizeintegerUse this field to specify the maximum number of results to be returned by the server. The server may further constrain the maximum number of results returned in a single page.
page_tokenstringUse next_page_token or prev_page_token returned from the previous request to list the next or previous page of clusters respectively.
sort_byobjectSort the list of clusters by a specific criteria.

SELECT examples​

Returns the most recent cluster diagnostics result for a cluster.

SELECT
checks,
diagnostics_status
FROM databricks_workspace.compute.clusters
WHERE name = '{{ name }}' -- required
AND deployment_name = '{{ deployment_name }}' -- required
;

INSERT examples​

Creates a new Spark cluster. This method will acquire new instances from the cloud provider if

INSERT INTO databricks_workspace.compute.clusters (
spark_version,
apply_policy_default_values,
autoscale,
autotermination_minutes,
aws_attributes,
azure_attributes,
clone_from,
cluster_log_conf,
cluster_name,
custom_tags,
data_security_mode,
dependency_mode,
docker_image,
driver_instance_pool_id,
driver_node_type_flexibility,
driver_node_type_id,
enable_elastic_disk,
enable_local_disk_encryption,
gcp_attributes,
init_scripts,
instance_pool_id,
is_single_node,
kind,
node_type_id,
num_workers,
policy_id,
remote_disk_throughput,
runtime_engine,
single_user_name,
spark_conf,
spark_env_vars,
ssh_public_keys,
total_initial_remote_disk_size,
use_ml_runtime,
worker_node_type_flexibility,
workload_type,
deployment_name
)
SELECT 
'{{ spark_version }}' /* required */,
{{ apply_policy_default_values }},
'{{ autoscale }}',
{{ autotermination_minutes }},
'{{ aws_attributes }}',
'{{ azure_attributes }}',
'{{ clone_from }}',
'{{ cluster_log_conf }}',
'{{ cluster_name }}',
'{{ custom_tags }}',
'{{ data_security_mode }}',
'{{ dependency_mode }}',
'{{ docker_image }}',
'{{ driver_instance_pool_id }}',
'{{ driver_node_type_flexibility }}',
'{{ driver_node_type_id }}',
{{ enable_elastic_disk }},
{{ enable_local_disk_encryption }},
'{{ gcp_attributes }}',
'{{ init_scripts }}',
'{{ instance_pool_id }}',
{{ is_single_node }},
'{{ kind }}',
'{{ node_type_id }}',
{{ num_workers }},
'{{ policy_id }}',
{{ remote_disk_throughput }},
'{{ runtime_engine }}',
'{{ single_user_name }}',
'{{ spark_conf }}',
'{{ spark_env_vars }}',
'{{ ssh_public_keys }}',
{{ total_initial_remote_disk_size }},
{{ use_ml_runtime }},
'{{ worker_node_type_flexibility }}',
'{{ workload_type }}',
'{{ deployment_name }}'
RETURNING
cluster_id,
driver_instance_pool_id,
driver_node_type_id,
instance_pool_id,
node_type_id,
policy_id,
spark_context_id,
cluster_name,
creator_user_name,
single_user_name,
autoscale,
autotermination_minutes,
aws_attributes,
azure_attributes,
cluster_cores,
cluster_log_conf,
cluster_log_status,
cluster_memory_mb,
cluster_source,
custom_tags,
data_security_mode,
default_tags,
dependency_mode,
docker_image,
driver,
driver_node_type_flexibility,
enable_elastic_disk,
enable_local_disk_encryption,
executors,
gcp_attributes,
init_scripts,
is_single_node,
jdbc_port,
kind,
last_restarted_time,
last_state_loss_time,
num_workers,
remote_disk_throughput,
runtime_engine,
spark_conf,
spark_env_vars,
spark_version,
spec,
ssh_public_keys,
start_time,
state,
state_message,
terminated_time,
termination_reason,
total_initial_remote_disk_size,
use_ml_runtime,
worker_node_type_flexibility,
workload_type
;

Lifecycle Methods​

Change the owner of the cluster. You must be an admin and the cluster must be terminated to perform

EXEC databricks_workspace.compute.clusters.change_owner 
@deployment_name='{{ deployment_name }}' --required 
@@json=
'{
"cluster_id": "{{ cluster_id }}", 
"owner_username": "{{ owner_username }}"
}'
;