data_quality_refreshes

Creates, updates, deletes, gets or lists a data_quality_refreshes resource.

Overview

Name	data_quality_refreshes
Type	Resource
Id	databricks_workspace.dataquality.data_quality_refreshes

Fields

The following fields are returned by SELECT queries:

get

Name	Datatype	Description
object_id	string	The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id
refresh_id	integer	Unique id of the refresh operation.
end_time_ms	integer	Time when the refresh ended (milliseconds since 1/1/1970 UTC).
message	string	An optional message to give insight into the current state of the refresh (e.g. FAILURE messages).
object_type	string	The type of the monitored object. Can be one of the following: `schema`or `table`.
start_time_ms	integer	Time when the refresh started (milliseconds since 1/1/1970 UTC).
state	string	The current state of the refresh. (MONITOR_REFRESH_STATE_CANCELED, MONITOR_REFRESH_STATE_FAILED, MONITOR_REFRESH_STATE_PENDING, MONITOR_REFRESH_STATE_RUNNING, MONITOR_REFRESH_STATE_SUCCESS, MONITOR_REFRESH_STATE_UNKNOWN)
trigger	string	What triggered the refresh. (MONITOR_REFRESH_TRIGGER_DATA_CHANGE, MONITOR_REFRESH_TRIGGER_MANUAL, MONITOR_REFRESH_TRIGGER_SCHEDULE, MONITOR_REFRESH_TRIGGER_UNKNOWN)

list

Name	Datatype	Description
object_id	string	The UUID of the request object. It is `schema_id` for `schema`, and `table_id` for `table`. Find the `schema_id` from either: 1. The [schema_id] of the `Schemas` resource. 2. In [Catalog Explorer] > select the `schema` > go to the `Details` tab > the `Schema ID` field. Find the `table_id` from either: 1. The [table_id] of the `Tables` resource. 2. In [Catalog Explorer] > select the `table` > go to the `Details` tab > the `Table ID` field. [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id
refresh_id	integer	Unique id of the refresh operation.
end_time_ms	integer	Time when the refresh ended (milliseconds since 1/1/1970 UTC).
message	string	An optional message to give insight into the current state of the refresh (e.g. FAILURE messages).
object_type	string	The type of the monitored object. Can be one of the following: `schema`or `table`.
start_time_ms	integer	Time when the refresh started (milliseconds since 1/1/1970 UTC).
state	string	The current state of the refresh. (MONITOR_REFRESH_STATE_CANCELED, MONITOR_REFRESH_STATE_FAILED, MONITOR_REFRESH_STATE_PENDING, MONITOR_REFRESH_STATE_RUNNING, MONITOR_REFRESH_STATE_SUCCESS, MONITOR_REFRESH_STATE_UNKNOWN)
trigger	string	What triggered the refresh. (MONITOR_REFRESH_TRIGGER_DATA_CHANGE, MONITOR_REFRESH_TRIGGER_MANUAL, MONITOR_REFRESH_TRIGGER_SCHEDULE, MONITOR_REFRESH_TRIGGER_UNKNOWN)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get	select	object_type, object_id, refresh_id, deployment_name		Get data quality monitor refresh. The call must be made in the same workspace as where the monitor was
list	select	object_type, object_id, deployment_name	page_size, page_token	List data quality monitor refreshes. The call must be made in the same workspace as where the monitor
create	insert	object_type, object_id, deployment_name, refresh		Creates a refresh. Currently only supported for the table object_type. The call must be made in
update	update	object_type, object_id, refresh_id, update_mask, deployment_name, refresh		(Unimplemented) Update a refresh
delete	delete	object_type, object_id, refresh_id, deployment_name		(Unimplemented) Delete a refresh
cancel	exec	object_type, object_id, refresh_id, deployment_name		Cancels a data quality monitor refresh. Currently only supported for the table object_type. The

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
deployment_name	string	The Databricks Workspace Deployment Name (default: dbc-abcd0123-a1bc)
object_id	string	The UUID of the request object. It is schema_id for schema, and table_id for table. Find the schema_id from either: 1. The [schema_id] of the Schemas resource. 2. In [Catalog Explorer] > select the schema > go to the Details tab > the Schema ID field. Find the table_id from either: 1. The [table_id] of the Tables resource. 2. In [Catalog Explorer] > select the table > go to the Details tab > the Table ID field. [Catalog Explorer]: https://docs.databricks.com/aws/en/catalog-explorer/ [schema_id]: https://docs.databricks.com/api/workspace/schemas/get#schema_id [table_id]: https://docs.databricks.com/api/workspace/tables/get#table_id
object_type	string	The type of the monitored object. Can be one of the following: schema or table.
refresh_id	integer	Unique id of the refresh operation.
update_mask	string	The field mask to specify which fields to update.
page_size	integer	:param page_token: str (optional)
page_token	string

SELECT examples

get

Get data quality monitor refresh. The call must be made in the same workspace as where the monitor was

SELECT
object_id,
refresh_id,
end_time_ms,
message,
object_type,
start_time_ms,
state,
trigger
FROM databricks_workspace.dataquality.data_quality_refreshes
WHERE object_type = '{{ object_type }}' -- required
AND object_id = '{{ object_id }}' -- required
AND refresh_id = '{{ refresh_id }}' -- required
AND deployment_name = '{{ deployment_name }}' -- required
;

list

List data quality monitor refreshes. The call must be made in the same workspace as where the monitor

SELECT
object_id,
refresh_id,
end_time_ms,
message,
object_type,
start_time_ms,
state,
trigger
FROM databricks_workspace.dataquality.data_quality_refreshes
WHERE object_type = '{{ object_type }}' -- required
AND object_id = '{{ object_id }}' -- required
AND deployment_name = '{{ deployment_name }}' -- required
AND page_size = '{{ page_size }}'
AND page_token = '{{ page_token }}'
;

INSERT examples

create

Creates a refresh. Currently only supported for the table object_type. The call must be made in

INSERT INTO databricks_workspace.dataquality.data_quality_refreshes (
refresh,
object_type,
object_id,
deployment_name
)
SELECT 
'{{ refresh }}' /* required */,
'{{ object_type }}',
'{{ object_id }}',
'{{ deployment_name }}'
RETURNING
object_id,
refresh_id,
end_time_ms,
message,
object_type,
start_time_ms,
state,
trigger
;

Manifest

# Description fields are for documentation purposes
- name: data_quality_refreshes
props:
  - name: object_type
    value: "{{ object_type }}"
    description: Required parameter for the data_quality_refreshes resource.
  - name: object_id
    value: "{{ object_id }}"
    description: Required parameter for the data_quality_refreshes resource.
  - name: deployment_name
    value: "{{ deployment_name }}"
    description: Required parameter for the data_quality_refreshes resource.
  - name: refresh
    description: |
      The refresh to create
    value:
      object_type: "{{ object_type }}"
      object_id: "{{ object_id }}"
      end_time_ms: {{ end_time_ms }}
      message: "{{ message }}"
      refresh_id: {{ refresh_id }}
      start_time_ms: {{ start_time_ms }}
      state: "{{ state }}"
      trigger: "{{ trigger }}"

UPDATE examples

update

(Unimplemented) Update a refresh

UPDATE databricks_workspace.dataquality.data_quality_refreshes
SET 
refresh = '{{ refresh }}'
WHERE 
object_type = '{{ object_type }}' --required
AND object_id = '{{ object_id }}' --required
AND refresh_id = '{{ refresh_id }}' --required
AND update_mask = '{{ update_mask }}' --required
AND deployment_name = '{{ deployment_name }}' --required
AND refresh = '{{ refresh }}' --required
RETURNING
object_id,
refresh_id,
end_time_ms,
message,
object_type,
start_time_ms,
state,
trigger;

DELETE examples

delete

(Unimplemented) Delete a refresh

DELETE FROM databricks_workspace.dataquality.data_quality_refreshes
WHERE object_type = '{{ object_type }}' --required
AND object_id = '{{ object_id }}' --required
AND refresh_id = '{{ refresh_id }}' --required
AND deployment_name = '{{ deployment_name }}' --required
;

Lifecycle Methods

cancel

Cancels a data quality monitor refresh. Currently only supported for the table object_type. The

EXEC databricks_workspace.dataquality.data_quality_refreshes.cancel 
@object_type='{{ object_type }}' --required, 
@object_id='{{ object_id }}' --required, 
@refresh_id='{{ refresh_id }}' --required, 
@deployment_name='{{ deployment_name }}' --required
;