genie_eval_runs

Creates, updates, deletes, gets or lists a genie_eval_runs resource.

Overview

Name	genie_eval_runs
Type	Resource
Id	databricks_workspace.dashboards.genie_eval_runs

Fields

The following fields are returned by SELECT queries:

get

Name	Datatype	Description
eval_run_id	string
created_timestamp	integer	Timestamp when the evaluation run was created (milliseconds since epoch).
eval_run_status	string	Current status of the evaluation run. (DONE, EVALUATION_CANCELLED, EVALUATION_FAILED, EVALUATION_TIMEOUT, NOT_STARTED, RUNNING)
last_updated_timestamp	integer	Timestamp when the evaluation run was last updated (milliseconds since epoch).
num_correct	integer	Number of questions answered correctly.
num_done	integer	Number of questions that have been completed.
num_needs_review	integer	Number of questions that need manual review.
num_questions	integer	Total number of questions in the evaluation run.
run_by_user	integer	User ID who initiated the evaluation run.

list

Name	Datatype	Description
eval_run_id	string
created_timestamp	integer	Timestamp when the evaluation run was created (milliseconds since epoch).
eval_run_status	string	Current status of the evaluation run. (DONE, EVALUATION_CANCELLED, EVALUATION_FAILED, EVALUATION_TIMEOUT, NOT_STARTED, RUNNING)
last_updated_timestamp	integer	Timestamp when the evaluation run was last updated (milliseconds since epoch).
num_correct	integer	Number of questions answered correctly.
num_done	integer	Number of questions that have been completed.
num_needs_review	integer	Number of questions that need manual review.
num_questions	integer	Total number of questions in the evaluation run.
run_by_user	integer	User ID who initiated the evaluation run.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get	select	space_id, eval_run_id, deployment_name		Get evaluation run details.
list	select	space_id, deployment_name	page_size, page_token	Lists all evaluation runs in a space.
create	insert	space_id, deployment_name		Create and run evaluations for multiple benchmark questions in a Genie space.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
deployment_name	string	The Databricks Workspace Deployment Name (default: dbc-abcd0123-a1bc)
eval_run_id	string
space_id	string	The ID associated with the Genie space where the evaluations will be executed.
page_size	integer	Maximum number of evaluation runs to return per page
page_token	string	Token to get the next page of results

SELECT examples

get

Get evaluation run details.

SELECT
eval_run_id,
created_timestamp,
eval_run_status,
last_updated_timestamp,
num_correct,
num_done,
num_needs_review,
num_questions,
run_by_user
FROM databricks_workspace.dashboards.genie_eval_runs
WHERE space_id = '{{ space_id }}' -- required
AND eval_run_id = '{{ eval_run_id }}' -- required
AND deployment_name = '{{ deployment_name }}' -- required
;

list

Lists all evaluation runs in a space.

SELECT
eval_run_id,
created_timestamp,
eval_run_status,
last_updated_timestamp,
num_correct,
num_done,
num_needs_review,
num_questions,
run_by_user
FROM databricks_workspace.dashboards.genie_eval_runs
WHERE space_id = '{{ space_id }}' -- required
AND deployment_name = '{{ deployment_name }}' -- required
AND page_size = '{{ page_size }}'
AND page_token = '{{ page_token }}'
;

INSERT examples

create

Create and run evaluations for multiple benchmark questions in a Genie space.

INSERT INTO databricks_workspace.dashboards.genie_eval_runs (
benchmark_question_ids,
space_id,
deployment_name
)
SELECT 
'{{ benchmark_question_ids }}',
'{{ space_id }}',
'{{ deployment_name }}'
RETURNING
eval_run_id,
created_timestamp,
eval_run_status,
last_updated_timestamp,
num_correct,
num_done,
num_needs_review,
num_questions,
run_by_user
;

Manifest

# Description fields are for documentation purposes
- name: genie_eval_runs
props:
  - name: space_id
    value: "{{ space_id }}"
    description: Required parameter for the genie_eval_runs resource.
  - name: deployment_name
    value: "{{ deployment_name }}"
    description: Required parameter for the genie_eval_runs resource.
  - name: benchmark_question_ids
    value:
      - "{{ benchmark_question_ids }}"
    description: |
      List of benchmark question IDs to evaluate. These questions must exist in the specified Genie space. If none are specified, then all benchmark questions are evaluated.