genie_eval_results

Creates, updates, deletes, gets or lists a genie_eval_results resource.

Overview

Name	genie_eval_results
Type	Resource
Id	databricks_workspace.dashboards.genie_eval_results

Fields

The following fields are returned by SELECT queries:

get_details

Name	Datatype	Description
benchmark_question_id	string	The ID of the benchmark question that was evaluated.
result_id	string	The unique identifier for the evaluation result.
space_id	string	The ID of the space the evaluation result belongs to.
▶actual_response	array	The actual response generated by Genie.
assessment	string	Assessment of the evaluation result: good, bad, or needs review (BAD, GOOD, NEEDS_REVIEW)
assessment_reasons	array	Reasons for the assessment score. Assessment reasons describe why a Genie response was scored as BAD. Deterministic values (compared against the ground truth result): - EMPTY_RESULT: Genie's generated SQL results were empty for this benchmark question. - RESULT_MISSING_ROWS: Genie's generated SQL response is missing rows from the provided ground truth SQL. - RESULT_EXTRA_ROWS: Genie's generated SQL response has more rows than the provided ground truth SQL. - RESULT_MISSING_COLUMNS: Genie's generated SQL response is missing columns from the provided ground truth SQL. - RESULT_EXTRA_COLUMNS: Genie's generated SQL response has more columns than the provided ground truth SQL. - SINGLE_CELL_DIFFERENCE: Single value result was produced but differs from ground truth result. - EMPTY_GOOD_SQL: The benchmark SQL returned an empty result. - COLUMN_TYPE_DIFFERENCE: The values between the results match but the column type is different. LLM judge ratings explain the factors driving BAD results: - LLM_JUDGE_MISSING_OR_INCORRECT_FILTER: Genie's generated SQL is missing a WHERE clause condition or has incorrect filter logic that excludes/includes wrong data. - LLM_JUDGE_INCOMPLETE_OR_PARTIAL_OUTPUT: Genie's generated SQL returns only some of the requested data or columns, missing parts of what the ground truth SQL returns. - LLM_JUDGE_MISINTERPRETATION_OF_USER_REQUEST: Genie's generated SQL fundamentally misunderstands what the user is asking for, addressing the wrong question or goal. - LLM_JUDGE_INSTRUCTION_COMPLIANCE_OR_MISSING_BUSINESS_LOGIC: Genie's generated SQL fails to apply specified instructions or business logic that should be followed. - LLM_JUDGE_INCORRECT_METRIC_CALCULATION: Genie's generated SQL uses incorrect logic or makes wrong assumptions when calculating metrics. - LLM_JUDGE_INCORRECT_TABLE_OR_FIELD_USAGE: Genie's generated SQL references wrong tables, columns, or uses fields that don't match the ground truth SQL's intent. - LLM_JUDGE_INCORRECT_FUNCTION_USAGE: Genie's generated SQL uses SQL functions incorrectly or inappropriately (wrong parameters, wrong function for the task, etc.). - LLM_JUDGE_MISSING_OR_INCORRECT_JOIN: Genie's generated SQL is missing necessary joins between tables or has incorrect join conditions/types that produce wrong results. - LLM_JUDGE_MISSING_OR_INCORRECT_AGGREGATION: Genie's generated SQL is missing GROUP BY clauses or has incorrect grouping that doesn't match the requested aggregation level. - LLM_JUDGE_FORMATTING_ERROR: Genie's generated SQL output has incorrect formatting, ordering (ORDER BY), or presentation issues that don't match expectations. - LLM_JUDGE_OTHER: LLM judge identified an error that doesn't fall into other categories. Deprecated LLM judge values (kept for backward compatibility, do not use): - LLM_JUDGE_MISSING_JOIN (deprecated) - LLM_JUDGE_WRONG_FILTER (deprecated) - LLM_JUDGE_WRONG_AGGREGATION (deprecated) - LLM_JUDGE_WRONG_COLUMNS (deprecated) - LLM_JUDGE_SYNTAX_ERROR (deprecated) - LLM_JUDGE_SEMANTIC_ERROR (deprecated)
eval_run_status	string	Current status of the evaluation run. (DONE, EVALUATION_CANCELLED, EVALUATION_FAILED, EVALUATION_TIMEOUT, NOT_STARTED, RUNNING)
▶expected_response	array	The expected responses from the benchmark.
manual_assessment	boolean	Whether this evaluation was manually assessed.

list

Name	Datatype	Description
benchmark_question_id	string	The ID of the benchmark question that was evaluated.
result_id	string	Unique identifier for this evaluation result.
space_id	string	The ID of the space the evaluation result belongs to.
benchmark_answer	string	Stored snapshot of original benchmark answer text.
created_by_user	integer	User ID who created evaluation result.
question	string	Stored snapshot of original benchmark question text.
status	string	Current status of this evaluation result. (DONE, EVALUATION_CANCELLED, EVALUATION_FAILED, EVALUATION_TIMEOUT, NOT_STARTED, RUNNING)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_details	select	space_id, eval_run_id, result_id, deployment_name		Get details for evaluation results.
list	select	space_id, eval_run_id, deployment_name	page_size, page_token	List evaluation results for a specific evaluation run.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
deployment_name	string	The Databricks Workspace Deployment Name (default: dbc-abcd0123-a1bc)
eval_run_id	string	The unique identifier for the evaluation run.
result_id	string	The unique identifier for the evaluation result.
space_id	string	The ID associated with the Genie space where the evaluation run is located.
page_size	integer	Maximum number of eval results to return per page.
page_token	string	Opaque token to retrieve the next page of results.

SELECT examples

get_details

Get details for evaluation results.

SELECT
benchmark_question_id,
result_id,
space_id,
actual_response,
assessment,
assessment_reasons,
eval_run_status,
expected_response,
manual_assessment
FROM databricks_workspace.dashboards.genie_eval_results
WHERE space_id = '{{ space_id }}' -- required
AND eval_run_id = '{{ eval_run_id }}' -- required
AND result_id = '{{ result_id }}' -- required
AND deployment_name = '{{ deployment_name }}' -- required
;

list

List evaluation results for a specific evaluation run.

SELECT
benchmark_question_id,
result_id,
space_id,
benchmark_answer,
created_by_user,
question,
status
FROM databricks_workspace.dashboards.genie_eval_results
WHERE space_id = '{{ space_id }}' -- required
AND eval_run_id = '{{ eval_run_id }}' -- required
AND deployment_name = '{{ deployment_name }}' -- required
AND page_size = '{{ page_size }}'
AND page_token = '{{ page_token }}'
;