Skip to content
KataraKatara documentation

Summarise how judging against a rubric has gone and what it cost

GET
/katara/v1/orgs/{org_id}/environments/{environment_id}/evaluation-rubrics/{asset_id}/summary
curl --request GET \
  --url https://api.katara.com/katara/v1/orgs/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/environments/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/evaluation-rubrics/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/summary \
  --cookie katara_session=<katara_session>

Available when the evaluation material service is configured. Needs evaluation read access. Counts the assessments judged against the rubric since the given time, thirty days ago by default and at most a year: how many were scored, not fully scorable or still pending, each criterion’s passes, failures and mean score, the judge’s charges and tokens as the scores record them, and how many live runs online sampling considered, selected and had judged for settings that use the rubric. It reads what the scores record and changes nothing.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
org_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
environment_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
asset_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000

Header Parameters

Section titled “Header Parameters”
X-Katara-Account-ID
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/

Optional mounted-account binding for cookie reads. A supplied different account returns account_changed.

Query Parameters

Section titled “Query Parameters”
since
string format: date-time

RFC 3339 time to count from.

Responses

Section titled “Responses”

200

Section titled “200”

The summary.

Media typeapplication/json

How judging against a rubric has gone since a time and what the judge cost, read from the recorded scores.

object
rubric_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
since
required
string format: date-time
assessments
required
integer
<= 9007199254740991
scored
required
integer
<= 9007199254740991
not_fully_scorable
required
integer
<= 9007199254740991
pending
required
integer
<= 9007199254740991
charged
required
object
ledger_kind
required
Allowed values: simulation testnet real
asset
required
Allowed values: USD_MICRO CREDIT_ATOMIC USDC_ATOMIC
amount
required
string
/^(0|[1-9][0-9]{0,37})$/
scale
required
Allowed values: 6 9
chain_id
Allowed values: 0 43113 43114
token
string
/^0x[0-9a-f]{40}$/
input_tokens
required
integer
<= 9007199254740991
output_tokens
required
integer
<= 9007199254740991
criteria
required
Array<object>
<= 32 items

One criterion’s results. mean_score is null while nothing has been scored against it.

object
criterion_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
scored
required
integer
<= 9007199254740991
passed
required
integer
<= 9007199254740991
failed
required
integer
<= 9007199254740991
not_scorable
required
integer
<= 9007199254740991
mean_score
required
number | null
<= 100
online
required
object
considered
required
integer
<= 9007199254740991
selected
required
integer
<= 9007199254740991
judged
required
integer
<= 9007199254740991
Example
{
  "charged": {
    "ledger_kind": "simulation",
    "asset": "USD_MICRO",
    "scale": 6,
    "chain_id": 0
  }
}

default

Section titled “default”

Request rejected or unavailable.

Media typeapplication/json
object
error
required
object
message
required
string
>= 1 characters <= 500 characters
type
required
string
>= 1 characters <= 100 characters
code
required

Stable; branch on it. Any operation can answer invalid_request, with param naming the refused field when there is one; human_session_required, when an API key, service account or workload calls an operation only a signed-in person may perform; reauthentication_required, when a person’s sign-in is too old for the operation; permission_denied; route_not_found and method_not_allowed, when no operation matches the path or method; and feature_unavailable, when the operation’s service is not enabled on this deployment.

string
>= 1 characters <= 100 characters
param
required

The request field a refusal is about, such as the agent_id or run_id query parameter; null when the refusal is not about one field.

string | null
request_id
required
string
>= 1 characters <= 100 characters
upstream_status
required
integer | null
action
Allowed values: sign_in reauthenticate request_access review fund retry reconcile refresh support
operation_id
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
snapshot_ref
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
Example
{
  "error": {
    "action": "sign_in"
  }
}