Skip to content
KataraKatara documentation

Record whether sampled live runs are judged for an agent

POST
/katara/v1/orgs/{org_id}/environments/{environment_id}/online-evaluations/{agent_id}
curl --request POST \
  --url https://api.katara.com/katara/v1/orgs/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/environments/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0/online-evaluations/2489E9AD-2EE2-8E00-8EC9-32D5F69181C0 \
  --header 'Content-Type: application/json' \
  --header 'Idempotency-Key: 2489E9AD-2EE2-8E00-8EC9-32D5F69181C0' \
  --header 'X-Katara-CSRF: 1' \
  --cookie katara_session=<katara_session> \
  --data '{ "agent_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "enabled": true, "sample_rate_ppm": 1, "rubric": { "asset_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "version_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "integrity": "example" }, "model_id": "example", "judge_agent_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "budget_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "consent_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "daily_cap": 1, "approval_expires_at": "2026-04-15T12:00:00Z", "output_target": { "store_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "store_version": 1, "key_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0", "key_version": 1, "retention_seconds": 1 } }'

Available when the evaluation material service is configured. Records a version of the sampling settings for one agent: whether live runs are judged, how often, against which reviewed rubric, by which model, and out of which budget and consent. Turning sampling on also approves the judge runs the sampler starts: it issues the judge’s two grants on that budget, writes the manifest each judge run takes and shares the judge’s documents with the judge and the organization’s automation member, and the response carries that approval. Turning it on takes the Live environment and a recent second-factor session, and answers 503 where judge runs cannot run. Sampled judging spends the setting’s budget, never run budget. Requires an organization owner or administrator signed in as a person; API keys, service identities and workloads are refused. Turning sampling off records a version and revokes the approval, so a gap in a series keeps a reason and a time. Recording a setting is not a score, a calibration or promotion evidence. Preserve the original key after an uncertain response; repeating it continues where an interrupted request stopped.

Authorizations

Section titled “Authorizations”

Parameters

Section titled “Parameters”

Path Parameters

Section titled “Path Parameters”
org_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
environment_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
agent_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000

The agent whose live runs the sampling setting belongs to.

Header Parameters

Section titled “Header Parameters”
Idempotency-Key
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
X-Katara-CSRF
Allowed value: 1

Browser cookie mutations require value 1 and an allowed Origin. Session bearer clients may omit this header.

X-Katara-Account-ID
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/

Required for cookie-authenticated changes. Use the UUID of the account that loaded the application. A different current cookie account returns account_changed before the handler runs. Session bearer and API key clients do not require this header.

Request Bodyrequired

Section titled “Request Bodyrequired”
Media typeapplication/json

Turns judging of sampled live runs on or off for one agent. Every field is part of the setting, because a score has to stay attributable to what produced it: the reviewed rubric that judged it, the model that did the judging, and the budget and consent that paid. Turning sampling on is also the one approval its judge runs are admitted under: it issues the judge a grant over itself, the sampled agent’s runs and the documents it reads, and a grant for its model calls, both on this budget and ending at approval_expires_at; writes the manifest every judge run takes; and shares the judge’s documents with the judge and the organization’s automation member, which admits the judge runs and approves nothing. It needs a recent second-factor session and the Live environment. Recording is append-only and takes an organization owner or administrator signed in as a person; turning sampling off, or on again, records a version and revokes the approval it replaces. The agent cannot be its own judge.

object
agent_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
enabled
required
boolean
sample_rate_ppm
required
integer
>= 1 <= 1000000
rubric
required
object
asset_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
version_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
integrity
required
string
/^sha256:[a-f0-9]{64}$/
model_id
required
string
>= 1 characters <= 200 characters
judge_agent_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
budget_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
consent_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
daily_cap

The most sampled runs judged in one UTC day. Defaults to 100. Only when turning sampling on.

integer
>= 1 <= 10000
approval_expires_at

When the approval this request makes ends: more than an hour and at most 30 days away. Required when turning sampling on.

string format: date-time
output_target

Where the judge’s output is written, and where the manifest every judge run takes is kept. Required when turning sampling on.

object
store_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
store_version
required
integer
>= 1 <= 9007199254740991
key_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
key_version
required
integer
>= 1 <= 9007199254740991
retention_seconds
required
integer
>= 60 <= 31536000
Examplegenerated
{
  "agent_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "enabled": true,
  "sample_rate_ppm": 1,
  "rubric": {
    "asset_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
    "version_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
    "integrity": "example"
  },
  "model_id": "example",
  "judge_agent_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "budget_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "consent_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
  "daily_cap": 1,
  "approval_expires_at": "2026-04-15T12:00:00Z",
  "output_target": {
    "store_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
    "store_version": 1,
    "key_id": "2489E9AD-2EE2-8E00-8EC9-32D5F69181C0",
    "key_version": 1,
    "retention_seconds": 1
  }
}

Responses

Section titled “Responses”

201

Section titled “201”

The settings as recorded, with the version a sample refers back to.

Media typeapplication/json
object
agent_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
enabled
required
boolean
sample_rate_ppm
required
integer
>= 1 <= 1000000
rubric
required
object
asset_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
version_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
integrity
required
string
/^sha256:[a-f0-9]{64}$/
model_id
required
string
>= 1 characters <= 200 characters
judge_agent_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
budget_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
consent_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
version
required
integer
>= 1 <= 9007199254740991
recorded_by
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
recorded_at
required
string format: date-time
daily_cap
required
integer
>= 1 <= 10000
approval

The approval recorded with a setting that turned sampling on: the grant over the judge and the sampled agent’s runs, the judge’s model grant, the manifest its judge runs take, the judge version and rubric review it pins, and where judge output is written. current says whether it still stands: both grants unrevoked and unexpired, the person who recorded it still an owner or administrator, and no stop on evaluation. A setting without an approval is never sampled.

object
grant_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
model_grant_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
manifest_ref
required
object
org_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
environment_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
version
required
integer
>= 1 <= 9223372036854776000
integrity
required
string
/^sha256:[a-f0-9]{64}$/
media_type
required
Allowed values: text/plain text/markdown application/json application/pdf application/vnd.openxmlformats-officedocument.wordprocessingml.document image/png image/jpeg text/csv application/zip
size_bytes
required
integer
<= 104857600
judge_version
required
object
id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
version
required
integer
>= 1 <= 9223372036854776000
integrity
required
string
/^sha256:[a-f0-9]{64}$/
review_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
output_target
required

Explicit destination for run artifacts. Store and key versions must still be active at admission and byte access. Choose retention from 60 seconds to 365 days; no silent default or destination substitution.

object
store_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
store_version
required
integer
>= 1 <= 9007199254740991
key_id
required
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
key_version
required
integer
>= 1 <= 9007199254740991
retention_seconds
required
integer
>= 60 <= 31536000
expires_at
required
string format: date-time
current
required
boolean
Example
{
  "approval": {
    "manifest_ref": {
      "media_type": "text/plain"
    }
  }
}

409

Section titled “409”

Agent authority or cookie account changed. Reinspect before a new intent. Preserve the original key to recover an uncertain committed operation.

default

Section titled “default”

Request rejected or unavailable.

Media typeapplication/json
object
error
required
object
message
required
string
>= 1 characters <= 500 characters
type
required
string
>= 1 characters <= 100 characters
code
required

Stable; branch on it. Any operation can answer invalid_request, with param naming the refused field when there is one; human_session_required, when an API key, service account or workload calls an operation only a signed-in person may perform; reauthentication_required, when a person’s sign-in is too old for the operation; permission_denied; route_not_found and method_not_allowed, when no operation matches the path or method; and feature_unavailable, when the operation’s service is not enabled on this deployment.

string
>= 1 characters <= 100 characters
param
required

The request field a refusal is about, such as the agent_id or run_id query parameter; null when the refusal is not about one field.

string | null
request_id
required
string
>= 1 characters <= 100 characters
upstream_status
required
integer | null
action
Allowed values: sign_in reauthenticate request_access review fund retry reconcile refresh support
operation_id
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
snapshot_ref
string format: uuid
/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/
Allowed value: 00000000-0000-0000-0000-000000000000
Example
{
  "error": {
    "action": "sign_in"
  }
}