# Evaluation datasets, rubrics and assessments

> Publish fixed evaluation cases and review the criteria used to judge a result.

Section: Synapse and Locus. Web page: https://docs.katara.com/products/evaluations/

Which of these operations answer depends on the services enabled for your environment. Publishing this material does
not run a model or produce scores. Funded assessments require an enabled judge execution service. Held-out calibration campaigns and blinded human reviews require the calibration service.
Reports remain uncalibrated; promotion requires a separate approved evidence workflow.

A dataset defines the cases an agent should handle. Each case has a stable ID, an
approved input reference and an approved expected-output reference. A rubric defines
the criteria for judging a result, with approved written instructions and a passing
threshold for each criterion.

For a returns assistant, a dataset might contain a request within the return window,
an expired return and a request with missing order details. A rubric could check
whether the response cites the correct policy and whether it requests an unauthorized
refund. A threshold is a requirement you choose; it is not a measured result.

## Prepare and publish

Upload, review and approve the case bodies and rubric instructions through the
[knowledge workflow](https://docs.katara.com/products/knowledge.txt). Evaluation versions reference those exact
approved content versions. Every reference must belong to the selected organization
and environment. A changed source requires a new evaluation version.

Choose a dataset designation at its first publication:

| Designation | Use |
| --- | --- |
| `development` | Cases used while developing or improving an agent |
| `held_out` | Cases reserved for checking a candidate after development |

The designation stays fixed across versions of that dataset. Keep development and
held-out examples separate when preparing material; the designation alone does not
prove that a model has never seen an example.

Use `publishEvaluationMaterial` / `publish_evaluation_material` with kind `dataset`
or `rubric`, the asset ID and its expected current version. The first publication
uses `expected_version: 0`. Later publications keep the asset ID and provide its
current version. Each accepted publication creates a new immutable version.

Rubric categories cover task success, source accuracy, citation accuracy, retrieval
quality, unwanted tool actions, safety and format compliance. Each criterion has
a passing score from 0 through 100 and a reference to its written instructions.

## Review the exact rubric

Use a signed-in account to preview the rubric's exact version and integrity value.
Read the returned criterion text, then explicitly acknowledge that version with
`reviewRubric` / `review_rubric`. An API key cannot perform this human review.

Review requires current permission to review the referenced content. A newer rubric
version needs its own review. A review records who approved those criteria and when;
it does not certify that a model judge is calibrated or that an agent passed.

## Inspect, change and revoke

`listEvaluationMaterial` / `list_evaluation_material` lists the latest accessible
version of each asset. `listEvaluationVersions` / `list_evaluation_versions` shows
its version history. `getEvaluationVersion` / `get_evaluation_version` reads one
exact version. Current source permissions apply to these reads. In the portal,
**Export Version** on a version's detail downloads that exact record as JSON.

To withdraw a version, call `revokeEvaluationVersion` / `revoke_evaluation_version`
with its current version and integrity value. Revocation cannot be undone. Publish
a new version when the material is ready again.

Keep the original request body and idempotency key after an interrupted publication,
review or revocation. Reusing them recovers the original operation. A changed version
or body requires a new review of the current state before another mutation.

## Run one funded assessment

An assessment judges one case against one exact, human-reviewed rubric version.
Prepare an approved candidate-output ContentRef, a tool-free judge agent version,
a supported Cortex model, current workload grants, an output destination and a
funded budget with inference and evaluation caps. An assessment is charged for two
things, on separate statement lines: the case, as one evaluation unit, and the judge's
model call, as inference. Test Compute uses real credits and requires explicit
paid-test consent. Simulation cannot submit a paid assessment.

1. Read `getEvaluationJudgeProfile` / `get_evaluation_judge_profile`. Pin the returned
   instruction and response hashes in the request. The profile uses temperature zero,
   at most 4,096 output tokens and one attempt.
2. Select the exact dataset version and case, rubric version and human review ID,
   candidate reference, model, agent version, grants, budget and output destination.
   Each case body, rubric instruction and candidate reference is limited to 128 KiB.
   A rubric has at most 32 criteria.
3. Call `assessmentManifest` / `assessment_manifest` with the scope and assessment
   intent. The helper returns the manifest JSON without a reference to itself.
   Upload those exact JSON bytes through the content workflow, then review and approve
   that exact version. Supply its ContentRef as `run.input_ref`. Changing the intent
   requires a new manifest upload and approval.
4. Submit `createEvaluationAssessment` / `create_evaluation_assessment` with the
   reviewed body and a new idempotency key. Keep both until the admission outcome is
   known. The returned operation's resource ID identifies both the assessment and run.

If the reply is lost or admission remains uncertain, retry the identical request with
its original key. A fresh key can create a second paid assessment. Do not silently
rebuild the manifest or replace references while recovering the original operation.
An authority or version conflict requires inspecting the current state before making
a new intent.

### What an assessment costs

The case is one evaluation unit at the evaluation price in force when the assessment
is admitted. Admission holds it against the budget's evaluation cap, and it is charged
once when the judge's call is charged, whatever the judge answered: a score, a
`not_fully_scorable` result or a reply that failed to parse. Retrying the admission
with its original key finds the same unit, and retrying the judge reads the recorded
outcome instead of calling the model again. Neither adds a unit, and neither do score
reads, calibration, review or promotion. A request refused before admission holds
nothing.

The judge's call is inference, charged at the model's rate. It counts toward the
inference cap and, because it judges a case, toward the evaluation cap as well; the
budget's total counts it once.

If the run ends without its judge call, the unit is released about half an hour after
it ends. If the call's outcome is unknown, the unit stays held as exposure until the
call is reconciled, and is then charged or released with it.

## Read the result

Use `listEvaluationAssessments` / `list_evaluation_assessments` to list at most 25
assessments per page, then `getEvaluationAssessment` / `get_evaluation_assessment`
for one record. Pass `next_cursor` unchanged as `after`; it is encrypted and bound to
current caller and scope authority. A changed authority can invalidate pagination.

`getEvaluationAssessmentScore` / `get_evaluation_assessment_score` distinguishes:

| Result | Meaning |
| --- | --- |
| `available` | A canonical parsed score exists. Its calibration status is `uncalibrated`. |
| `pending` | No canonical score exists. This does not establish that the run is running or healthy. |
| Access or source error | Current permission or source authority failed. Do not display this as pending or as a score of zero. |

A criterion is either scored from 0 through 100 with its pass decision, or
`not_scorable` with a reason. A `not_fully_scorable` assessment must not be represented
as an overall pass. Inspect the run separately for execution and charge outcomes;
a charged call can end without a parsed score.

Compare the score's assessment and admission digests, criterion IDs, thresholds and
source provenance with the current assessment before displaying it. SDK applications
can pass the assessment record to `validateAssessmentScoreState` /
`validate_assessment_score_state` for this comparison. Reading never retries inference.
A score does not establish calibration, agent promotion or real-provider settlement.

**Export Result** on an assessment in the portal downloads the assessment record
together with the score state it was shown with.

## Quality and cost over time

One score says how one run went. `evaluationSummary` / `evaluation_summary` says
how judging against a rubric has gone since a time you name, as an RFC 3339 `since`,
thirty days back by default and a year at most. It counts the assessments judged
against any version of the rubric and how many were scored, not fully scorable or
still waiting for a score. For each criterion it gives how many were scored, passed,
failed or could not be scored, and the mean score; a criterion the current version
no longer has is listed under its ID. The judge's cost is the credits settlement
charged for each judging call, with the tokens it read and wrote. The evaluation units
for the cases are not part of it; the statement's evaluation line has them.

For settings that judge live traffic against the rubric, the summary also counts the
completed runs sampling considered, selected and had judged. Selected includes runs
drawn and then skipped for the daily cap or the budget.

The summary reads recorded scores and changes nothing. It needs
`synapse:evaluations:read`, and its scores are as uncalibrated as each one is on its
own. In the portal, open a rubric version under **Synapse → Evaluations** and read it
under **Quality and Cost**.

## Collect blinded calibration evidence

Open **Synapse → Calibration** to create a campaign. Select exact published held-out
and development dataset versions, one canonical completed assessment for every held-out
case, and an explicit acceptance policy. All held-out cases must be included. The
service checks one exact judge, model profile and reviewed rubric, source separation,
current access and canonical score provenance. Campaigns contain at most 1,000 cases.

Enter confidence, minimum case and human class counts, minimum human agreement,
label coverage, judge coverage and judge agreement, and maximum false-positive and
false-negative rates for every criterion. Rate thresholds use parts per million:
950000 means 95%. Missing thresholds are rejected. The immutable campaign pins this
policy before labels are collected.

The TypeScript `evaluationCalibration` and Python `evaluation_calibration` interfaces
provide `create`, `list`, `get`, `assign`, `assignments`, `assignment`, `content`, `label`,
`seal`, `certificate`, `compare` and `comparison`. Campaign reads require the creator's
current authority. `list` returns the campaigns you created in the current reset
generation, newest first, up to 25 a page, each with the certificate that sealed it;
pass `next_cursor` as `after` until it is absent. The list does not rebuild a campaign
from its sources, so `get` can still refuse one whose sources were withdrawn. The
portal's **Calibration** page shows the same list.

Send the campaign ID to independent reviewers. In **Blinded Reviews**, each reviewer
requests an assignment using their own account. The service chooses the next eligible
case. The response includes only the assignment ID, exact input, expected and candidate
references, and rubric criteria. It does not reveal the campaign creator, judge,
model, score or other reviewers' labels.

`assignments` lists a reviewer's own open assignments, newest first, up to 25 a page.
An assignment that expired, or whose sources the reviewer can no longer review, is
left out, so a page can hold fewer entries than the limit while `next_cursor` is
present. Nothing in the list names the campaign, the case, the judge, its scores or
another reviewer.

Read each assigned content part through `content`. Criterion reads require one of the
assigned criterion IDs; arbitrary content references are refused. Each body is bounded
to 128 KiB, and the SDK checks its byte count and SHA-256 integrity. Current assignment
and source authority remain held while the response is written. Assignment access does
not grant general content access.

Label every criterion `pass`, `fail` or `not_scorable`. An unscorable label needs one
of `insufficient_evidence`, `ambiguous_criterion` or `outside_expertise`. Submission
is immutable. Reviewers must be independent of the candidate submitter and campaign
creator. The service collects two independent labels and requests adjudication when
needed; callers cannot choose an assignment's case or another reviewer's identity.

## Inspect certificates and paired comparisons

The creator seals a certificate after human review is complete. It binds the exact
campaign, immutable labels and current canonical judge evidence. Inspect missing
labels, abstentions, confusion counts and simultaneous confidence bounds per criterion.
`insufficient_evidence`, `criteria_failed` and `criteria_met` are statistical report
states. All retain `calibration_status: uncalibrated`; none is promotion approval.

Compare two certificates from the same held-out cohort. An `outputs` comparison
requires the same judge. A `judges` comparison requires the same outputs and an explicit
choice of shared human labels from one certificate. The result records paired success
counts, missing evidence, abstentions and the observed difference. It does not claim
statistical significance or authorize promotion.

Use the same body and idempotency key after an unknown campaign, assignment, label,
certificate or comparison outcome. The portal retains only recovery metadata across
reload: the action, request digest and original key. Reselect the exact inputs to retry;
review content and labels are not kept in browser storage. Changing account, organization,
environment or mode discards stale responses.

## Permissions

Reading material requires `synapse:evaluations:read`. Publishing and revoking require
`synapse:evaluations:write`. Publication also checks current read access to every
approved content reference. Human rubric review uses `synapse:reviews:write` and a
verified account session. Resource restrictions apply to the dataset or rubric asset. Assessment creation also
requires run creation authority. Score reads require `synapse:runs:read`, the original
requester and current source, rubric reviewer and resource authority. Workload
credentials cannot use these customer assessment read routes. Calibration routes require
a human session; campaign writes use evaluation authority and blinded labels use current
review authority. Listing campaigns also needs a role that can read the datasets and
scores a campaign is built from, and listing assignments needs an owner, administrator
or reviewer role. API keys and workload or service credentials are refused.

## Judging live traffic

A dataset says how an agent handled cases somebody wrote down. It does not say
how the agent is handling the work it is doing now, and an agent that met its
bar in March is not thereby meeting it today.

Online evaluation samples completed live runs and has the judge score them
against the rubric the agent was promoted on. It is off until an owner or
administrator turns it on. Turning it on records the rubric version, the model,
the judge and the budget it spends, so a score stays attributable to what
produced it, and turning it off is a new version rather than an edit: a gap in
a trend line has a reason with a name and a time on it.

The decision for each completed run is taken once and kept beside the rate it
was compared against, so the sampling can be rechecked rather than believed. A
sampled case carries no expected output, because nobody wrote one before the
run happened, and the material for it is published by the recorded setting
rather than by a person.

### Turn it on

Owners and administrators turn it on from the agent's page in the portal, under
**Online Evaluation**, or with `setOnlineEvaluation` / `set_online_evaluation`.
Turning it on needs a person's signed-in session, in a Live environment; an API key
cannot do it. Besides the rubric version, model, judge, budget and consent, the request
names:

| Field | What it sets |
| --- | --- |
| `sample_rate_ppm` | The share of completed runs drawn, in parts per million |
| `daily_cap` | The most drawn runs judged in one UTC day. Defaults to 100, at most 10,000 |
| `approval_expires_at` | When the approval ends: more than an hour away and at most 30 days. The portal offers 30 days |
| `output_target` | The store, key and retention the judge's output is written with |

Turning it on is the approval its judge runs are admitted under, so nobody has to
approve each one. It issues the judge two grants from you, both ending at
`approval_expires_at` and both paid from the setting's budget: one over the judge,
the sampled agent's runs and the documents the judge reads, and one for its model
calls. The judge runs are started by the organization's automation member, a member
every organization has that nobody can sign in as. It spends under those grants and
approves nothing. `onlineEvaluation` / `online_evaluation` returns the setting with
its `approval`: both grant IDs, the manifest every judge run takes, the judge version
and rubric review it pinned, the output target, when it ends, and `current`, which
says whether it still stands. The grants are listed with your other grants and
appear in access reviews.

### Which runs are judged

A run is drawn only if it ran in Live and completed after the setting was turned
on. Test runs are never drawn, and neither is a run that completed earlier. The
draw comes from the run's ID and a value kept with that version of the setting, so
it is the same each time it is checked. The decision is taken once per run, and a
run is judged at most once. The judge reads the run's input and output where they
are stored; nothing is copied for it. Its assessments are listed and summarized
like any other, but send no `synapse.evaluation_ready`, `synapse.evaluation_failed`
or `synapse.evaluation_unknown` notice, because no person asked for them.

When the day's cap is reached, or the budget cannot cover another judge run, a
drawn run is recorded as skipped with the reason, `daily_cap` or `budget`, and
not judged later. The person who turned the setting on receives
`synapse.online_evaluation_skipped` once a day for each agent and reason. The
setting stays on; the next day, or a budget with funds, resumes judging.

### Stop it

Turning the setting off records a new version and revokes the approval's two
grants. Sampling also stops when the approval stops standing: either grant revoked
or expired, the person who turned it on no longer an owner or administrator, an
evaluation stop control in force, or the environment reset. Runs drawn and not yet
admitted are then given up. Turning it on again issues a new approval and revokes
the previous one.

See [access and API keys](https://docs.katara.com/products/access.txt) for account binding, Test/Live separation
and request recovery. The [product API reference](https://docs.katara.com/product-api.txt) documents the exact
request and response fields.