Evaluation datasets, rubrics and assessments

Which of these operations answer depends on the services enabled for your environment. Publishing this material does not run a model or produce scores. Funded assessments require an enabled judge execution service. Held-out calibration campaigns and blinded human reviews require the calibration service. Reports remain uncalibrated; promotion requires a separate approved evidence workflow.

A dataset defines the cases an agent should handle. Each case has a stable ID, an approved input reference and an approved expected-output reference. A rubric defines the criteria for judging a result, with approved written instructions and a passing threshold for each criterion.

For a returns assistant, a dataset might contain a request within the return window, an expired return and a request with missing order details. A rubric could check whether the response cites the correct policy and whether it requests an unauthorized refund. A threshold is a requirement you choose; it is not a measured result.

Prepare and publish

Upload, review and approve the case bodies and rubric instructions through the knowledge workflow. Evaluation versions reference those exact approved content versions. Every reference must belong to the selected organization and environment. A changed source requires a new evaluation version.

Choose a dataset designation at its first publication:

Designation	Use
development	Cases used while developing or improving an agent
held_out	Cases reserved for checking a candidate after development

The designation stays fixed across versions of that dataset. Keep development and held-out examples separate when preparing material; the designation alone does not prove that a model has never seen an example.

Use publishEvaluationMaterial / publish_evaluation_material with kind dataset or rubric, the asset ID and its expected current version. The first publication uses expected_version: 0. Later publications keep the asset ID and provide its current version. Each accepted publication creates a new immutable version.

Rubric categories cover task success, source accuracy, citation accuracy, retrieval quality, unwanted tool actions, safety and format compliance. Each criterion has a passing score from 0 through 100 and a reference to its written instructions.

Review the exact rubric

Use a signed-in account to preview the rubric’s exact version and integrity value. Read the returned criterion text, then explicitly acknowledge that version with reviewRubric / review_rubric. An API key cannot perform this human review.

Review requires current permission to review the referenced content. A newer rubric version needs its own review. A review records who approved those criteria and when; it does not certify that a model judge is calibrated or that an agent passed.

Inspect, change and revoke

listEvaluationMaterial / list_evaluation_material lists the latest accessible version of each asset. listEvaluationVersions / list_evaluation_versions shows its version history. getEvaluationVersion / get_evaluation_version reads one exact version. Current source permissions apply to these reads. In the portal, Export Version on a version’s detail downloads that exact record as JSON.

To withdraw a version, call revokeEvaluationVersion / revoke_evaluation_version with its current version and integrity value. Revocation cannot be undone. Publish a new version when the material is ready again.

Keep the original request body and idempotency key after an interrupted publication, review or revocation. Reusing them recovers the original operation. A changed version or body requires a new review of the current state before another mutation.

Run one funded assessment

An assessment judges one case against one exact, human-reviewed rubric version. Prepare an approved candidate-output ContentRef, a tool-free judge agent version, a supported Cortex model, current workload grants, an output destination and a funded budget with inference and evaluation caps. An assessment is charged for two things, on separate statement lines: the case, as one evaluation unit, and the judge’s model call, as inference. Test Compute uses real credits and requires explicit paid-test consent. Simulation cannot submit a paid assessment.

1. Read getEvaluationJudgeProfile / get_evaluation_judge_profile. Pin the returned instruction and response hashes in the request. The profile uses temperature zero, at most 4,096 output tokens and one attempt.
2. Select the exact dataset version and case, rubric version and human review ID, candidate reference, model, agent version, grants, budget and output destination. Each case body, rubric instruction and candidate reference is limited to 128 KiB. A rubric has at most 32 criteria.
3. Call assessmentManifest / assessment_manifest with the scope and assessment intent. The helper returns the manifest JSON without a reference to itself. Upload those exact JSON bytes through the content workflow, then review and approve that exact version. Supply its ContentRef as run.input_ref. Changing the intent requires a new manifest upload and approval.
4. Submit createEvaluationAssessment / create_evaluation_assessment with the reviewed body and a new idempotency key. Keep both until the admission outcome is known. The returned operation’s resource ID identifies both the assessment and run.

If the reply is lost or admission remains uncertain, retry the identical request with its original key. A fresh key can create a second paid assessment. Do not silently rebuild the manifest or replace references while recovering the original operation. An authority or version conflict requires inspecting the current state before making a new intent.

What an assessment costs

The case is one evaluation unit at the evaluation price in force when the assessment is admitted. Admission holds it against the budget’s evaluation cap, and it is charged once when the judge’s call is charged, whatever the judge answered: a score, a not_fully_scorable result or a reply that failed to parse. Retrying the admission with its original key finds the same unit, and retrying the judge reads the recorded outcome instead of calling the model again. Neither adds a unit, and neither do score reads, calibration, review or promotion. A request refused before admission holds nothing.

The judge’s call is inference, charged at the model’s rate. It counts toward the inference cap and, because it judges a case, toward the evaluation cap as well; the budget’s total counts it once.

If the run ends without its judge call, the unit is released about half an hour after it ends. If the call’s outcome is unknown, the unit stays held as exposure until the call is reconciled, and is then charged or released with it.

Read the result

Use listEvaluationAssessments / list_evaluation_assessments to list at most 25 assessments per page, then getEvaluationAssessment / get_evaluation_assessment for one record. Pass next_cursor unchanged as after; it is encrypted and bound to current caller and scope authority. A changed authority can invalidate pagination.

getEvaluationAssessmentScore / get_evaluation_assessment_score distinguishes:

Result	Meaning
available	A canonical parsed score exists. Its calibration status is uncalibrated.
pending	No canonical score exists. This does not establish that the run is running or healthy.
Access or source error	Current permission or source authority failed. Do not display this as pending or as a score of zero.

A criterion is either scored from 0 through 100 with its pass decision, or not_scorable with a reason. A not_fully_scorable assessment must not be represented as an overall pass. Inspect the run separately for execution and charge outcomes; a charged call can end without a parsed score.

Compare the score’s assessment and admission digests, criterion IDs, thresholds and source provenance with the current assessment before displaying it. SDK applications can pass the assessment record to validateAssessmentScoreState / validate_assessment_score_state for this comparison. Reading never retries inference. A score does not establish calibration, agent promotion or real-provider settlement.

Export Result on an assessment in the portal downloads the assessment record together with the score state it was shown with.

Quality and cost over time

One score says how one run went. evaluationSummary / evaluation_summary says how judging against a rubric has gone since a time you name, as an RFC 3339 since, thirty days back by default and a year at most. It counts the assessments judged against any version of the rubric and how many were scored, not fully scorable or still waiting for a score. For each criterion it gives how many were scored, passed, failed or could not be scored, and the mean score; a criterion the current version no longer has is listed under its ID. The judge’s cost is the credits settlement charged for each judging call, with the tokens it read and wrote. The evaluation units for the cases are not part of it; the statement’s evaluation line has them.

For settings that judge live traffic against the rubric, the summary also counts the completed runs sampling considered, selected and had judged. Selected includes runs drawn and then skipped for the daily cap or the budget.

The summary reads recorded scores and changes nothing. It needs synapse:evaluations:read, and its scores are as uncalibrated as each one is on its own. In the portal, open a rubric version under Synapse → Evaluations and read it under Quality and Cost.

Collect blinded calibration evidence

Open Synapse → Calibration to create a campaign. Select exact published held-out and development dataset versions, one canonical completed assessment for every held-out case, and an explicit acceptance policy. All held-out cases must be included. The service checks one exact judge, model profile and reviewed rubric, source separation, current access and canonical score provenance. Campaigns contain at most 1,000 cases.

Enter confidence, minimum case and human class counts, minimum human agreement, label coverage, judge coverage and judge agreement, and maximum false-positive and false-negative rates for every criterion. Rate thresholds use parts per million: 950000 means 95%. Missing thresholds are rejected. The immutable campaign pins this policy before labels are collected.

The TypeScript evaluationCalibration and Python evaluation_calibration interfaces provide create, list, get, assign, assignments, assignment, content, label, seal, certificate, compare and comparison. Campaign reads require the creator’s current authority. list returns the campaigns you created in the current reset generation, newest first, up to 25 a page, each with the certificate that sealed it; pass next_cursor as after until it is absent. The list does not rebuild a campaign from its sources, so get can still refuse one whose sources were withdrawn. The portal’s Calibration page shows the same list.

Send the campaign ID to independent reviewers. In Blinded Reviews, each reviewer requests an assignment using their own account. The service chooses the next eligible case. The response includes only the assignment ID, exact input, expected and candidate references, and rubric criteria. It does not reveal the campaign creator, judge, model, score or other reviewers’ labels.

assignments lists a reviewer’s own open assignments, newest first, up to 25 a page. An assignment that expired, or whose sources the reviewer can no longer review, is left out, so a page can hold fewer entries than the limit while next_cursor is present. Nothing in the list names the campaign, the case, the judge, its scores or another reviewer.

Read each assigned content part through content. Criterion reads require one of the assigned criterion IDs; arbitrary content references are refused. Each body is bounded to 128 KiB, and the SDK checks its byte count and SHA-256 integrity. Current assignment and source authority remain held while the response is written. Assignment access does not grant general content access.

Label every criterion pass, fail or not_scorable. An unscorable label needs one of insufficient_evidence, ambiguous_criterion or outside_expertise. Submission is immutable. Reviewers must be independent of the candidate submitter and campaign creator. The service collects two independent labels and requests adjudication when needed; callers cannot choose an assignment’s case or another reviewer’s identity.

Inspect certificates and paired comparisons

The creator seals a certificate after human review is complete. It binds the exact campaign, immutable labels and current canonical judge evidence. Inspect missing labels, abstentions, confusion counts and simultaneous confidence bounds per criterion. insufficient_evidence, criteria_failed and criteria_met are statistical report states. All retain calibration_status: uncalibrated; none is promotion approval.

Compare two certificates from the same held-out cohort. An outputs comparison requires the same judge. A judges comparison requires the same outputs and an explicit choice of shared human labels from one certificate. The result records paired success counts, missing evidence, abstentions and the observed difference. It does not claim statistical significance or authorize promotion.

Use the same body and idempotency key after an unknown campaign, assignment, label, certificate or comparison outcome. The portal retains only recovery metadata across reload: the action, request digest and original key. Reselect the exact inputs to retry; review content and labels are not kept in browser storage. Changing account, organization, environment or mode discards stale responses.

Permissions

Reading material requires synapse:evaluations:read. Publishing and revoking require synapse:evaluations:write. Publication also checks current read access to every approved content reference. Human rubric review uses synapse:reviews:write and a verified account session. Resource restrictions apply to the dataset or rubric asset. Assessment creation also requires run creation authority. Score reads require synapse:runs:read, the original requester and current source, rubric reviewer and resource authority. Workload credentials cannot use these customer assessment read routes. Calibration routes require a human session; campaign writes use evaluation authority and blinded labels use current review authority. Listing campaigns also needs a role that can read the datasets and scores a campaign is built from, and listing assignments needs an owner, administrator or reviewer role. API keys and workload or service credentials are refused.

Judging live traffic

A dataset says how an agent handled cases somebody wrote down. It does not say how the agent is handling the work it is doing now, and an agent that met its bar in March is not thereby meeting it today.

Online evaluation samples completed live runs and has the judge score them against the rubric the agent was promoted on. It is off until an owner or administrator turns it on. Turning it on records the rubric version, the model, the judge and the budget it spends, so a score stays attributable to what produced it, and turning it off is a new version rather than an edit: a gap in a trend line has a reason with a name and a time on it.

The decision for each completed run is taken once and kept beside the rate it was compared against, so the sampling can be rechecked rather than believed. A sampled case carries no expected output, because nobody wrote one before the run happened, and the material for it is published by the recorded setting rather than by a person.

Turn it on

Owners and administrators turn it on from the agent’s page in the portal, under Online Evaluation, or with setOnlineEvaluation / set_online_evaluation. Turning it on needs a person’s signed-in session, in a Live environment; an API key cannot do it. Besides the rubric version, model, judge, budget and consent, the request names:

Field	What it sets
sample_rate_ppm	The share of completed runs drawn, in parts per million
daily_cap	The most drawn runs judged in one UTC day. Defaults to 100, at most 10,000
approval_expires_at	When the approval ends: more than an hour away and at most 30 days. The portal offers 30 days
output_target	The store, key and retention the judge’s output is written with

Turning it on is the approval its judge runs are admitted under, so nobody has to approve each one. It issues the judge two grants from you, both ending at approval_expires_at and both paid from the setting’s budget: one over the judge, the sampled agent’s runs and the documents the judge reads, and one for its model calls. The judge runs are started by the organization’s automation member, a member every organization has that nobody can sign in as. It spends under those grants and approves nothing. onlineEvaluation / online_evaluation returns the setting with its approval: both grant IDs, the manifest every judge run takes, the judge version and rubric review it pinned, the output target, when it ends, and current, which says whether it still stands. The grants are listed with your other grants and appear in access reviews.

Which runs are judged

A run is drawn only if it ran in Live and completed after the setting was turned on. Test runs are never drawn, and neither is a run that completed earlier. The draw comes from the run’s ID and a value kept with that version of the setting, so it is the same each time it is checked. The decision is taken once per run, and a run is judged at most once. The judge reads the run’s input and output where they are stored; nothing is copied for it. Its assessments are listed and summarized like any other, but send no synapse.evaluation_ready, synapse.evaluation_failed or synapse.evaluation_unknown notice, because no person asked for them.

When the day’s cap is reached, or the budget cannot cover another judge run, a drawn run is recorded as skipped with the reason, daily_cap or budget, and not judged later. The person who turned the setting on receives synapse.online_evaluation_skipped once a day for each agent and reason. The setting stays on; the next day, or a budget with funds, resumes judging.

Stop it

Turning the setting off records a new version and revokes the approval’s two grants. Sampling also stops when the approval stops standing: either grant revoked or expired, the person who turned it on no longer an owner or administrator, an evaluation stop control in force, or the environment reset. Runs drawn and not yet admitted are then given up. Turning it on again issues a new approval and revokes the previous one.

See access and API keys for account binding, Test/Live separation and request recovery. The product API reference documents the exact request and response fields.