> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qoder.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Outcome Evaluation
> Set acceptance criteria on a Session with user.define_outcome and read the evaluation result.
An Outcome sets a gradable acceptance goal on a Session: `description` states the task and `rubric` states the acceptance criteria. After the Agent finishes its work, a separate evaluation thread grades the output against the rubric and returns a verdict such as `satisfied` or `failed`.
For the recommended end-to-end usage, see [Outcome Best Practices](/cloud-agents/api/sessions/outcomes-best-practices).
## How it works
Defining an Outcome starts an Agent turn with `description` as its input. When the turn ends, the evaluation thread grades the output. If the output does not pass, the verdict is fed back to the Agent for revision until it passes or the iteration limit is reached.
Send `user.define_outcome` with `description` and `rubric`. The server generates `outcome_id`.
The Agent produces its output from `description`. When the turn ends normally, evaluation starts directly and the Session stays `running`.
The evaluation thread grades the output against the rubric. The event stream emits `span.outcome_evaluation_start` and `span.outcome_evaluation_ongoing`.
When `span.outcome_evaluation_end.result` is `needs_revision`, the Agent starts a revision turn based on the verdict. The client does not need to send an event.
The Outcome reaches `satisfied`, `failed`, `max_iterations_reached`, or `interrupted`, and `completed_at` is set. After the evaluation thread returns a verdict, the Agent runs one wrap-up turn to write its final reply, and the Session then returns to `idle`.
A Session can have only one active Outcome at a time. After an Outcome reaches a terminal state, you can define another one on the same Session; the Session keeps the earlier evaluation records.
## Write a rubric
A rubric is a Markdown document that lists the acceptance criteria one by one. The evaluation thread grades each criterion separately, so more specific criteria produce more stable verdicts.
```markdown
# Performance report rubric
## Deliverables
- /data/report.html exists
- The report contains throughput and latency figures, with at least one chart each
## Quality of conclusions
- The top 3 bottlenecks are listed explicitly
- Each bottleneck cites a code location or profiling data as evidence
## Exclusions
- No placeholder content, TODOs, or sample data
```
The evaluation thread can read only the rubric, the sandbox, and the Agent's reply. It cannot read `description`. Put any expected values or source data the criteria depend on into the rubric.
- **Make every criterion verifiable.** Write "the CSV contains a numeric `price` column", not "the data quality is good".
- **Require concrete evidence.** For example a fetched page, a traceable computation, or a `file:line` reference.
- **Describe the goal, not the steps.** If a command prescribed by the rubric is unavailable, the evaluation cannot proceed.
- **State what is not acceptable.** For example "mirrors, reposts, and search snippets are not accepted as evidence".
- **Specify the feedback format.** Require a per-criterion pass or fail with the gap.
- **State what to ignore.** For example formatting, or extra deliverables unrelated to the criteria.
- If you have no rubric yet, ask a model to analyze a known-good output and turn the analysis into a rubric.
With vague criteria the evaluation thread tends to return `satisfied`; overly strict criteria add evaluation iterations. Tune the rubric with `max_iterations: 1` first and confirm that one iteration produces the correct verdict.
To reuse a rubric across Sessions, [upload it as a file](/cloud-agents/api/files/upload) and reference it with `{"type":"file","file_id":"file_..."}`.
## Define an Outcome
`POST /api/v1/cloud/sessions/{session_id}/events`
```json
{
"events": [
{
"type": "user.define_outcome",
"description": "Deliver a runnable performance report that identifies the top 3 bottlenecks",
"rubric": {
"type": "text",
"content": "Must produce /data/report.html; include throughput and latency figures; list the top 3 bottlenecks with evidence"
},
"max_iterations": 1
}
]
}
```
| Field | Type | Required | Description |
| ---------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `description` | string | Yes | The task description, used as the input of the Agent turn |
| `rubric` | object | Yes | The acceptance criteria, used only by the evaluation thread. Either `{"type":"text","content":"..."}` or `{"type":"file","file_id":"file_..."}`; unknown fields are not allowed |
| `max_iterations` | integer | No | Maximum number of evaluation iterations, an integer between 1 and 20. Defaults to 3 |
- `user.define_outcome` immediately starts a billable Agent turn. Put the task in `description` and only the acceptance criteria in `rubric`.
- The rubric (text or file content) is limited to 64 KiB; a larger rubric returns `400`. When `rubric.type` is `file`, the referenced file must be readable.
- Every evaluation iteration consumes credits. A larger `max_iterations` allows more evaluations in the worst case.
- `outcome_id` (prefix `outc_`) is generated by the server and cannot be included in the request.
## Evaluation state
Each Outcome has one entry in the `outcome_evaluations` array of the Session object, returned by [Get Session](/cloud-agents/api/sessions/get), [List Sessions](/cloud-agents/api/sessions/list), and [Search Sessions](/cloud-agents/api/sessions/search).
| Field | Type | Description |
| -------------- | -------------- | ----------------------------------------------------------------------------------------- |
| `type` | string | Always `outcome_evaluation` |
| `outcome_id` | string | Outcome ID with the `outc_` prefix |
| `description` | string | The `description` passed when the Outcome was defined |
| `iteration` | integer | Current evaluation iteration, starting at 0 |
| `result` | string | Current state. See the table below |
| `explanation` | string \ | null | Explanation from the most recent evaluation; `null` before the first evaluation completes |
| `completed_at` | string \ | null | When the Outcome reached a terminal state; `null` until then |
Values of `result`:
| Value | Description |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pending` | Defined; the Agent has not started working |
| `running` | The Agent is working or revising |
| `evaluating` | The evaluation thread is grading |
| `satisfied` | Terminal. All criteria are met |
| `max_iterations_reached` | Terminal. The output still did not pass after the maximum number of iterations |
| `failed` | Terminal. The evaluation thread determined that the task cannot satisfy the rubric (for example `description` contradicts the rubric), or it returned no verdict |
| `interrupted` | Terminal. The Outcome was cancelled, interrupted, or reclaimed by the server |
## Evaluation events
The record of each evaluation iteration is available on the event stream ([List events](/cloud-agents/api/sessions/events/list) or [Stream events](/cloud-agents/api/sessions/events/stream)):
| Event | Fields | Description |
| --------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| `span.outcome_evaluation_start` | `outcome_id`, `iteration` | An evaluation iteration started |
| `span.outcome_evaluation_ongoing` | `outcome_id`, `iteration` | The evaluation is in progress. The interval is not fixed; do not base a timeout on it |
| `span.outcome_evaluation_end` | `outcome_id`, `iteration`, `result`, `explanation`, `outcome_evaluation_start_id`, `usage` | An evaluation iteration ended |
- In addition to the terminal values, `span.outcome_evaluation_end.result` can be `needs_revision`, which means the iteration did not pass and iterations remain. When the last iteration does not pass, the value is `max_iterations_reached`.
- `usage.credits` is the credits consumed by the iteration. `usage` is omitted when the iteration has no metering record.
- When an Outcome is interrupted before evaluation starts, the event stream contains only `span.outcome_evaluation_end` with `result` set to `interrupted`. There is no matching `span.outcome_evaluation_start`, and `outcome_evaluation_start_id` is an empty string.
## Session status during an Outcome
The server switches between the Agent turn, evaluation, and revision without emitting `session.status_idle` or `session.status_running`. From the moment the Outcome is defined until the wrap-up turn ends, the Session stays `running`, and sending `user.message` or `user.define_outcome` returns `409`.
`completed_at` is set when the evaluation thread returns a terminal verdict, which is before the wrap-up turn ends:
| Purpose | Condition |
| ---------------------------------------------------- | -------------------------------------------------- |
| Read the verdict | `outcome_evaluations[].completed_at` is not `null` |
| Read the Agent's final reply, or send the next event | Session `status` is back to `idle` |
One Outcome produces several `agent.message` events: one for the Agent turn, one for each revision turn, and one for the wrap-up turn. The final reply is the one after the terminal `span.outcome_evaluation_end`. `interrupted`, and a `failed` caused by the evaluation thread returning no verdict, have no wrap-up turn.
The event stream can still carry the following status events during an Outcome. They do not mean the Outcome has ended:
| Event | Cause | Action |
| ---------------------------------------------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `session.status_idle` with `stop_reason.type` set to `requires_action` | A tool call is waiting for confirmation | See [Tool confirmation](#tool-confirmation) |
| `session.status_rescheduled` followed by `session.status_running` | Automatic model request retry | None. See [Automatic model request retries](/cloud-agents/sessions#automatic-model-request-retries) |
### Tool confirmation
The evaluation thread inherits the Agent's tool permissions so that it can verify the output in the sandbox. A tool configured as `always_ask` pauses for confirmation whether the Agent or the evaluation thread calls it. The flow is the same as the [Pending Action Flow](/cloud-agents/permission-policies#pending-action-flow):
1. The event stream emits `agent.tool_use` with `evaluated_permission` set to `ask`, and `session.status_idle` with `stop_reason.type` set to `requires_action`.
2. The client sends `user.tool_confirmation` with the event ID from `stop_reason.event_ids`.
3. The evaluation continues after confirmation.
While waiting for confirmation, the Session `status` remains `running` and `outcome_evaluations[].result` is `running` or `evaluating`. A pending action does not time out, so the evaluation does not continue until it is confirmed.
An `agent.tool_use` issued by the evaluation thread carries the `session_thread_id` of an internal thread that is not returned in the thread list. After the call is confirmed, the event is not kept in the event history.
For unattended evaluation, configure the read-only tools the evaluation needs (such as `Read`, `Bash`, `Glob`, and `Grep`) as `always_allow`.
### Agent turn errors
Only an Agent turn that ends normally is evaluated. When an Agent turn or a revision turn ends with an error after automatic retries are exhausted:
- The event stream emits `session.error` with `error.retry_status.type` set to `exhausted`, and no `session.status_idle`. The Session `status` returns to `idle`.
- The Outcome stays active: `result` is `pending` or `running`, and `completed_at` is `null`.
- Defining a new Outcome returns `409`. Sending `user.message` starts a new turn, which is still evaluated against the current Outcome's rubric when it ends.
- If no new turn starts for a period of time, the server reclaims the Outcome: `result` becomes `interrupted` and the event stream emits `span.outcome_evaluation_end`.
To end the Outcome immediately, [cancel the Session](#cancel-and-interrupt).
## Request constraints
| Constraint | Result |
| --------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `user.define_outcome` is sent in the same request as `user.message`, `user.tool_result`, or `user.custom_tool_result` | The whole request returns `400` and no event takes effect |
| One request contains several `user.define_outcome` events | Events are processed in order: the first takes effect, later ones return `409`, and the first is not rolled back |
| The Session is processing a turn (`running` or `rescheduling`) | Returns `409` |
| The Session already has an active Outcome (`result` is `pending`, `running`, or `evaluating`) | Returns `409` |
A Session can be `idle` and still have an active Outcome (see [Agent turn errors](#agent-turn-errors)). Before sending `user.define_outcome`, confirm that the Session is `idle` and has no active Outcome:
```bash
curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
-H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
| jq '{status, active_outcomes: [.outcome_evaluations[] | select(.completed_at == null)] | length}'
```
An Outcome can be defined only through the `user.define_outcome` event; it is not a field of Session creation. A Deployment can carry the event in `initial_events`.
## Retrieve deliverables
Files the Agent delivers with `DeliverArtifacts` belong to the current Session. A successful delivery emits [`agent.artifact_delivered`](/cloud-agents/api/sessions/schemas#agent-artifact-delivered-event) on the event stream.
```bash
# List the Session's files
curl -s "https://api.qoder.com/api/v1/cloud/files?scope_id=$SESSION_ID" \
-H "Authorization: Bearer $QODER_ACCESS_TOKEN"
# Get the download URL and download the file
curl -s "https://api.qoder.com/api/v1/cloud/files/$FILE_ID/content" \
-H "Authorization: Bearer $QODER_ACCESS_TOKEN" | jq -r '.url' | xargs curl -s -o report.html
```
- A revision turn that delivers again creates a new File. Use the `file_id` of the last `agent.artifact_delivered` to get the final version.
- A file can take a few seconds after delivery to appear in the list.
- The Agent may also return the result directly in `agent.message`. Clients should handle both cases.
For the endpoints, see [Delivering and retrieving session outputs](/cloud-agents/files#delivering-and-retrieving-session-outputs).
## Cancel and interrupt
Either of the following moves the active Outcome to `interrupted`:
- [Cancel the Session](/cloud-agents/api/sessions/cancel): `POST /api/v1/cloud/sessions/{session_id}/cancel`.
- Send `user.interrupt` without `session_thread_id`, which has the same effect as cancelling the Session.
To change the task while an Outcome is running, cancel first and then define a new Outcome.
## Full example
```bash
# 1. Define the Outcome
curl -s -X POST "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID/events" \
-H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"events": [
{
"type": "user.define_outcome",
"description": "Analyze this repository for performance problems and write the findings to /data/report.html",
"rubric": {"type": "text", "content": "/data/report.html exists; it lists the top 3 bottlenecks with evidence; no placeholder content"},
"max_iterations": 1
}
]
}'
# 2. Wait for the Outcome to reach a terminal state
until [ "$(curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
-H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
| jq -r '.outcome_evaluations[-1].completed_at // empty')" != "" ]; do sleep 5; done
# 3. Read the verdict
curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
-H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
| jq '.outcome_evaluations[-1] | {result, iteration, explanation}'
# 4. Wait for the wrap-up turn to end
until [ "$(curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
-H "Authorization: Bearer $QODER_ACCESS_TOKEN" | jq -r '.status')" = "idle" ]; do sleep 2; done
# 5. List the deliverables
curl -s "https://api.qoder.com/api/v1/cloud/files?scope_id=$SESSION_ID" \
-H "Authorization: Bearer $QODER_ACCESS_TOKEN" | jq '.data[] | {id, filename}'
```
## Recommendations
| Scenario | Recommendation |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| You only need a pass or fail verdict | `max_iterations: 1`; state the pass condition and the failure condition in the rubric |
| The Agent should revise based on the verdict | `max_iterations` greater than 1; evaluation cost grows with iterations |
| The work is done and you only need it verified | Defining an Outcome still starts an Agent turn; ask it in `description` to verify and fix the existing output |
| You only need to know whether something was delivered | An Outcome is not needed; use `agent.artifact_delivered` and the Session status |
## Errors
| HTTP status | When |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `400` | Invalid `rubric` structure; `max_iterations` outside 1–20; rubric larger than 64 KiB; a disallowed field such as `outcome_id`; sent in the same request as `user.message` or a tool result event |
| `404` | The Session does not exist; the file referenced by `rubric.file_id` does not exist |
| `409` | The Session is processing a turn; the Session already has an active Outcome |