Outcome Evaluation

An Outcome sets a gradable acceptance goal on a Session: description states the task and rubric states the acceptance criteria. After the Agent finishes its work, a separate evaluation thread grades the output against the rubric and returns a verdict such as satisfied or failed. For the recommended end-to-end usage, see Outcome Best Practices.

How it works

Defining an Outcome starts an Agent turn with description as its input. When the turn ends, the evaluation thread grades the output. If the output does not pass, the verdict is fed back to the Agent for revision until it passes or the iteration limit is reached.

Define the Outcome

Send user.define_outcome with description and rubric. The server generates outcome_id.

Agent works

The Agent produces its output from description. When the turn ends normally, evaluation starts directly and the Session stays running.

Evaluation

The evaluation thread grades the output against the rubric. The event stream emits span.outcome_evaluation_start and span.outcome_evaluation_ongoing.

Revision

When span.outcome_evaluation_end.result is needs_revision, the Agent starts a revision turn based on the verdict. The client does not need to send an event.

Terminal state

The Outcome reaches satisfied, failed, max_iterations_reached, or interrupted, and completed_at is set. After the evaluation thread returns a verdict, the Agent runs one wrap-up turn to write its final reply, and the Session then returns to idle.

A Session can have only one active Outcome at a time. After an Outcome reaches a terminal state, you can define another one on the same Session; the Session keeps the earlier evaluation records.

Write a rubric

A rubric is a Markdown document that lists the acceptance criteria one by one. The evaluation thread grades each criterion separately, so more specific criteria produce more stable verdicts.

# Performance report rubric

## Deliverables
- /data/report.html exists
- The report contains throughput and latency figures, with at least one chart each

## Quality of conclusions
- The top 3 bottlenecks are listed explicitly
- Each bottleneck cites a code location or profiling data as evidence

## Exclusions
- No placeholder content, TODOs, or sample data

The evaluation thread can read only the rubric, the sandbox, and the Agent's reply. It cannot read description. Put any expected values or source data the criteria depend on into the rubric.

Writing tips

Make every criterion verifiable. Write "the CSV contains a numeric price column", not "the data quality is good".
Require concrete evidence. For example a fetched page, a traceable computation, or a file:line reference.
Describe the goal, not the steps. If a command prescribed by the rubric is unavailable, the evaluation cannot proceed.
State what is not acceptable. For example "mirrors, reposts, and search snippets are not accepted as evidence".
Specify the feedback format. Require a per-criterion pass or fail with the gap.
State what to ignore. For example formatting, or extra deliverables unrelated to the criteria.
If you have no rubric yet, ask a model to analyze a known-good output and turn the analysis into a rubric.

With vague criteria the evaluation thread tends to return satisfied; overly strict criteria add evaluation iterations. Tune the rubric with max_iterations: 1 first and confirm that one iteration produces the correct verdict.

To reuse a rubric across Sessions, upload it as a file and reference it with {"type":"file","file_id":"file_..."}.

Define an Outcome

POST /api/v1/cloud/sessions/{session_id}/events

{
  "events": [
    {
      "type": "user.define_outcome",
      "description": "Deliver a runnable performance report that identifies the top 3 bottlenecks",
      "rubric": {
        "type": "text",
        "content": "Must produce /data/report.html; include throughput and latency figures; list the top 3 bottlenecks with evidence"
      },
      "max_iterations": 1
    }
  ]
}

Field	Type	Required	Description
`description`	string	Yes	The task description, used as the input of the Agent turn
`rubric`	object	Yes	The acceptance criteria, used only by the evaluation thread. Either `{"type":"text","content":"..."}` or `{"type":"file","file_id":"file_..."}`; unknown fields are not allowed
`max_iterations`	integer	No	Maximum number of evaluation iterations, an integer between 1 and 20. Defaults to 3

user.define_outcome immediately starts a billable Agent turn. Put the task in description and only the acceptance criteria in rubric.
The rubric (text or file content) is limited to 64 KiB; a larger rubric returns 400. When rubric.type is file, the referenced file must be readable.
Every evaluation iteration consumes credits. A larger max_iterations allows more evaluations in the worst case.
outcome_id (prefix outc_) is generated by the server and cannot be included in the request.

Evaluation state

Each Outcome has one entry in the outcome_evaluations array of the Session object, returned by Get Session, List Sessions, and Search Sessions.

Field	Type	Description
`type`	string	Always `outcome_evaluation`
`outcome_id`	string	Outcome ID with the `outc_` prefix
`description`	string	The `description` passed when the Outcome was defined
`iteration`	integer	Current evaluation iteration, starting at 0
`result`	string	Current state. See the table below
`explanation`	string \| null	Explanation from the most recent evaluation; `null` before the first evaluation completes
`completed_at`	string \| null	When the Outcome reached a terminal state; `null` until then

Values of result:

Value	Description
`pending`	Defined; the Agent has not started working
`running`	The Agent is working or revising
`evaluating`	The evaluation thread is grading
`satisfied`	Terminal. All criteria are met
`max_iterations_reached`	Terminal. The output still did not pass after the maximum number of iterations
`failed`	Terminal. The evaluation thread determined that the task cannot satisfy the rubric (for example `description` contradicts the rubric), or it returned no verdict
`interrupted`	Terminal. The Outcome was cancelled, interrupted, or reclaimed by the server

Evaluation events

The record of each evaluation iteration is available on the event stream (List events or Stream events):

Event	Fields	Description
`span.outcome_evaluation_start`	`outcome_id`, `iteration`	An evaluation iteration started
`span.outcome_evaluation_ongoing`	`outcome_id`, `iteration`	The evaluation is in progress. The interval is not fixed; do not base a timeout on it
`span.outcome_evaluation_end`	`outcome_id`, `iteration`, `result`, `explanation`, `outcome_evaluation_start_id`, `usage`	An evaluation iteration ended

In addition to the terminal values, span.outcome_evaluation_end.result can be needs_revision, which means the iteration did not pass and iterations remain. When the last iteration does not pass, the value is max_iterations_reached.
usage.credits is the credits consumed by the iteration. usage is omitted when the iteration has no metering record.
When an Outcome is interrupted before evaluation starts, the event stream contains only span.outcome_evaluation_end with result set to interrupted. There is no matching span.outcome_evaluation_start, and outcome_evaluation_start_id is an empty string.

Session status during an Outcome

The server switches between the Agent turn, evaluation, and revision without emitting session.status_idle or session.status_running. From the moment the Outcome is defined until the wrap-up turn ends, the Session stays running, and sending user.message or user.define_outcome returns 409. completed_at is set when the evaluation thread returns a terminal verdict, which is before the wrap-up turn ends:

Purpose	Condition
Read the verdict	`outcome_evaluations[].completed_at` is not `null`
Read the Agent's final reply, or send the next event	Session `status` is back to `idle`

One Outcome produces several agent.message events: one for the Agent turn, one for each revision turn, and one for the wrap-up turn. The final reply is the one after the terminal span.outcome_evaluation_end. interrupted, and a failed caused by the evaluation thread returning no verdict, have no wrap-up turn. The event stream can still carry the following status events during an Outcome. They do not mean the Outcome has ended:

Event	Cause	Action
`session.status_idle` with `stop_reason.type` set to `requires_action`	A tool call is waiting for confirmation	See Tool confirmation
`session.status_rescheduled` followed by `session.status_running`	Automatic model request retry	None. See Automatic model request retries

Tool confirmation

The evaluation thread inherits the Agent's tool permissions so that it can verify the output in the sandbox. A tool configured as always_ask pauses for confirmation whether the Agent or the evaluation thread calls it. The flow is the same as the Pending Action Flow:

The event stream emits agent.tool_use with evaluated_permission set to ask, and session.status_idle with stop_reason.type set to requires_action.
The client sends user.tool_confirmation with the event ID from stop_reason.event_ids.
The evaluation continues after confirmation.

While waiting for confirmation, the Session status remains running and outcome_evaluations[].result is running or evaluating. A pending action does not time out, so the evaluation does not continue until it is confirmed. An agent.tool_use issued by the evaluation thread carries the session_thread_id of an internal thread that is not returned in the thread list. After the call is confirmed, the event is not kept in the event history. For unattended evaluation, configure the read-only tools the evaluation needs (such as Read, Bash, Glob, and Grep) as always_allow.

Agent turn errors

Only an Agent turn that ends normally is evaluated. When an Agent turn or a revision turn ends with an error after automatic retries are exhausted:

The event stream emits session.error with error.retry_status.type set to exhausted, and no session.status_idle. The Session status returns to idle.
The Outcome stays active: result is pending or running, and completed_at is null.
Defining a new Outcome returns 409. Sending user.message starts a new turn, which is still evaluated against the current Outcome's rubric when it ends.
If no new turn starts for a period of time, the server reclaims the Outcome: result becomes interrupted and the event stream emits span.outcome_evaluation_end.

To end the Outcome immediately, cancel the Session.

Request constraints

Constraint	Result
`user.define_outcome` is sent in the same request as `user.message`, `user.tool_result`, or `user.custom_tool_result`	The whole request returns `400` and no event takes effect
One request contains several `user.define_outcome` events	Events are processed in order: the first takes effect, later ones return `409`, and the first is not rolled back
The Session is processing a turn (`running` or `rescheduling`)	Returns `409`
The Session already has an active Outcome (`result` is `pending`, `running`, or `evaluating`)	Returns `409`

A Session can be idle and still have an active Outcome (see Agent turn errors). Before sending user.define_outcome, confirm that the Session is idle and has no active Outcome:

curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  | jq '{status, active_outcomes: [.outcome_evaluations[] | select(.completed_at == null)] | length}'

An Outcome can be defined only through the user.define_outcome event; it is not a field of Session creation. A Deployment can carry the event in initial_events.

Retrieve deliverables

Files the Agent delivers with DeliverArtifacts belong to the current Session. A successful delivery emits agent.artifact_delivered on the event stream.

# List the Session's files
curl -s "https://api.qoder.com/api/v1/cloud/files?scope_id=$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN"

# Get the download URL and download the file
curl -s "https://api.qoder.com/api/v1/cloud/files/$FILE_ID/content" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" | jq -r '.url' | xargs curl -s -o report.html

A revision turn that delivers again creates a new File. Use the file_id of the last agent.artifact_delivered to get the final version.
A file can take a few seconds after delivery to appear in the list.
The Agent may also return the result directly in agent.message. Clients should handle both cases.

For the endpoints, see Delivering and retrieving session outputs.

Cancel and interrupt

Either of the following moves the active Outcome to interrupted:

Cancel the Session: POST /api/v1/cloud/sessions/{session_id}/cancel.
Send user.interrupt without session_thread_id, which has the same effect as cancelling the Session.

To change the task while an Outcome is running, cancel first and then define a new Outcome.

Full example

# 1. Define the Outcome
curl -s -X POST "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID/events" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "events": [
      {
        "type": "user.define_outcome",
        "description": "Analyze this repository for performance problems and write the findings to /data/report.html",
        "rubric": {"type": "text", "content": "/data/report.html exists; it lists the top 3 bottlenecks with evidence; no placeholder content"},
        "max_iterations": 1
      }
    ]
  }'

# 2. Wait for the Outcome to reach a terminal state
until [ "$(curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  | jq -r '.outcome_evaluations[-1].completed_at // empty')" != "" ]; do sleep 5; done

# 3. Read the verdict
curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  | jq '.outcome_evaluations[-1] | {result, iteration, explanation}'

# 4. Wait for the wrap-up turn to end
until [ "$(curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" | jq -r '.status')" = "idle" ]; do sleep 2; done

# 5. List the deliverables
curl -s "https://api.qoder.com/api/v1/cloud/files?scope_id=$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" | jq '.data[] | {id, filename}'

Recommendations

Scenario	Recommendation
You only need a pass or fail verdict	`max_iterations: 1`; state the pass condition and the failure condition in the rubric
The Agent should revise based on the verdict	`max_iterations` greater than 1; evaluation cost grows with iterations
The work is done and you only need it verified	Defining an Outcome still starts an Agent turn; ask it in `description` to verify and fix the existing output
You only need to know whether something was delivered	An Outcome is not needed; use `agent.artifact_delivered` and the Session status

Errors

HTTP status	When
`400`	Invalid `rubric` structure; `max_iterations` outside 1–20; rubric larger than 64 KiB; a disallowed field such as `outcome_id`; sent in the same request as `user.message` or a tool result event
`404`	The Session does not exist; the file referenced by `rubric.file_id` does not exist
`409`	The Session is processing a turn; the Session already has an active Outcome

​How it works

​Write a rubric

​Define an Outcome

​Evaluation state

​Evaluation events

​Session status during an Outcome

​Tool confirmation

​Agent turn errors

​Request constraints

​Retrieve deliverables

​Cancel and interrupt

​Full example

​Recommendations

​Errors

How it works

Write a rubric

Define an Outcome

Evaluation state

Evaluation events

Session status during an Outcome

Tool confirmation

Agent turn errors

Request constraints

Retrieve deliverables

Cancel and interrupt

Full example

Recommendations

Errors