Outcome Best Practices

This page describes the recommended way to use Outcomes and walks through a complete example. For fields and mechanics, see Outcome Evaluation.

Flow overview

An Outcome that passes evaluation runs in this order:

The client sends user.define_outcome with the task in description and the acceptance criteria in rubric.
The Agent works from description and the Session enters running.
The evaluation thread grades the output. The event stream emits span.outcome_evaluation_start, span.outcome_evaluation_ongoing, and span.outcome_evaluation_end.
span.outcome_evaluation_end.result is satisfied, and outcome_evaluations[].completed_at is set on the Session object.
The Agent writes its final reply and the Session returns to idle.

Task and acceptance criteria

description is the input of the Agent turn and rubric is used only by the evaluation thread. Neither side can read the other:

Content	Location	Visible to
Task, deliverable path, output format, source data	`description`	Agent
Acceptance criteria, expected values, failure condition	`rubric`	Evaluation thread

If the task is written only in rubric, the Agent cannot read it.
The evaluation thread cannot read description. Put any expected values or source data the criteria depend on into rubric, and avoid criteria such as "check against the task description".

How to write acceptance criteria:

Type	Example
Deliverable location	`/data/report.html` exists
Specific content	Contains throughput and latency figures, with at least one chart each
Evidence	Each bottleneck cites a code location or profiling data
Failure condition	`failed` when the file is absent from the sandbox and the Agent states it cannot finish

Each criterion should describe one thing. Split "complete", "runnable", and "evidenced" into three criteria instead of combining them.

Evaluation iterations

max_iterations is the upper bound on the number of evaluations. Every evaluation consumes credits.

Value	When to use
`1`	You only need a pass or fail verdict. If the evaluation does not pass, the Outcome reaches `max_iterations_reached`
`2`–`3` (default 3)	The Agent should revise based on the verdict. A revision turn starts automatically after `needs_revision`
Greater than `3`	The task needs several revisions to converge

Tune the rubric with max_iterations: 1 first. Raise the limit only after one iteration produces the correct verdict.

When to send

Send user.define_outcome in its own request, and only when the Session is idle and has no active Outcome:

curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  | jq '{status, active_outcomes: [.outcome_evaluations[] | select(.completed_at == null)] | length}'

You can send the event when status is idle and active_outcomes is 0. See Request constraints.

Detect completion

The Session stays running for the whole Outcome. Choose the condition by purpose:

Purpose	Condition
Read the verdict	`outcome_evaluations[].completed_at` is not `null`
Read the Agent's final reply, or send the next event	Session `status` is back to `idle`

Read explanation together with result. explanation contains the per-criterion assessment and can be shown to users or written to logs. On the event stream, a session.status_idle with stop_reason.type set to requires_action means a tool call is waiting for confirmation, not that the Outcome has ended. See Tool confirmation.

Retrieve deliverables

When the rubric requires a file, ask the Agent in description to write it under /data/ and deliver it with DeliverArtifacts. A successful delivery emits agent.artifact_delivered on the event stream.

A revision turn that delivers again creates a new File. Use the file_id of the last agent.artifact_delivered to get the final version.
A file can take a few seconds after delivery to appear in the file list.
The Agent may also return the result directly in its reply. Clients should handle both cases.

For the endpoints, see Delivering and retrieving session outputs.

Example: order summary

In this example the Agent computes the order count, the total amount, and the top-spending customer from an order CSV, writes them to /data/summary.json, and delivers the file. The Outcome then evaluates whether the result is correct.

1. Create the Agent

The evaluation thread inherits the Agent's tool permissions. Configure the tools the evaluation needs as always_allow so that the Outcome runs without manual confirmation.

AGENT_ID=$(curl -s -X POST "https://api.qoder.com/api/v1/cloud/agents" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "order-summary-agent",
    "model": "auto",
    "system": "You are a data assistant. Produce exactly the files the user asks for.",
    "tools": [{
      "type": "agent_toolset_20260401",
      "enabled_tools": ["Bash", "Read", "Write", "Glob", "Grep", "DeliverArtifacts"],
      "configs": [
        {"name": "Bash",  "permission_policy": {"type": "always_allow"}},
        {"name": "Read",  "permission_policy": {"type": "always_allow"}},
        {"name": "Write", "permission_policy": {"type": "always_allow"}},
        {"name": "Glob",  "permission_policy": {"type": "always_allow"}},
        {"name": "Grep",  "permission_policy": {"type": "always_allow"}},
        {"name": "DeliverArtifacts", "permission_policy": {"type": "always_allow"}}
      ]
    }]
  }' | jq -r '.id')

2. Create the Session

SESSION_ID=$(curl -s -X POST "https://api.qoder.com/api/v1/cloud/sessions" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"agent\": \"$AGENT_ID\", \"environment_id\": \"$ENVIRONMENT_ID\", \"title\": \"order summary\"}" \
  | jq -r '.id')

3. Define the Outcome

The task and the source data go in description; the expected values go in rubric. max_iterations is 2, which allows the Agent to revise once based on the verdict.

cat > define.json <<'EOF'
{
  "events": [{
    "type": "user.define_outcome",
    "description": "Below is an order CSV. Compute the number of orders, the total amount, and the customer with the highest spend, write them to /data/summary.json as {\"order_count\": <int>, \"total_amount\": <number>, \"top_customer\": <string>}, then deliver the file with DeliverArtifacts. Finish with a one-sentence reply.\n\norder_id,customer,amount\n1001,alice,120.50\n1002,bob,80.00\n1003,alice,35.25\n1004,carol,210.00\n1005,bob,60.00",
    "rubric": {
      "type": "text",
      "content": "# Order summary rubric\n\nThe expected values below were computed from the source data in advance.\n\n## Deliverable\n- /data/summary.json exists in the sandbox (read the file; do not rely on the Agent's reply)\n- Its content is valid JSON with exactly the fields order_count, total_amount, top_customer\n\n## Values\n- order_count == 5\n- total_amount == 505.75 (0.01 tolerance)\n- top_customer == \"carol\"\n\n## Verdict\n- Any criterion unmet: needs_revision, listing each gap and how to fix it\n- The file is absent from the sandbox and the Agent states it cannot finish: failed"
    },
    "max_iterations": 2
  }]
}
EOF

curl -s -X POST "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID/events" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d @define.json

4. Read the verdict

until [ "$(curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  | jq -r '.outcome_evaluations[-1].completed_at // empty')" != "" ]; do sleep 3; done

curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  | jq '.outcome_evaluations[-1] | {result, iteration, explanation, completed_at}'

Example response:

{
  "result": "satisfied",
  "iteration": 0,
  "explanation": "Read /data/summary.json directly. It is valid JSON with exactly three keys: order_count=5, total_amount=505.75, top_customer=\"carol\" — matching every rubric criterion. Agent reply values are consistent with the file but were not relied upon; the file itself is the evidence.",
  "completed_at": "2026-09-24T03:15:54.986863Z"
}

5. Read the final reply

The wrap-up turn may still be running when completed_at is set. Wait for the Session to return to idle, then read the last agent.message:

until [ "$(curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" | jq -r '.status')" = "idle" ]; do sleep 2; done

curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID/events?limit=100" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  | jq -r '[.data[] | select(.type == "agent.message")][-1].content[0].text'

6. Retrieve the deliverable

FILE_ID=$(curl -s "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID/events?limit=100" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" \
  | jq -r '[.data[] | select(.type == "agent.artifact_delivered")][-1].file_id')

curl -s "https://api.qoder.com/api/v1/cloud/files/$FILE_ID/content" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN" | jq -r '.url' | xargs curl -s

Output:

{"order_count": 5, "total_amount": 505.75, "top_customer": "carol"}

7. Archive the Session

curl -s -X POST "https://api.qoder.com/api/v1/cloud/sessions/$SESSION_ID/archive" \
  -H "Authorization: Bearer $QODER_ACCESS_TOKEN"

Event stream

Event order for this example when the first evaluation passes (span.model_request_* and agent.thinking omitted):

session.status_running
user.define_outcome
agent.tool_use                     Write
agent.tool_use                     DeliverArtifacts
agent.artifact_delivered           summary.json
agent.message                      Agent turn reply
span.outcome_evaluation_start      iteration=0
span.outcome_evaluation_ongoing
span.outcome_evaluation_end        result=satisfied
agent.message                      final reply
session.status_idle                stop_reason.type=end_turn

If the Agent's first output does not pass, span.outcome_evaluation_end has result set to needs_revision, followed by the revision turn's agent.tool_use, a new agent.artifact_delivered, and the evaluation events with iteration=1. The Session then holds two summary.json files, and the last agent.artifact_delivered refers to the final version.

Checklist

The task, deliverable path, and output format are in description; the acceptance criteria are in rubric.
rubric does not depend on the content of description; expected values or source data are written into rubric.
Every criterion can be verified on its own, and the failure condition is stated.
The first release uses max_iterations: 1 to confirm that one iteration produces the correct verdict.
The read-only tools the evaluation needs are always_allow, or the client handles requires_action.
user.define_outcome is sent in its own request, only when the Session is idle and has no active Outcome.
Retries of the same request carry an Idempotency-Key.
completed_at is used to detect that the Outcome has ended; the client waits for the Session to return to idle before reading the final reply or sending the next event.
explanation is read and recorded.
The deliverable is retrieved with the file_id of the last agent.artifact_delivered.

​Flow overview

​Task and acceptance criteria

​Evaluation iterations

​When to send

​Detect completion

​Retrieve deliverables

​Example: order summary

​1. Create the Agent

​2. Create the Session

​3. Define the Outcome

​4. Read the verdict

​5. Read the final reply

​6. Retrieve the deliverable

​7. Archive the Session

​Event stream

​Checklist