Evaluation

Evaluate the state the agent leaves behind.

Responses are evidence. The result is what happens after customers, admins, operators and time react to the agent.

A scenario fixes the starting conditions

A scenario combines an exact checkpoint with an agent assignment, horizon, simulation configuration and evaluators.

Checkpoint: Starting state, event queue and clock.
Assignment: The work and information visible to the agent.
Horizon: How long the world runs before final evaluation.
Evaluators: Immediate, delayed and aggregate success conditions.

Run a matched agent comparison

const start = await world.checkpoints.create({
  name: "support-quarter-start",
});

for (const candidate of [agentA, agentB]) {
  const branch = await world.forks.create({
    from: start.id,
    name: candidate.name,
  });

  const access = await branch.interfaces.connect("zendesk", {
    scopes: ["read_tickets", "write_tickets"],
  });

  await candidate.run({
    task: "Own the support queue for 90 simulated days.",
    baseUrl: access.baseUrl,
    token: access.accessToken,
  });

  await branch.clock.advance({ days: 90, mode: "until-idle" });

  const report = await branch.evaluate({
    metrics: ["retention", "sla_breach_rate", "backlog"],
    invariants: ["refunds_match_ledger"],
  });

  await report.export("json");
}

Each candidate inherits the same customers, support admins, billing operators, records, pending schedules and simulated clock. Only the tested agent changes.

Outcome evaluation follows the complete run

01StartRestore the exact checkpoint

02OperateAgent, customers, admins and operators mutate one world

03AdvanceRelease delayed events and reactions

04ScoreCompare final state and outcomes

This allows a locally correct action to fail later: a promise may never fire, a refund may duplicate, or a temporary backlog reduction may produce higher churn.

Evaluators inspect different evidence

Evaluator	Checks	Example
Invariant	Rules that must always hold	Total refunded never exceeds captured payment.
State predicate	Required final records or relationships	Ticket resolved and audit entry linked to the refund.
Delayed check	State after time and actors react	Scheduled follow-up fires only if the balance is still due.
Outcome metric	Aggregate business result	Retention, margin, backlog or SLA over 90 days.
Trace check	How the result was reached	No privileged endpoint or forbidden mutation was attempted.

Forks make comparisons fair

Two agents begin from the same checkpoint, including pending events, hidden conditions and actor state. Only the intended experimental variable changes.

CheckpointIdentical past and pending future

Agent A / Agent BMatched world configuration

Outcome diffState, violations and business metrics

A fresh generated copy is weaker than a fork. It may look similar while differing in hidden actor state, pending schedules or historical conditions.

A run produces inspectable artifacts

Starting definition version, seed and checkpoint.
Agent calls, responses, errors and timing.
Simulated-user actions and scheduled events.
Before/after state diff and invariant results.
Immediate, delayed and aggregate outcome metrics.