Agent Reliability & Evaluation
How do we know a coding agent has actually completed its task? A workflow connecting the mission contract, the builder’s handoff, deterministic checks, a separate evaluator’s decision and explicit acceptance.
From task to evidence-backed verdict
Workflow schematic · not a performance measurement- Step 1: Mission contract
- Step 2: Agent execution
- Step 3: Builder handoff
- Step 4: Deterministic checks
- Step 5: Separate evaluator
- Step 6: Recorded verdict
The unit of analysis is the work the agent leaves behind — commits, state, handoff and evidence — not its final message.
01 / Purpose
The problem: a convincing completion is not proof
When an agent writes code, its final response may say a mission is complete even when requirements are missed, evidence is uncommitted or a critical rule has not been checked. A useful evaluation must examine the work itself and record what was actually verified.
This work emerged from building AgroUnu, a farm-management platform developed with bounded coding-agent missions and a separate evaluation process.
02 / Method
What the evaluation checks
- 01Written mission contract and acceptance criteria
- 02Static and deterministic domain checks: types, schemas, invariants, transitions
- 03Integration and journey tests against disposable infrastructure
- 04Deliberately seeded contradictions and adversarial cases
- 05Git, state and handoff inspection by a separate evaluator
- 06Critical vetoes and an explicit verdict: accept or reject
03 / Observed failure
A rejection that the process surfaced
SYS-01 was the mission that built the agent-control system itself. Its evaluator rejected it, then rejected the first repair as well.
Excerpt from recorded evaluator report
“only five of six required seeded contradictions fail deterministically”
SYS-01 evaluator report, 27 October 2026 (private). The same report found that a transition could pass both validators without an evaluation verdict. Repair cycle 1, the same day, passed 5 of 7 scenarios and detected all 11 seeded contradictions, but was rejected on a critical veto: acceptance could depend on uncommitted files.
Rejected · twice
The builder’s completion was not treated as acceptance.
Internal evaluation by a separate AI agent session; not third-party certification.
04 / Evidence model
Evidence before claims
Each mechanism leaves a record. These records exist in the programme repository; none is published yet, and each would need review and sanitising before release.
| Evidence class | Question it answers | Publishing requirement |
|---|---|---|
| Mission contract | What was the agent asked to deliver? | Sanitised contract or excerpt |
| Builder handoff | What revision, state and evidence did it leave? | Provenance-checked handoff record |
| Deterministic checks | Which invariants and seeds were tested? | Commands, configurations, pass/fail results |
| Evaluator decision | Why was the result accepted or rejected? | Criteria, scenario table and dated verdict |
Read the AgroUnu case study ↗Explore other engineering work ↗
05 / Scientific & operational scope
What this does not establish
AgroUnu is pre-pilot and its recorded evaluations use synthetic farm scenarios. The evidence is one mission and two evaluation rounds. It does not establish frontier-model post-training performance, general reliability across agent architectures, or third-party certification.
The source code, reports and handoffs are private. Test counts alone do not establish production readiness.
The next meaningful investigation is a bounded public replay set: measure evaluator detection, false alarms and cost against explicit baselines.