Software factory loops, inner, outer and meta
Three feedback loops for delivering software, improving agent harnesses and checking the evaluations. Definitions, evidence and failure modes explained.
A software factory is a repeatable system that turns an accepted specification into verified, releasable software. In an agentic factory, AI agents carry out parts of that work through tools. The factory also supplies the environment, constraints, checks and release authority around them.
Running that system raises three different questions. Did this code change meet its specification? Would a different agent setup deliver better results? Are the evaluations measuring the results we actually need? We use inner, outer and meta loop to distinguish those questions.
The three loops at a glance
- Inner loop: changes the product. Input: an accepted specification and the current code. Output: a verified change, or evidence that it is not ready to release.
- Outer loop: changes the agent harness. Input: delivery failures and evaluation results. Output: a versioned change to how agents work, supported by comparative evidence.
- Meta loop: changes the evaluation method. Input: evidence that the tests or scoring miss something important. Output: a revised task set, grader or measurement protocol.
Inner loop: deliver a specified change
A specification defines the intended behaviour and constraints. Acceptance criteria make success observable. A regression is an unintended loss of behaviour that previously worked. These are different checks. A feature can meet its new acceptance criteria while breaking something elsewhere.
The delivery cycle is specification, implementation, verification, review and authorised release. Failed checks return evidence to implementation. A failed specification check may require a clarified requirement rather than another code edit. The feedback must identify which part is wrong.
Verification needs more than a green unit test. A unit test checks a small piece in isolation; an integration test checks components working together; a user-journey check exercises the behaviour through the real interface. Choose checks for the change and its risks. An independent review then examines the proposed change against the specification and the applicable standards.
The release gate is a decision point with explicit conditions and an authorised decision-maker. Completing a coding attempt does not itself satisfy that gate. The useful output is the change together with evidence, such as the exact revision tested, results, reviewer findings and any remaining limitations.
Outer loop: improve the harness
An agent harness is the surrounding software that manages an agent run through instructions, context, available tools, an execution environment, orchestration and checks. Changing a harness does not necessarily change the model or train its weights. The same model can behave differently with different tools, context or stopping rules.
Anthropic distinguishes a trace, the recorded sequence of an agent run, from its outcome, the resulting state. A confident final message is part of the trace. Whether the requested change actually works is an outcome question. This distinction matters when diagnosing failures. See Demystifying evals for AI agents.
The outer loop starts with a failure hypothesis. Perhaps the agent cannot find the relevant module, a tool hides useful errors, or progress is lost between sessions. Inspect the trace and environment before choosing an intervention. A longer prompt is only one possible change.
- Run the current harness as the baseline on a defined task set.
- Make a targeted, versioned change that addresses the diagnosed cause.
- Rerun comparable tasks and inspect both scores and failures.
- Check for regressions and validate on tasks held back from tuning.
- Adopt the change only if the evidence supports it under the agreed cost and reliability constraints.
This is the pattern described in LangChain's harness improvement guide. Its optimisation and holdout split gives the candidate somewhere to improve and somewhere separate to demonstrate whether that improvement transfers.
Long-running work also needs durable state. Anthropic's long-running agent guidance describes structured setup, incremental work and progress records that help a later session continue. Those are harness design choices, separate from the feature the agent is building.
Meta loop: improve how improvement is judged
An evaluation is a structured measurement of performance on defined tasks. A task contains an input and success criteria. A trial is one attempt at that task. A grader applies a scoring rule to the result. Repeated trials matter because an agent can succeed once and fail on another attempt. These terms follow Anthropic's evaluation guide linked above.
We use meta loop for revising that measurement system. Its trigger is a mismatch between evaluation results and useful performance. For example, a grader might accept a plausible explanation without checking the resulting files, or a task set might contain only easy cases. The change belongs in the evaluation, because the measurement itself is deficient.
- Task coverage: which real demands and failure conditions are represented?
- Grader validity: does the scoring rule check the intended outcome, and does human inspection agree with it?
- Reliability: are there enough trials to distinguish a consistent gain from a lucky run?
- Comparability: are the model version, tools, starting environment and resource budgets recorded and controlled?
- Contamination: has information from the supposedly held-out tasks influenced the harness being tested?
Overfitting means an improvement depends too closely on the tasks used to develop it. Leakage occurs when a solution, expected answer or held-out failure feeds back into tuning. Once a held-out set repeatedly guides revisions, it no longer provides the same independent check. Reserve fresh cases for subsequent validation.
Freeze the evaluation protocol for a baseline-versus-candidate comparison. If the grader changes halfway through, rerun both versions under the revised protocol. Research on evaluating harness evolution also stresses matching feedback and inference budgets. Extra attempts and extra feedback can improve a score without establishing that the harness itself is better.
What to record for a useful comparison
We recommend recording the harness revision, model identifier, task-set version, environment, grader version, trial count and resource limits. Report success rate alongside cost, elapsed time and human interventions. Break results down by task category so an overall gain cannot hide a serious regression.
Keep the three decisions separate. A product change passes its delivery checks. A harness change earns adoption through a controlled comparison. An evaluation change needs evidence that it measures the intended behaviour more faithfully. Each loop can inform the others, but changing the target while scoring an attempt makes the result hard to trust.