when models improve they're getting more "one-shotting" capabilities right. so prompting it to "make it better" may yield something on a long time horizon (assuming updating models used). I mean it makes sense that this approach sucks by default and you need stopping conditions and stuff
but isn't that the point of high intelligence to solve more stuff with less input?
The stopping-rule section matches our experience uncomfortably well. We run a daily execution loop in production at leapd.ai — an agent that plans a founder's day, executes the tasks, and reports back — and the thing that finally made it converge wasn't a better verifier. It was giving up on the verifier as the stop condition and using the day boundary instead. The loop stops because the day ends, then re-plans tomorrow with the outcome as new state.
That's a worse loop by your Section 4 criteria and a much better system. It caps spend per cycle deterministically, and it moves the "was this actually done" judgment to the next planning pass, where there's more information than the agent had mid-task. Your Lighthouse run is the same failure in miniature: the agent knew at try 5, the evaluator overruled it 14 times. A cheap external clock would have beaten the expensive smart judge.
The failure mode we still haven't solved is verdict classification — an empty diff can mean "nothing needed changing" or "the agent silently failed," and those need opposite responses from the loop. Would love to see more written about that.
Test suites, type checkers, CI. None of that was written for agents, it just happened to be sitting there when they arrived, which is a big part of why coding loops converged first. Pushing this into design or research means somebody builds the checker before the loop can work, and that's slow, unglamorous work almost nobody is funded to do.
when models improve they're getting more "one-shotting" capabilities right. so prompting it to "make it better" may yield something on a long time horizon (assuming updating models used). I mean it makes sense that this approach sucks by default and you need stopping conditions and stuff
but isn't that the point of high intelligence to solve more stuff with less input?
The stopping-rule section matches our experience uncomfortably well. We run a daily execution loop in production at leapd.ai — an agent that plans a founder's day, executes the tasks, and reports back — and the thing that finally made it converge wasn't a better verifier. It was giving up on the verifier as the stop condition and using the day boundary instead. The loop stops because the day ends, then re-plans tomorrow with the outcome as new state.
That's a worse loop by your Section 4 criteria and a much better system. It caps spend per cycle deterministically, and it moves the "was this actually done" judgment to the next planning pass, where there's more information than the agent had mid-task. Your Lighthouse run is the same failure in miniature: the agent knew at try 5, the evaluator overruled it 14 times. A cheap external clock would have beaten the expensive smart judge.
The failure mode we still haven't solved is verdict classification — an empty diff can mean "nothing needed changing" or "the agent silently failed," and those need opposite responses from the loop. Would love to see more written about that.
Test suites, type checkers, CI. None of that was written for agents, it just happened to be sitting there when they arrived, which is a big part of why coding loops converged first. Pushing this into design or research means somebody builds the checker before the loop can work, and that's slow, unglamorous work almost nobody is funded to do.
The compiler that memorized its test inputs says it all. Though your four conditions still assume someone can write the target state down.
I turned the verification loop into a runnable tutorial using Python, now with 500+ GitHub stars: https://github.com/hardness1020/awesome-agent-architecture/tree/main/sections/21-loop-engineering
Le dance avec loupe