On a per-trial timeout the trial is graded against a local workspace holding none of the agent's work, so it fails at contract.install regardless of what the agent actually achieved, and it still counts in the pass rate.
downloadWorkspace at agent.ts:200 receives abort.signal, which the timeout has already aborted, so it throws at its first remote command, before the rm and extract at :426. The catch-path retry at :215 is gated on !abort.signal.aborted and so is skipped. syncedWorkspace stays false, and being function-local it never reaches the event stream, so classifyAgentOutcome cannot see that the sync failed.
The guard for this already exists. run.ts:455 returns infra when a result reports success with zero usage, and its comment names the case exactly: "contract.install on a blank workspace the agent never saw". A timeout produces null usage, so hasZeroUsageResult is true there. The && !timeoutNote on that line is what lets it through.
I read this from source and have not observed it end to end, so it is worth confirming before acting.
Disclosure: AI-assisted.
On a per-trial timeout the trial is graded against a local workspace holding none of the agent's work, so it fails at
contract.installregardless of what the agent actually achieved, and it still counts in the pass rate.downloadWorkspaceatagent.ts:200receivesabort.signal, which the timeout has already aborted, so it throws at its first remote command, before thermand extract at:426. The catch-path retry at:215is gated on!abort.signal.abortedand so is skipped.syncedWorkspacestays false, and being function-local it never reaches the event stream, soclassifyAgentOutcomecannot see that the sync failed.The guard for this already exists.
run.ts:455returnsinfrawhen a result reports success with zero usage, and its comment names the case exactly: "contract.install on a blank workspace the agent never saw". A timeout produces null usage, sohasZeroUsageResultis true there. The&& !timeoutNoteon that line is what lets it through.I read this from source and have not observed it end to end, so it is worth confirming before acting.
Disclosure: AI-assisted.