Stopping two passing agent PRs from breaking main together

On October 5, 2026, two pull requests in Jarvis each passed their gate and merged 37 seconds apart. Together they stopped JarvisKit, the app’s shared Swift code, from compiling, and main stayed red for about 64 minutes. A gate verdict is a claim about one tree: the base it started from plus the PR. Merge two of them and main becomes a third tree that nobody tested.

The fix isn’t a better gate. It’s one landing path that checks the exact tree that will become main, under a lock, and then confirms main is that tree. I’ve learned that three times, in three harnesses.

37 seconds apart

Jarvis is the control plane that replaced Omnigent. In early October its build pipeline was bash, run by the babysitter — the agent session that supervises the builders. About 10 builders ran at once. Each PR’s gate tested it merged onto main as main stood when the gate started. Then the builder merged it by hand with gh pr merge --match-head-commit. That flag pins the PR’s head. It says nothing about the base the gate tested.

#2342 merged at 14:35:11 UTC and #2343 at 14:35:48. Each one added the same model and the same client call, in different files. Git saw no conflict. The compiler saw two declarations. What noticed it was other PRs’ gates, failing because main itself didn’t compile. The fix merged at 15:39.

The babysitter’s own account, written that afternoon: “The cause was my mistake. My pipeline brief told builders to merge by hand after a gate.” The brief was its own. The rule it was missing was mine, and I’d already paid for it once.

Solved once already

Omnigent hit the same problem from the inside, two months earlier. On August 8 its gate turned out to be testing GitHub’s pull/N/merge ref, which GitHub re-mints when the PR’s head is pushed, not when main moves. A PR could be tested as a merge into a main the repo had already left. The gate started building the merge itself.

On August 9 the agents found the window between gate and merge. Main moved about every 18 minutes against a gate of 205 seconds at the median, and nothing held the two together. The commit that closed it put the ship path behind one merge slot and said it plainly: “The expensive half of a merge queue already existed … Only the mutual exclusion was missing.”

On September 12 the gate went to batches, and a batch member could reuse the batch’s verdict only while main’s tree matched the tree the previous member left. Trees, not SHAs, because GitHub’s merge commit has a new SHA over the same tree. Five days later, gates running side by side in one checkout shared Git’s FETCH_HEAD, and two Jarvis PRs got red verdicts for code that wasn’t theirs. The fix on September 18 bound every gate to its own fetched refs.

So by mid-September I had a merge slot, a tree comparison, and a gate bound to the exact head it tested. None of it lived anywhere the next harness would read. The babysitter’s scripts started over without it.

Refuse, speculate, or lock

The first repair, on October 5, refused any merge where main had moved in the same area since the gate, and sent the PR back to re-gate. The cost showed at once. Overnight it landed 17 merges in 6 hours 20 minutes, and a third of gate runs ended refused. Over the next day 36% of gate-hours were passing gates thrown away because main moved first. iOS gates took 76 to 87 minutes, so a re-gate was expensive.

The agents tried the parallel answer next: a speculative queue that stacked each PR on top of the ones ahead of it. It ran for an hour and a half. 47% of its gates were cancelled, and merges fell to 2 an hour. With gates passing about 65% of the time, stacking deeper throws away work more often.

On October 7 I wrote down what I wanted. The smoke tests ran in about 5 minutes, so test the merged tree under a lock, one landing at a time. “Why not just have a lock,” I wrote, and I figured it “would give roughly 12 merges per hour.” I asked for batching the same night. The babysitter shipped three pieces: a check of the merged tree (build, lint, types, smoke) that measured 333 seconds, a lock around landing, and batching, so the PRs waiting on the lock share one check.

That instinct is older than the fleet. Around 2008 and 2009 I was a database administrator at Beckett Media, one of my first clients out of university. Doing individual transactions there “was slow, but then after that batching things up into batch transactions onto database was faster.” I’ve had the idea of micro-batches “for well over a decade,” and the reason hasn’t changed: “to be able to keep throughput going.”

The check caught the same failure on its first live run. A PR and the main it would land on both declared the same setup() method: a clean Git merge, and a redeclaration error at build. It never landed. From October 6 to 10, 85 of 366 merged-tree checks failed, 23%. Each was a PR that had passed its own gate and failed on current main. That’s an upper bound on what would have broken main, since a red main or a flaky test fails the check too.

Trees, not SHAs

On October 10 I rebuilt Jarvis as v2, and its pipeline launched without the merge train. Each PR got its own gate and one merge per 5-minute tick. 62 PRs piled up, 35 of them already through review. I’d solved that in v1, and I said so.

The train came back the next day with one addition. Before every landing, main must be exactly what the previous landing left. After every landing, main’s tree must equal the tree that was checked. From the train script, trimmed:

# after each landing, main's TREE must equal the gated prefix tree PFX[i]
# (GitHub's merge commit has its own SHA, so compare trees, not SHAs)
t_main=$(git -C "$SRC" rev-parse "origin/main^{tree}")
t_want=$(git -C "$W" rev-parse "${PFX[$i]}^{tree}")

That day v2 merged 315 PRs, and one train landed 22 of them in 2 minutes 51 seconds. On October 17 the guard fired for real: after the first of three landings, main’s tree didn’t match, and the train stopped before landing the other two.

Side doors

A rule about the landing path only holds if it’s the only path. On October 16 the babysitter wrote its own per-PR landers and ran 3 or 4 beside the train. 84 gates, 52 failed, 21 of those on merge conflicts between the landers themselves, 26 merges. The rule since then: one merge path per repository, and an audit that fails on any merge to main the train didn’t record.

The next day a PR landed on a scoped pass that never ran the tests of the code that depended on it. Main went red, and that one landing cost 70-plus minutes with no merges. A landing proof is now a full run of the exact tree that will become main. Fast feedback and proof are different jobs.

What serializing costs

The lock itself is cheap. What costs is work thrown away, or waiting.

v1’s best day, 320 merges, and v2’s best, 315, both came right after the train was in place. I don’t read that as the lock making the fleet faster. Build caching, more hosts, and an easier card mix changed in the same days. They do show that a serialized landing step didn’t stop the fleet from setting its best days.

With one person merging a few PRs a day, main rarely moves inside a gate’s window, and a green check on the PR is close enough to a green check on main. With 10 builders, 76-minute gates, and 131 merges on October 5 alone, “passed alone” says little about “passes together.” Agents make it worse in one specific way. Two cards written from the same spec each add the helper they need, and nobody remembers someone else just wrote it.

Why it keeps falling out

Each harness dropped the rule because each was a fresh start. “It’s hard when you try to start something from scratch to take all the learning and lessons and port them over,” and it stays hard “unless you’re very systematic about it.”

The systematic option has its own cost. A spec-first rewrite “causes issues of its own where you don’t see anything running for a long time,” and v2 is in that spot today. I “still don’t have … really anything that’s running in V2 that I can use on a day-to-day basis.” The other way is a minimal product used from day one and grown as needed, which is how v1 lost the rule in the first place. Either way the job is the same: “keep track of the lessons that you previously learned and find ways to bake it in whenever you can.” The babysitter keeps a numbered bake-in list of lessons for Jarvis to absorb. Entries 74, 75, and 204 are this one.

The standard answer

None of this is new. bors states the rule: “the exact integration of pull requests that end up in the main branch have already been tested.” The bors-ng README says it “implements a continuous-testing workflow where the main branch never breaks,” testing batches and splitting a failed batch. It now marks the project deprecated and points to GitHub’s merge queue. GitHub’s merge queue “creates temporary branches with a special prefix to validate pull request changes,” each built “with the latest version of the base_branch as well as changes from pull requests ahead of it in the queue.”

My speculative queue was the GitHub shape, and at a 65% pass rate it lost. The train that stuck is the bors shape: batch, check once, bisect on failure. What a fleet adds on top is the tree comparison after landing, and refusing any second path to main.

Still open

v2’s own daemon has a lander that refuses unless main is still at the expected commit and merges with a head guard. It isn’t wired into the running pipeline yet. Until it is, the tree check lives in the babysitter’s script, which is the kind of place this lesson has fallen out of before.

I also have no red-main series for v1, only the named incidents with their durations. The rule I’m holding: a verdict is about one exact tree, and every path to main carries that tree or doesn’t land.

Underneath

The Omnigent and Jarvis timings, counts, and code are from my fleets’ own records: Git history for merge times and daily counts, the pipeline’s event log for checks, refusals, and trains, and the commits that shipped each change. The bors, bors-ng, and GitHub merge queue pages were accessed October 29, 2026. Graydon Hoare’s post on the “not-rocket-science rule,” which bors links as the source of the rule, no longer resolves, so I don’t quote it.

More field notes

Start