Make the CI fix agent's failure-context pre-fetch fail loudly - #1296
Open
ytdb-ci-bot[bot] wants to merge 10 commits into
Open
ytdb-ci-bot[bot] wants to merge 10 commits into
ytdb-ci-bot[bot] wants to merge 10 commits into
Conversation
added 10 commits
September 4, 2026 17:00
The agent dispatched for run 33870767133 received an empty context: run.json
`{}`, jobs.json `{"jobs":[]}`, every list `[]`, and "No failed-job logs
available." The runner's own diagnostics date the whole "Gather CI failure
context" step at 37 ms, which is far too fast for six REST calls to GitHub.
The self-hosted Hetzner image ships no `gh`, and the step installed only `jq`.
Every `gh` call therefore exited 127. Each fetch was written as
gh run view ... > run.json 2>/dev/null || echo '{}' > run.json
so `2>/dev/null` hid "command not found" and the `||` branch wrote a
placeholder. The step exited 0 and reported success.
The silence was the expensive part. A placeholder and a real empty result are
byte-identical, and the prompt told the agent that "everything about the
failure has been pre-fetched" and that an empty associated-prs.json *means* the
failure was a direct develop push. So the agent could not detect the gap, and
read absent data as fact.
Installing `gh` alone would only make the next outage - an expired token, a
rate limit, a log GitHub has already discarded - fail the same silent way.
So the placeholder path stays, because a partial context is still worth
investigating, but it can no longer pass for success: every datum records its
outcome in fetch-status.json (`ok`, `failed`, `unavailable`, `partial`), a
derived datum starts from its parent's status rather than claiming `ok` over an
empty file, each failure is logged with the reason gh gave, and the step ends
with one warning naming everything degraded. The prompt now points the agent at
that ledger and tells it to treat a non-`ok` datum as absent.
Missing prerequisites are also fatal up front now, where the cause is one line
of log away instead of six steps downstream.
The degraded-data summary assigns from a command substitution, and the step runs under `bash -e`, where that construct propagates failure. A jq error while reading fetch-status.json would therefore have killed the step at exit 2 - immediately before the one line that says the context is incomplete, and after everything else had already been written. Verified both ways against the extracted block: with the guard the step exits 0 and prints the summary, without it the step exits 2.
The two aggregates (annotations, pr-comments) marked themselves `partial` as soon as one per-item call failed, which also caught the case where every call failed and nothing came back at all. That distinction decides what the agent does. `partial` says the file is usable but incomplete, so the agent reads it; `failed` says the file is a placeholder, so the agent does not. Reporting `partial` over an aggregate that received nothing therefore points the agent at an empty file it has been told to trust - the same absent-versus-empty confusion the ledger exists to remove. Counting attempts and failures separately gives the three-way rule: none failed is `ok`, all failed is `failed`, anything between is `partial`. Zero failures over zero attempts stays `ok`, because a run with no failed jobs genuinely has no annotations. Four new test functions cover the rule and the gaps around it: an aggregate where every call fails, one where a pr-comments call fails, a run with no failed jobs, and a fetch that exits 0 with a body jq cannot parse.
Six review dimensions ran over the previous commits. The substantive findings,
and what each one cost:
The prompt tested ledger entries by file name (`run.json`) while the ledger is
keyed without the extension (`run`). Every such lookup returns null, null is
not usable, and the prompt's own rule then discards a perfectly healthy datum -
the original bug in mirror image. The keys are now stated explicitly and a test
fails on any file name used as the subject of a ledger predicate.
`gh api` writes the HTTP error body to stdout before it checks the status code,
and the aggregate loop appended stdout straight to the merge input. One 404
therefore pushed {"message":"Not Found"} in beside the arrays, `jq -s add`
refused to add an object to an array, and the merge failed - discarding the
annotations that DID arrive and grading the datum `failed`. That is the exact
data loss `partial` exists to prevent, and the old test passed only because the
stub was kinder than GitHub. Each response is now buffered and merged only
after it parses.
Neither aggregate passed `--paginate`, so a run with more than thirty failing
tests was graded `complete: true` over a truncated first page - a false
completeness claim, which is the class of defect this ledger exists to remove.
`partial` carried a definition and no action rule, so the blanket
"not ok means absent" rule discarded partial aggregates the step had gone out
of its way to preserve. The ledger now derives `usable` and `complete` per
datum, and the prompt branches on those two booleans rather than mapping a
four-value enum to an action.
Step 1 offered "infrastructure failure -> no-fix" as the only one-hop exit, and
a blank context looks exactly like one from the agent's seat. That inference is
now prohibited by name, with a fifth category and an investigation path that
terminates in a stated sentinel status.
Step 0c told the agent to hunt for a rival fix branch and then keep
`existing_pr` as null. Committing onto that branch makes the publish step push
successfully and then fail opening a second PR for the same head, reporting a
red job for a fix that shipped. The instruction now takes the fresh-branch path
and reports the suspicion instead.
`/tmp/fix-agent-summary.md` was a bare name on a persistent runner, so an agent
that crashed before writing left the previous run's summary to be reported as
this one's. It moves to the job-scoped result directory.
Also: the ledger's own build failure is now loud rather than a silent `{}`; id
enumeration failures no longer grade an empty aggregate `ok`; all-failed
aggregates report `failed` rather than `partial`; fetch errors are scrubbed of
CR and query strings, so remote error text cannot forge a workflow command or
echo a signed blob URL; a non-numeric run id fails up front; a sentinel naming
`develop` or `main` is refused before the push; and the context directory is
purged rather than merely created.
The suite grows to 28 test functions. It now executes the prerequisite step
too, rather than pattern-matching its text, and pins what is fetched - run id,
repo, endpoints, paging - not only how the result is graded. Six mutations that
previously survived are now caught, including a wrong run id, a wrong
repository, and reverting the buffering fix. Folded into the required CI Status
gate, because a path-filtered advisory workflow cannot block a merge and this
repo has already been bitten by exactly that.
`gh api --paginate` concatenates one JSON array per page on stdout, so a buffered item body can hold several top-level arrays. Both the per-item parse check and the merge already handle that, but nothing proved it, and paging was added specifically to stop a truncated first page being graded `complete`. A regression there would trade one silent truncation for another.
The publish step pushes the sentinel's branch with an App token carrying contents: write, and it pushes before it opens the pull request, so a bad branch value lands even when PR creation then fails. The existing validation checks only the character set, which accepts `develop` and `main`. That matters because much of what the agent reads is text any GitHub user can write: PR titles, PR bodies and issue comments all reach it through existing-fix-prs.json and pr-comments.json. An injected claim that the fix branch for this failure is `develop` should not be able to turn a fix into a direct push that bypasses review and every CI gate. Both branch names are now refused before the push. Tests cover the gate's whole input surface - missing sentinel, unknown status, shell metacharacters in the branch, empty branch, non-numeric PR number, and an embedded newline - since this is the step that decides whether a write-scoped token gets used at all.
The status ledger is accumulated as tab-separated key/status/reason rows, so a TAB in remote error text adds a field and a NEWLINE adds a row. Removing the scrub proves the consequence rather than implying it: the ledger comes back missing keys, and a forged `run<TAB>ok` row relabels a failed fetch as a healthy one - a GitHub error message rewriting the very record that says whether GitHub answered. `tr -c '[:print:]' ' '` already handled it, because TAB is not in `[:print:]`, but nothing said so and nothing would have noticed its removal. Two mutations are now caught: dropping the control-character squeeze (9 checks) and dropping the query-string strip that keeps signed blob URLs out of the log (3 checks).
Four dimensions re-reviewed the fixes and found more in them. The substantive ones: `jq -e` takes its exit status from its LAST output and emits one output per input, so the per-item parse check graded only the final page of a `--paginate` response. An earlier page that was valid JSON of the wrong type passed the check and was then dropped by the merge's type filter, publishing `complete: true` over data that never made it - the same false-completeness claim one level down from the bug this all started with. A single error object with gh exiting 0 read as "this job genuinely has no annotations". Slurping and requiring every page to be an array fixes all three shapes, which are now pinned as scenarios. `gh pr list --limit 50` gave no signal when it truncated, and a truncated list graded complete would let the agent open a second fix PR over one it never saw. A list returned at exactly the limit is now `partial`. Step 8's PR menu still offered the branch choice Step 0c now forbids, and it was the only line pairing with `FIX_READY_POSSIBLE_DUPLICATE` - so the status pointed the agent back at the path that makes the publish step push successfully and then fail opening a second PR. Three of the review gate's selectable agents do not exist under those names: `.claude/agents/` has `review-bugs` and `review-concurrency` rather than a combined `review-bugs-concurrency`, and `review-test-quality` covers what the table called `review-test-behavior` and `review-test-completeness`. An agent following that table asks for names the Agent tool cannot resolve, and the dimension is silently skipped while the gate reports itself satisfied. I hit this myself running the gate on this change. A test now fails on any name with no file behind it. The /tmp cleanup step ran before the two Zulip steps, so the files those write were never removed. It moves to the end of the job and covers them. `uuidgen` lives in uuid-runtime, which is not essential on the image that was already missing jq and gh; under `bash -e` a failing substitution in an assignment would have killed the run at the starting gun. Replaced with `$$`. Every GitHub read is now wrapped in `timeout 120`. Each failure this step handles is a fast error, so an unbounded call that stalls would consume the whole 12-hour job budget and publish no ledger at all; bounded, the grading records it as failed and the agent still gets a usable partial context. Also: the publish step consumes the branch and PR number the gate validated instead of re-reading them raw from the sentinel; an empty pr_title is defaulted rather than left to fail `gh pr create` after the push has landed; a non-fast-forward push rebases once and retries rather than losing a fix to a race; the write-up is posted as a comment on the existing-PR path, where pr-body.md was previously never read; and the gate-diagnosis fallback chain terminates instead of demanding numbers the agent cannot get. Step 0 was rewritten rather than patched: two reviewers independently found it at four times the section cap with the usable/complete rule stated four times. It is now three labelled units, the flag semantics appear once as the decision table, and the top-level `complete` and `degraded` fields the agent's first command prints are defined - `degraded` includes the usable `partial` data, so the previous silence there invited discarding a datum the table says to use. The suite reaches 37 test functions and now runs shellcheck at `info` over all four extracted shell blocks, closing the gap where 350 lines of shell inside YAML were invisible to every linter.
The non-numeric id guard in the aggregate loop reached `continue` before `tried` was incremented, so a skipped id left no trace in the failed/tried arithmetic. `failed -eq 0` then took the zero-over-zero branch that exists for a run with genuinely no failed jobs, and a run whose ids were all non-numeric published an empty file as `ok` and `complete: true`. That is the original bug's shape one level down: the agent told "GitHub has no annotations" about jobs the step declined to ask about. A skipped id now counts as a failure. The prompt also still made a data file the subject of a ledger predicate in two places -- "`logs.txt` can be `ok`", "`run.json`'s `event` ... when that datum is usable" -- where the keys are `logs` and `run`. The check meant to catch that was matching a fixed set of English phrasings, so it caught one drift in twelve and missed both of these. It is now a proximity rule: no bare data-file name on a line that also mentions the ledger, with two named exemptions. Three pieces of hardening had no coverage and now do. `timeout` writes nothing to stderr when it fires, so a timed-out fetch recorded a failure with an empty reason and logged "Fetch 'logs' failed:" with nothing after the colon -- the defect the aggregate path already calls out, reintroduced by the timeout wrapper itself. A `hanging_logs` scenario drives the expiry path, which needed the budgets to be env-overridable. A `long_error` scenario pins the 300-byte cap, without which a whole HTML error page would land in the file the agent reads first. `realistic_api_404` existed but nothing asserted the property it was built for, so reverting the response buffering left the suite green. It is now asserted directly, including that gh's error body reaches no data file while the ledger's reason may still quote its message. Also: the log fetch and the paginated aggregates get a 600-second budget rather than 120, because both are bulk transfers and `logs` is the most valuable datum to lose to a premature kill; the open-PR limit rises to 200 and is declared once, so the truncation guard cannot drift from it; the Zulip POST is bounded, since it now sits above the cleanup step and a stall there would cancel the cleanup rather than run it; a redacted query string leaves a `?<redacted>` marker instead of vanishing; and the sentinel gate asserts its own exit status on every input, since it runs `if: always()` and is the only path to publishing. 41 test functions, ~10s. Four more mutations verified caught.
Each of these guards my own new code, and each was verified only by reading it. The two `|| true`s on the apt calls are documented as load-bearing: without them a failing apt aborts the step at apt's own exit status, with a dpkg-lock message and no statement of which tool is missing or why it matters. The stub apt now takes an exit status, so the guarded path is exercised rather than asserted. The context directory is purged rather than only created, because $RUNNER_TEMP persists across jobs on the self-hosted pool and a file an earlier version of this step left there is indistinguishable to the agent from this run's input. A scenario now pre-seeds the directory. `missing_from_ledger` and its `::error::` existed for a code path that forgets to `record`, and nothing had ever produced one. A jq wrapper drops a row from the status TSV on its way into the ledger builder, so the self-check is exercised on the only input that can trigger it. 44 test functions. All three mutations verified caught.
Test Count Gate ResultsTolerance: 5% drop allowed per module Overall: ✅ 35623 tests (baseline: 35623, +0)
|
Coverage Gate ResultsThresholds: 85% line, 70% branch ✅ No changed Java files — coverage gate skipped. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ghas a hard prerequisite of the CI Failure Fix Agentjob, which had been assuming a binary its runner image does not ship.
ledger, so an empty context file can no longer pass for a successful empty
answer.
stub
gh, and folded it into the requiredCI Statusgate.Context
This PR fixes the mechanism that was supposed to diagnose
run 33870767133.
It does not fix that run's test failure, because the failure is still unknown —
see What is not fixed here.
A prior branch addresses the same cause
origin/ci-fix/20260804-162533-fix-agent-gh-prereq(6 commits, 4 weeks old,+456/-71 on this same workflow) diagnosed the identical root cause — its first
commit is titled "Fix CI fix agent's blind context pre-fetch" — and introduces
its own
fetch-status.jsonprotocol. It never merged, which is why the agentwas still blind a month later.
I could not tell whether its pull request is still open:
existing-fix-prs.jsonwas one of the placeholders, so duplicate detection was exactly as blind as
everything else. Per this PR's own new Step 0c guidance I took a fresh branch
rather than committing onto a branch whose PR number I cannot know, and I am
flagging the overlap here instead. Please reconcile the two before merging —
close one, or rebase this onto that branch.
Root Cause
The agent dispatched for the failed nightly integration run received an empty
failure context:
run.jsonwas{},jobs.jsonwas{"jobs":[]}, every listwas
[], andlogs.txtreadNo failed-job logs available.The runner's own diagnostics time the whole "Gather CI failure context" step at
37 ms — 16:44:44.5759Z to 16:44:44.6130Z. Six REST calls to GitHub cannot
complete in 37 ms.
command -v ghreturns nothing on the self-hosted Hetzner image, and the stepbefore the pre-fetch installed only
jq. Everyghcall therefore exited 127,and each fetch was written as
so
2>/dev/nulldiscardedgh: command not foundand the||branch wrote aplaceholder. The step exited 0 and reported success.
The silence was the expensive part. A placeholder is byte-identical to a real
empty result, and the prompt told the agent that "everything about the failure
has been pre-fetched" and that an empty
associated-prs.jsonmeans thefailure was a direct
developpush. The agent could not detect the gap, so itwould read absent data as fact.
Installing
ghalone would leave that intact: the next expired token, ratelimit, or log GitHub has already discarded fails the same silent way.
Changes
.github/workflows/ci-failure-fix-agent.ymlPrerequisites.
Install jqbecomesInstall CLI prerequisites (jq, gh).Both tools are detected, installed when absent, then verified; a tool still
missing fails the job there, where the cause is one line of log away, instead
of degrading the agent's input six steps later.
The ledger.
Gather CI failure contextnow publishesfetch-status.jsonbeside the data:
{ "complete": false, "degraded": ["annotations"], "missing_from_ledger": [], "data": { "annotations": { "status": "partial", "usable": true, "complete": false, "reason": "1 of 2 annotations fetches failed: HTTP 404" } } }statusis one ofok,partial,failed,unavailable;usableandcompleteare derived from it so the agent branches on two booleans ratherthan mapping a four-value enum to an action. The placeholder path stays,
because a partial context is still worth investigating — what it can no longer
do is look like success. Specifically:
::warning::rather than discarded,and carried into the ledger as
reason;annotationsover afailed
jobsfetch isunavailable, notokover an empty file;failed, notpartial—partialmeans "some of it arrived", and grading an empty file
partialwould pointthe agent at data it had been told to trust;
jqparse failurecannot run the loop zero times and grade the empty result
ok;gh apiwrites the HTTP error body to stdout before it checks the status code, so
appending in place pushed
{"message":"Not Found"}in beside the arrays,jq -s addrefused to add an object to an array, and one 404 discarded everyannotation that had arrived;
--paginate, so a run with more than thirty failingtests is not graded
completeover a truncated first page;failure raises an
::error::with an all-degraded fallback;::warning::naming everything degraded.Log hygiene. Fetch errors pass through a
scrubhelper that strips querystrings and squeezes non-printables. Both matter: a bare CR ends a line for the
runner, so remote error text beginning
::stop-commands::would silence thewarnings this step exists to emit, and a TAB would add a field to the ledger's
TSV — a GitHub error message could forge a
run<TAB>okrow and relabel afailed fetch as healthy. Query strings go because
gh run view --log-failedfollows a redirect to blob storage and a failure at that hop prints a signed
URL, which is not a registered secret and so is not masked.
Input validation. A
failed_run_urlwhose tail is not numeric fails thestep up front, and a sentinel naming
developormainis refused before thepush. The publish step pushes with an App token carrying
contents: writebefore it opens the PR, and much of the agent's context is text any GitHub
user can write, so an injected branch claim must not be able to become a direct
push past review.
Stale-state fixes. The context and result directories are purged rather
than merely created, and the agent's summary moves from the bare
/tmpname tothe job-scoped result directory — on a persistent runner an agent that crashed
before writing left the previous run's summary to be reported as this one's. A
cleanup step removes the prompt and the multi-megabyte JSON log after the
artifact upload.
The prompt. Updated wherever it depended on the old contract: the
no-GitHub-access preamble, Step 0a (rewritten around the ledger, with the key
convention stated, the placeholder shapes named, and the degenerate-ledger
cases enumerated), Step 0b (repurposed from a no-op to "note what arrived"),
Step 0c's duplicate detection, Step 1's classification, Step 1a's gate
diagnosis, Step 2's missing-test-name fallback, Step 7's PR-body template,
Step 8's summary template, and the Important Rules list. Two of those are
behavioural rather than cosmetic:
and a blank context is indistinguishable from one from the agent's seat. That
inference is now prohibited by name, and a fifth category carries an
investigation path terminating in a stated sentinel status.
existing_prnull. Committing onto that branch makes the publish step push successfully and
then fail opening a second PR for the same head — a red job for a fix that
shipped. The instruction now takes the fresh-branch path and reports the
suspicion instead.
.github/scripts/ci-fix-context-test.py(new)Extracts four of the workflow's shell blocks — the prerequisite step, the
pre-fetch, the sentinel gate and the publish step — and executes the first
three under
bash -ewith a stubgh. 44 test functions cover: every fetchsucceeding,
ghabsent, a failed parent datum, a run payload with noheadSha, aggregates failing in part and in whole, a run with no failed jobs,an id the step refuses to use, a body
jqcannot parse, an unenumerable idsource, a realistic
gh api404 with its body on stdout, paged and mixed-typepaged responses, a truncated open-PR list and the boundary one under it, a
stalled fetch that must be killed and explained, a 200 KB error body, a
multi-line CR-bearing error, control characters aimed at the ledger's TSV, a
non-numeric run id, a stale context directory from an earlier dispatch, a
failing apt, an unrecorded ledger key, and the sentinel gate's whole input
surface.
It also pins what is fetched — run id, repository, endpoints,
--jsonfields, paging, timeout wrapping — not only how the result is graded;
cross-checks that the prompt's ledger references use real keys rather than
file names; asserts every agent name the review gate selects exists in
.claude/agents/; and runs shellcheck atinfoover all four blocks, closingthe gap where ~400 lines of shell inside YAML were invisible to every linter.
Stdlib only:
actions/setup-pythonprovides a bare interpreter with no PyYAML,so the workflow is parsed by text.
.github/workflows/ci-fix-context-tests.yml(new) andmaven-pipeline.ymlThe new workflow gives fast path-filtered feedback, following the existing
youtrack-notifier-tests.ymlpattern. Because a path-filtered workflow is nota required status check — and this repo has already had a schema violation
reach
developthat way — the suite is also folded into the requiredCI Statusjob, beside the workflow TOC check and before theshould_buildearly exit.
What is not fixed here
The nightly test failure itself is undiagnosed. With no logs, no test name
and no annotations, there was nothing to work from but inference. What I could
establish:
develophas already moved past the tested tree.08eaf8c966(YTDB-1294)landed at 10:41;
face6d13ca(YTDB-1289) landed at 15:17 while the failingrun was still going, and it rewrote 5 of the production files YTDB-1294 had
touched, including both
SelectExecutionPlannerandIndexOrderedPlanner. Arun on
face6d13cawas queued behind the failing one.test-linuxortest-windowsfailing,which narrows it to six legs.
pr-integration-tests.ymlalready ran green on PR YTDB-1294: Keep Gremlin order records missing the key #1295, so a deterministicfailure there is unlikely. The nightly-only surfaces are the JDK 25
integration legs, and the Windows and ARM legs — those two pass
-P ci-integration-testswithclean package, which setsyoutrackdb.test.env=ciand so runs the unit suite on disk storage,something no PR ever does.
clean verify -P ci-integration-tests -Dyoutrackdb.test.env=cionthis branch reproduced nothing on Linux x86 / JDK 21 (see the test plan).
Whoever picks the failure up should read
jobs.jsonfrom the next dispatch,which will now actually contain the failing leg.
Motivation
CI failure on develop: https://github.com/JetBrains/youtrackdb/actions/runs/33870767133
The dispatched fix agent could not diagnose it, because the mechanism meant to
hand it the evidence failed silently and told it the evidence was complete.
Every future nightly failure inherits that blindness until this is fixed.
Test plan
a wrong repository, reverting the response buffering, reverting the
unslurped
jq -epage check, dropping--paginate, dropping thecontrol-character scrub, removing the protected-branch refusal, dropping
the apt guards, dropping the context purge, and hardcoding
missing_from_ledgeremptyinfoover all four extracted shell blocksclean verify -P ci-integration-tests -Dyoutrackdb.test.env=ciinstruction completeness, prompt design, workflow consistency, test
quality; three verification rounds)
never identified, because the context this PR fixes was empty. See
What is not fixed here.
coverage-gate.pyskipsKnown gaps
Three review rounds ran across seven dimensions. Both round-3 reviewers
reported no blockers. What I chose not to fix, all of it pre-existing and
outside the pre-fetch:
degraded path now writes one before its long reproduction, but the normal
path's first commit happens at Step 7, so a budget kill anywhere in
verification or review loses the work with no sentinel and no summary.
step 6.9 either side of it, so its only termination is the job timeout.
develop, sothey see the other PR's commits as if they were the agent's, and the
coverage gate can fail on lines this agent never touched.
share one error message, and the
gh pr commentguard is silent when thesentinel carried no body file.
Each fails closed — into a reported
no-fix, never a bad push or a silentlylost fix — which is why I left them rather than widening this change further.