Our release automation reported success on every run for three months while doing nothing at all. Five releases shipped in that time, so nobody noticed.
What was supposed to happen
The site uses release-please. It reads conventional commit messages on the main branch, works out the next version, and opens a pull request that bumps the version file and the changelog. Merging that pull request is the release.
It runs on a Forgejo instance rather than GitHub. One part of release-please does not work there: the step that creates the release object after the pull request merges is a silent no-op. We knew that, and we had written a small workflow to cover it. That workflow watches the main branch, reads the version file, and creates the missing tag. The tag triggers the image build.
What actually happened
The last release pull request release-please opened was on 27 May. Every release
after that was tagged by our workaround. The automation itself opened nothing
for three months and reported success every time it ran.
The mechanism is one label. When release-please opens a release pull request it
marks it autorelease: pending. It clears that label from the same step that
creates the release object, which is the step that does nothing here. So the
label went on and never came off. release-please treats a merged pull request
still marked pending as a release in flight, and declines to open another.
One merged pull request from May, carrying a label, blocked every release after it.
The workaround is what hid it
This is the part I keep thinking about. The workaround worked. It cut the tag, the image built, the deployment went out. Five releases reached production exactly as intended.
So the observable symptoms of a broken release pipeline were: releases shipping normally, a green tile on every run, and the absence of a pull request nobody was waiting for. There was nothing to notice.
A workaround that fixes the symptom removes the pressure that would have found the cause. That is what a workaround is for, and it is also why this ran for a quarter.
Finding it without being able to read the log
Job logs are not retrievable through Forgejo’s API. Every route returns 404. The web interface has them, and only for somebody signed in.
So the diagnosis had to come from an experiment rather than from reading. The control was already there: the workflow had run four times that day, with the label present, and opened nothing each time. That is four trials with a known result.
Then one variable changed. Remove the label from the merged pull request from May. Trigger the workflow.
A release pull request appeared ten seconds later.
That is a stronger result than a log line would have been. A log says what the program reported about itself. The experiment says what the program does when one input changes, and the four runs before it say what it does when that input does not.
The reasoning that nearly buried it
The stale label had been considered and dismissed, on the grounds that it had been stuck since May while five further tags were cut. If the label blocked releases, releases would have stopped.
Those tags came from the workaround, not from release-please. The workaround exists precisely because release-please cannot tag here. So the tags were evidence the workaround worked and said nothing whatsoever about the thing they were being used to exonerate.
The fact that would have pointed straight at it was available and cheap: this repository has eight tags and zero release objects. Zero, ever. That is what “the step that creates release objects has never once succeeded” looks like from outside. Nobody asked, because nobody counts a thing they assume exists.
The fix, and why removing the label was not it
Removing the label unblocked the queue and would have re-blocked it on the next release, because the new pull request carried the same label and would keep it for the same reason.
The workaround already did the job the broken step could not. It now does the other half: once the tag exists, it flips any merged pull request still marked pending. It runs whether or not that push created the tag, so a label stuck by some future failure clears itself rather than waiting for someone to notice again.
Verified by watching it, not by reading it. Before the merge, the pull request was marked pending and the tag did not exist. After, the tag existed and the label read tagged. Then the next commit to the main branch produced a release pull request with nobody touching anything, which is the thing that had not happened since May.
What I would take from it
A green that cannot go red carries no information. This one had been incapable of reporting the failure it was having, because the failure was “did nothing”, and doing nothing successfully is indistinguishable from having nothing to do.
The question worth asking of any automated check is not whether it passes. It is what state of the world would make it fail, and whether that state has ever been tested.