We took our site from 222 HTML validation errors to zero, and then nothing held it there. A number you fix once decays. So the validator now fails the build.
Report or gate
A check that reports and stops nothing trains everyone to ignore it. After a month nobody reads the output, and the person who eventually does cannot tell a new error from one that has been sitting there since spring.
The same argument runs the other way for a check that fires during correct operation. A gate that is red while everything is fine is worse than no gate, because it teaches people that red means nothing.
So the two questions are whether a failure means something is actually wrong, and whether anyone is stopped by it. HTML validity answers both. The errors were real, none of them fired on correct markup, and the fix in every case was to correct the markup rather than to widen the rule.
Where it has to live
Our main branch requires exactly one status check to pass before a merge. Not every job in the pipeline: one named context.
That detail decides the design. A validator running as its own job would report a failure that does not block anything, because the branch protection rule does not know about it. It would be a report wearing a gate’s clothes.
So the validator runs inside the job that is required. It fails the build, and a failed build cannot merge.
Rendering the pages
The validator reads files. Something has to produce them.
A small task renders every public route to a directory by calling the application’s endpoint directly, rather than starting a server and fetching over HTTP. No port to pick, no waiting for readiness, no chance of validating a stale process. It exits non-zero if any route answers anything other than 200, so a broken page is a failure rather than a page the validator never sees.
Its list of routes comes from the router and the content, not from the sitemap. That is the whole reason the login page’s two errors surfaced: every previous check walked the sitemap, and that page is not in it.
Three things the container told me that I had assumed
The first version of the step was written from documentation. All three of its assumptions were wrong, and running it inside the actual CI image took about a minute to find out.
The image is Alpine. The validator’s Linux bundle ships its own Java runtime
built against glibc, which does not run on musl. Java is not in the image at
all, so it needed installing. And the version I pinned in the download URL does
not exist as a tag, because the project publishes every release under a rolling
latest tag.
That last one has a consequence. If there is no version to pin to, the URL always serves the newest build, and a gate that silently changes what it enforces is not a gate. So the pin is a checksum. If upstream publishes a new release, the step fails and names both hashes.
It fired about a day later. Three unrelated things went red at once, which is what a loud failure is supposed to look like, and the fix was to run the new version against the same pages, confirm it produced the same verdict, and then move the hash. That distinction is the entire value of the pin: it moves the hash, not the standard. Bumping it without checking would have turned a pin into a delay.
Proving it can fail
The build passed with the validator in it. That proves the job ran. It does not prove the validator ran.
So we pushed a branch that reintroduced one of the original 222 errors and opened a pull request. The build went red.
That alone is not conclusive either, because a red build could come from the test suite. On that same commit, locally, the suite passed and the validator exited non-zero. Nothing in the tests covers that attribute, so the red can only have come from the validator.
A gate that has only ever been green is a gate whose first real execution nobody can question. It costs one throwaway branch to find out, before the first time it matters rather than during it.
What it caught immediately
Widening the route list past the sitemap found a page nothing had ever checked,
carrying two errors: a style block where HTML does not permit one, and a
heading level skipped because that page’s only heading was its title.
The gate paid for itself before it ran once.