What Two Days of Agent-Driven Development Actually Caught

Our own website had 222 HTML validation errors. It now has none. The interesting part is not the number, it is which of the errors a person would ever have found.

The errors

The site is Phoenix, seventeen static pages and seventeen articles. Running the W3C Nu HTML Checker over every URL in the sitemap produced 222 errors in five classes:

• 74 from an img tag carrying a sizes attribute beside a srcset built from x descriptors. HTML forbids that pairing. sizes was doing nothing anyway, because with x descriptors the browser selects by device pixel ratio.
• 68 from a LiveView attribute on a link and a script tag, on every page.
• 45 from skipped heading levels inside eight articles.
• 35 from a Matrix room link. A room alias starts with #, which is illegal in a URL fragment, so the link was malformed on every page that rendered the footer.

None of that is visible. Every page rendered correctly, returned 200, and scored 100 on Lighthouse. A validator error is not a bug you can see.

The one that mattered most was invisible twice

The first run checked the home page and found three errors. Running it across all 34 URLs found 222 in five classes. Two of the classes existed only on pages the home page could not reach.

Then the route list itself turned out to be the problem. Every check derived its list of pages from sitemap.xml. The login page is not in the sitemap. When the list came from the router instead, that page produced two more errors nobody had seen: a style block inside main, where HTML does not permit one, and a heading jump from h1 straight to h3.

Three separate checks all inherited the same blind spot, because they all derived their scope the same way. The gap was not in a fact. It was in a list.

The instruction that made it worse

The obvious fix for the heading errors is to stop using the site logo as the h1 and promote the page title. Doing exactly that makes the validator angrier. The page title becomes h1, the section headings stay h3, and now seventeen pages skip a level where before they merely used the wrong element.

Every heading outside the article pages had to move up one level together. The CSS scale moved with them, so h1 now carries what h2 carried, and headless screenshots of two pages against production came back byte-identical. The appearance did not change, which is how I know the scale shift was right rather than merely plausible.

Two of the measurements were lying

This is the part worth reading.

Every new test was checked by breaking the code it covered and confirming it went red. Three of those checks reported green for mutations that were really failures.

The first was file timestamps. The harness restored the original file with cp, which set its modification time to the same second as the existing compiled artefact. The build system compared the two, saw no difference, and skipped the rebuild. The tests then ran against the previous version of the source. Forcing a rebuild did not help, because it forced the development build and the tests read a different one.

The second was line numbers. Targeting a specific test by file and line ran whichever test now sat at that line. Inserting tests above shifts everything below, and nothing errors, because there is always a test at that line.

The third came later, watching a fix land. A loop polling the build status broke out immediately and reported failure. It was reading the status of the commit from before the fix, so it exited on precisely the state it existed to wait past.

All three named their subject by a coordinate that can move: a timestamp that collides, a line number that shifts, a commit that advances. Each was pointed correctly when it was written.

That is why all three failed the same direction. A coordinate that has moved still resolves to something, and the something is plausible. A name that had simply been wrong would have errored and cost nothing.

One test could not fail at all

A test compared the team page against its structured data. It asserted that the number of people in the schema equalled the number in the roster module, and both were read from the same module. Deleting a member changed both sides. The test stayed green.

It is now two tests. One compares the rendered page against the rendered schema, reading neither from the module, which catches a template that stops iterating. The other lists the four names as literals, which catches a roster that loses somebody. Breaking the code in each way now fails one and not the other, which is the only reason either is worth having.

What this says about working with agents

The agent found things nobody would have looked for, at a rate no person would sustain, and it also produced three checks that could not see their own subject and one test that could not fail. Both of those are the same capability.

The useful discipline is not reviewing the output more carefully. It is asking what a check would have to do to be wrong, and then arranging for it to be wrong on purpose once. A test that has only ever been green is a test whose first real execution nobody can question.

We now gate the build on zero validator errors, and the gate fails the build rather than reporting. Before trusting it, we pushed a branch that reintroduced one of the original errors and confirmed the build went red. On that same commit the test suite passed, so the red can only have come from the validator. Without that second half, a red proves the job ran and nothing more.