I run an AI pipeline that ships content automatically. Last month it shipped something that passed 121 automated tests the night it went out.
I want to be specific about what those 121 tests actually checked, because the gap between "121 green checks" and "this thing works" is the whole lesson. Every one of them asked a machinery question: did the job fire on schedule, did the API call return success, did the row land in the right table with the right columns filled in. Real questions, worth testing, I don't regret writing any of them. But not one of the 121 asked the only question that mattered: was the output any good.
It shipped green. Dashboards lit up, the schedule ran on time, rows appeared where they should. By every metric the suite tracked, the system worked exactly as designed. It was also producing content nobody would want to read, and nothing in the pipeline said so, because nothing in the pipeline was built to check that.
What actually caught it was me sitting down and reading the output by hand, the old fashioned way. That did more to protect the thing I'm building than the entire test suite, because reading the actual output is the only check that measures what you were trying to produce, not the plumbing that carries it.
It's an easy trap because it looks like rigor. 121 passing tests feels safer than 3. Green checkmarks are legible and easy to report. "Is it any good" is subjective, slow, resistant to automation, so it's the thing that gets skipped when you're racing to ship. I skipped it, and I'm the only one checking, which makes it worse, not better.
The fix wasn't more tests of the same kind. It was one different kind of check: nothing reaches me for approval anymore without a quality score attached that has nothing to do with whether the pipeline ran and everything to do with whether the output is worth someone's time.
If you're running any kind of automated pipeline, does anything in it actually check whether the output is good, or does it only check whether the machinery ran?