Most AI-generated testing starts from the code: point a model at the diff, get tests back. Convenient, and backwards. Tests derived from the implementation inherit the implementation's blind spots — if the code misunderstood the requirement, the tests now confirm the misunderstanding. You get green checkmarks on wrong behavior.
Our pipeline starts from intent. The test plan draws on three sources: the ticket — what was supposed to change; the diff — what actually changed, with its new branches and failure paths; and the project's memory — what has gone wrong in that area before. Only the ticket defines correct behavior. The diff adds surface to check; it never gets a vote on what the result should be.
From that plan, the agent drives Playwright through the affected flows on staging and lands a verdict back in the ticket, with evidence, so a human reviews a conclusion instead of assembling one.
"With evidence" is doing real work in that sentence. A pass requires proof a human can inspect — for anything with a UI, the rendered screen itself, not a green log line. In its fullest form, the proof is a chaptered video of the run with a verdict card per case: reviewing means jumping to a chapter, not replaying a session. And it requires the database to agree: a page can render success while the wrong row gets written, and the right row can exist behind a broken screen. Either alone can lie; the pipeline checks both. Verdicts aren't binary either: a case blocked by the environment is recorded as blocked, with what would unblock it — never dressed up as a failure, never quietly skipped.
The unglamorous part nobody writes about is test data. Most of e2e isn't clicking buttons — it's getting an account into the right state first. Read-only runs can borrow an existing account through guarded impersonation; anything that mutates state gets a fresh one. And when the pipeline can't reach the state a case needs, it doesn't improvise on half-ready data — it marks the case blocked and says exactly what a human must prepare.
And the memory is maintained like code. Every project carries a file of QA gotchas: the traps that burned us once. When a run surfaces a new one, it becomes a pull request — reviewed, dated, merged. The pipeline gets smarter through the same process the codebase does.
The effect on the team is the interesting part. QA didn't shrink — it moved up a level. The whole loop is anchored in Jira: a run starts from a ticket and its verdict lands back in the same ticket, so this is how QA works here day to day — reviewing generated test plans and hunting the edge cases the pipeline wouldn't imagine, not hand-writing the fortieth checkout scenario.
All of it runs on top of a codebase already at 72% unit-test coverage. The pipeline doesn't replace that layer — it verifies what unit tests can't prove on their own: that the shipped behavior, end to end, matches the ticket.
None of this is a product we bought, and none of it is bound to a vendor. The whole methodology is a git repository of markdown skills: Claude Code runs it today, Codex can run it tomorrow, and the process doesn't change. That's the actual lesson. The process is the asset. The agent is just the executor.