Testing When Claude Writes the Tests Too

Draft material for Ch 16 — “Testing When Claude Writes the Tests Too” (Part IV). Durable testing judgment the reader should carry, framed as engineering principles — what a trustworthy test looks like — not a catalog of what goes wrong. These principles predate AI and outlast any one tool. What changes in the AI era is the economics: Claude writes the tests cheaply and quickly, so the typing is no longer the cost. The scarce input becomes the reader’s judgment about whether the tests are worth trusting. This is the positive, principles-side companion to engineering-principles-catalog.md, and shares its altitude and voice.

Each principle uses the catalog’s shape in lighter form: the statement, plain language, why it matters, the opportunity with Claude, and a one-line handle to carry. Entries can be expanded to the full catalog template later.

The Principles

1. A test exists to catch a change in behavior — so it must be able to fail

The principle: A test earns its place only if it can go red. If breaking the code it covers leaves the test green, the test is decoration, not protection.

In plain language: A smoke detector that never beeps isn’t reassuring — it’s broken. A test that can’t fail tells you nothing, no matter how many you have.

Why it matters: A green suite feels like safety, but green only means “the tests agree with the code as written.” The value is in the red: the moment a future change quietly breaks something and a test stops you. A test that can’t fail can’t do that one job.

The opportunity with Claude: Claude will produce a passing suite on the first try, which is exactly when to be suspicious. Direct it: “break the behavior on purpose and show me a test goes red.” If none does, the tests describe the code instead of constraining it.

Carry: “If I break the code, does a test turn red?”

2. Test real behavior, not the implementation

The principle: Assert the observable outcome through the same interface a real caller uses. Don’t assert the internal steps the code happened to take.

In plain language: Test what the thing does, not how it’s wired inside. If you rewrite the insides but the behavior is identical, the tests should still pass.

Why it matters: Tests bound to internals break every time you refactor — punishing improvement — while still missing whether the behavior is actually right. Tests bound to behavior survive refactors and fail only when something a user would notice changes.

The opportunity with Claude: Claude, having just written the implementation, will reach for it when writing tests — mirroring the steps it just wrote. Give it the behavior contract instead: the inputs, the expected outputs, the error cases. Let it test the promise, not the plumbing.

Carry: “Would this test still pass if I rewrote the insides and kept the behavior?”

3. Exercise the real thing; mock only true externalities

The principle: Run the actual code path end to end where it matters. Reserve mocks for genuine boundaries you don’t control — a paid API, the clock, the network — not for the code under test.

In plain language: If you fake everything, all you’ve proven is that your fakes were called. Use the real components; stand in for only the things that are slow, costly, or outside your control.

Why it matters: Over-mocking produces tests that pass while the real system is broken, because the parts that actually interact were never run together. The bugs that hurt most live in the seams between real components — exactly what mocks paper over.

The opportunity with Claude: Claude defaults to heavy mocking because it’s easy and isolated. Push back: “exercise the real path; mock only the external service and the clock.” It will wire the real components together if you ask — that’s labor, not judgment.

Carry: “What real behavior is this test actually exercising?”

4. Same inputs, same result — control the sources of nondeterminism

The principle: A test must give the same verdict every time it runs. Control the clock, randomness, ordering, and any network or environment dependence so the result is a function of the inputs alone.

In plain language: A test that passes now and fails an hour later — or passes on your machine and fails in the cloud — isn’t telling you about your code. It’s telling you about the weather.

Why it matters: A flaky test poisons the whole suite: people learn to ignore red, and a real failure hides among the noise. Determinism is what makes red mean something.

The opportunity with Claude: Claude will happily use the real time, real random values, and whatever order things happen to run in. Name the nondeterminism and have it inject a fixed clock, a seeded generator, and explicit ordering. It knows the techniques; it just won’t apply them unprompted.

Carry: “If I run this 100 times, does it pass 100 times — here and in CI?”

5. Each run starts clean and stands alone — idempotent and isolated

The principle: Every test sets up its own state and tears it down, depending on no other test and leaving nothing behind. Running the suite twice gives the same result as running it once.

In plain language: Tests shouldn’t share leftovers. Each one brings its own ingredients, cooks in its own pot, and washes up after — so the order they run in, and how many times, never changes the outcome.

Why it matters: Shared or leftover state creates failures that appear only in certain orders or only on the second run — the hardest kind to diagnose. Isolation is also what lets tests run in parallel and lets a single failure be reproduced on its own.

The opportunity with Claude: Ask Claude for the isolation primitive — a fresh temp directory, an ephemeral database, a clean fixture per test — and it will build it. The judgment is insisting on it; the setup/teardown boilerplate is exactly the labor to hand off.

Carry: “Does this test depend on anything another test did — or on having run before?”

6. When a test fails, it should say what broke

The principle: A failure should report enough to diagnose without rerunning under a debugger: what was expected, what actually happened, and which case produced it.

In plain language: “Something’s wrong” wastes your time. “Expected 3, got 0, when the cart was empty” hands you the bug.

Why it matters: The point of a test is to shorten the distance between a regression and its fix. A bare pass/fail forces re-investigation; a good failure message is the investigation, already done.

The opportunity with Claude: Claude writes terse assertions by default. Ask for descriptive failures — expected-vs-actual, the input that triggered it, a one-line label per case. This costs nothing at authoring time and pays back every time something breaks.

Carry: “If this fails in six months, will the message alone tell me why?”

7. Fast enough to run constantly; thorough when it counts — tier the suite

The principle: Split tests by cost. A fast smoke tier runs on every change to give near-instant feedback; a slower, fuller tier (real services, end-to-end flows) runs before merge or release. The split is deliberate, not accidental.

In plain language: You want a quick check you’ll actually run a hundred times a day, and a deep check you run at the gates that matter. Forcing all testing into one speed means it’s either too slow to run often or too shallow to trust.

Why it matters: Tests only protect you if they’re run. A suite too slow to run gets skipped; a suite too shallow to catch real problems gives false confidence. Tiering keeps both feedback loops honest and bounds total runtime.

The opportunity with Claude: Claude can tag and organize tests into tiers and wire each tier to the right trigger. Tell it which feedback loop each test serves; it will arrange the rest.

Carry: “Is this fast enough to run on every change — and is the deep version running before release?”

8. A test isn’t done until it runs where everyone runs it

The principle: Tests must pass on the platforms your users (and collaborators) actually use, and must run automatically — in CI, on a clean checkout — not only on the author’s machine.

In plain language: “Works on my machine” is the oldest excuse in software. A test that only runs where it was written protects only the person who wrote it.

Why it matters: The machine that wrote the code is the one place a bug is least likely to show — it has all the right versions, paths, and quirks already. Running on a clean, automated environment is what catches the missing dependency, the wrong path separator, the assumption that didn’t travel.

The opportunity with Claude: Claude can set up the automated runner and the clean-environment matrix as readily as it writes the tests themselves. The judgment is deciding it must run everywhere it matters; the configuration is labor to delegate.

Carry: “Does this pass on a clean machine that isn’t mine — automatically?”

The thread

Eight principles, one instinct: a test is only worth what it can catch. When Claude writes both the code and the tests, the typing stops being the cost and the trust becomes the cost. Every principle here is a way of asking the same question — can this test fail, for the right reason, every time, where it matters? — and directing Claude to build the version that can, since the building is now the cheap part and the asking is the scarce one.

Found something wrong, unclear, or plainly disagreeable? Open an issue