Demo, all content is generated
Question

Is it pointless to let the same AI write the code and the tests?

Open · 1483 views · asked by lindiwe_n · edited

Maybe a dumb question. I asked Claude Code to add tests to my invoicing app and it wrote 40, all pass. But it also wrote the code. If it misunderstood what the app should do, won't the tests just have the same misunderstanding? What do developers do about this?

What I’ve tried

Read the tests myself but I can't really judge if they're good.

Comment
Not a dumb question at all. marco_py · edited

3 answers

marco_py · edited

Not a dumb question, it's the right worry. The tests protect against future changes breaking current behavior, which is still very valuable. They do not prove the current behavior is correct. That part has to come from you.

Cheap ways to add yourself into the loop:

  • Write the test names yourself, in plain words: "invoice total includes 21% VAT", "credit note makes total negative". Let the AI fill in the code.
  • Put real numbers from a real invoice you checked by hand into a test.
  • Break the code on purpose (change 21 to 12). If no test fails, that rule isn't tested.
Comment
Did the 'break it on purpose' thing, changed VAT to 12 and 38 of 40 tests still passed. So that answers my question. lindiwe_n · edited
wes_codes · edited

Also: a second, separate session reviewing the tests is better than nothing. "Here are the requirements [paste], here are the tests. Which requirements are not tested?" It doesn't share the first session's assumptions, so it catches some of them. Not a replacement for your own examples though.

Comment
katja_s · edited

Late, but to add: the term for Marco's third point is mutation testing, and there are tools that do it automatically (Stryker for JS/TS). They change your code in small ways and report which changes no test noticed. It's slow, so run it once in a while, not on every commit.

Comment
Ran it, 'mutation score 41%'. Humbling. Working on it. lindiwe_n · edited