10

I inherited a 12-year-old billing module with zero tests and a maze of nested conditionals. Every time I asked Claude to "refactor this to be cleaner" it happily rewrote the control flow and silently changed a rounding edge case. On legacy code that is how you get paged at 3am.

The fix was to make the refactor illegal until it has pinned the current behavior. I force it to write characterization tests that assert whatever the code does today, run them in my head against the messy version, and only then propose changes that keep every test green.

The prompt below has caught two behavior-drifts for me already. Anyone have a cleaner way to make it flag which tests it is unsure about?

THE PROMPT
You are refactoring legacy code that has no tests. Do NOT change behavior yet.

STEP 1 - Characterize. Read the target function and write a suite of characterization tests that assert its CURRENT behavior, including the ugly parts. Cover: happy path, each branch, boundary values, and any implicit rounding/truncation/timezone/null handling you can infer. Name each test after the behavior it pins, e.g. test_rounds_half_up_on_tax. If a behavior looks like a bug, still pin it and mark the test with a comment `# SUSPECTED BUG - preserved intentionally`.

STEP 2 - Confidence pass. List every test and rate your confidence (high/med/low) that the assertion matches the real code. For any `low`, quote the exact lines you are unsure about and ask me before assuming.

STEP 3 - Refactor. Only after I confirm, propose changes as a series of small commits. After each commit, restate which characterization tests still pass. Any test that would change is a STOP: surface it, do not rewrite it.

Target:
{PASTE_FUNCTION}
3the STOP-on-changed-test rule is the whole game. without it the model treats a red test as a thing to edit, not a warning.linter_liam 2 months ago
add a comment

2 Answers

7

This is the right instinct but STEP 1 leaks: the model writes tests from what it thinks the code does, not what it does. I add a golden-master step in the middle. Have it generate 50 random-but-realistic inputs, ask you to run the real function on them, paste back the outputs, and only then let it write asserts against those recorded outputs. Now the tests are anchored to reality instead of the model's reading comprehension.

THE PROMPT
Between STEP 1 and 2, insert: 'Emit a table of 50 diverse inputs. I will run the real code and paste the outputs. Write asserts ONLY against outputs I provide, never against your prediction.'
5

One caveat from getting burned: the SUSPECTED BUG - preserved intentionally marker is gold for review, but tell it to also open a separate list of those markers at the end so you can triage them as real tickets later. Otherwise they get buried in the diff and you ship the bug forever because now it is 'tested'.

Your Answer