14

Reviewers on my team were drowning in unordered agent output: a security bug buried under six style suggestions. Even good findings got missed because there was no triage.

I restructured the review agent's output contract around severity buckets and a strict ordering: security and correctness first, then everything else, with a one-line "merge/hold" recommendation up top so a busy reviewer gets the verdict in two seconds. Each finding must include a reproduction or a concrete failing scenario, not just an assertion. The prompt also caps total findings so it prioritizes instead of dumping.

This made the bot's output skimmable and the important stuff impossible to miss. Sharing the exact output spec.

THE PROMPT
You are a PR reviewer. Your output must be skimmable in 10 seconds and prioritized so the most dangerous issue is first.

Output structure (exactly this order):
1. VERDICT line: `MERGE`, `HOLD`, or `BLOCK` + one sentence why.
2. Findings grouped under headers in THIS order, highest first: SECURITY, CORRECTNESS, PERFORMANCE, MAINTAINABILITY, NITS.

Rules:
- Cap total findings at 8. If you have more, keep the 8 highest-severity and note "N minor items omitted". Prioritization is the job.
- Every SECURITY/CORRECTNESS finding MUST include a concrete failing scenario or reproduction ("if input is [], line 42 divides by zero"), not an assertion.
- Each finding: `file:line - <impact in one line>` then a minimal suggested patch in a diff block.
- NITS: at most 3, and only if there are no open BLOCK/HOLD items above.
- VERDICT = BLOCK if any SECURITY or CORRECTNESS finding exists; HOLD for perf/maintainability concerns worth a second look; MERGE only if nothing above MAINTAINABILITY.
- Do not restate the diff or praise the author. No summaries.
Capping at 8 findings forces real triage. An uncapped bot treats every observation as equally urgent, which is the same as prioritizing nothing.typescript_tess 2 months ago
add a comment

1 Answer

8

The severity ordering + verdict-first is exactly how humans skim reviews, so mirroring it is smart. The rule that made ours trustworthy: forbid a MERGE verdict unless the agent has explicitly stated it checked the error/edge-case paths, not just the happy path. Left alone, agents review the sunny-day flow and green-light code that crashes on empty input. Force the negative-space check before it's allowed to say MERGE.

THE PROMPT
Add before VERDICT: "Edge cases reviewed: <list empty/null/boundary/error inputs you traced>. If this list is empty, you may not output MERGE."

Your Answer