57

I have a research agent with search, fetch, and a calculator tool. It gets into loops: searches, gets a mediocre result, searches again with a nearly identical query, and burns tokens/latency without new information. Sometimes it calls the calculator on numbers it already has.

I've been trying to encode a "tool discipline" section in the system prompt: state a hypothesis before each call, don't repeat a call whose result you already have, and prefer acting on existing evidence over gathering more. It's better but not airtight.

Is prompt-level discipline enough, or do people enforce a call-dedup/budget in the harness? Looking for the pattern that actually converges.

THE PROMPT
You are a research agent with tools: search(query), fetch(url), calc(expr). Tools cost time and money; use them like a careful investigator, not a search engine on autopilot.

Discipline rules:
1. Before any tool call, state: HYPOTHESIS (what you expect to learn) and WHY_NOT_ALREADY_KNOWN (why existing context can't answer this). If you can't fill both, don't call the tool.
2. Never issue a search whose query is >80% similar to one you already ran. If a result was weak, change strategy (different terms, fetch a specific source), don't rephrase.
3. Maintain a short KNOWN list of facts you've gathered. Consult it before every call. Never re-derive or re-fetch a known fact.
4. You have a budget of 6 tool calls for this task. Announce remaining budget before each call. When budget hits 0, answer with what you have and flag uncertainty.
5. Prefer answering with current evidence over one more call. Ask: "would this call change my answer?" If not, skip it.

When done, output the answer plus a CITATIONS list mapping each claim to the tool result that supports it.
1Similarity threshold on queries is clever. We use trigram similarity in the wrapper and block anything over 0.8 with a nudge to change strategy.async_annie 1 month ago
add a comment

2 Answers

22

Prompt discipline reduces it; a harness-level dedup kills it. We hash normalized tool args and if the same (tool, args) hash repeats, we don't re-run it, we return the cached result plus a system note: "you already called this; here is the result, do not call it again." That note in the observation stream is what actually breaks the loop, because the model sees the consequence of repeating itself.

THE PROMPT
Harness injects on duplicate call: `[SYSTEM] Duplicate tool call detected. Cached result below. Calling this again is not allowed; act on it or change approach.`
12

The hard call budget is doing a lot of work in your prompt, but announce it as a decreasing number in the observation, not just once. Models are much better at "you have 2 calls left" as fresh context each turn than at counting their own history. We also log the HYPOTHESIS lines and grade them offline; agents that skip the hypothesis are the ones that loop.

Your Answer