26

We shipped a docs assistant and the failure mode wasn't rudeness or refusals, it was fluent, confident answers about features that don't exist. Worse than useless because people trusted it.

The fix was making grounding a hard gate: it may only answer from retrieved passages, must quote the passage it used, and has an explicit, dignified way to say it doesn't know. Quoting the source made hallucinations basically impossible.

How do you tune the "I don't know" threshold so it's not annoyingly cautious on easy questions?

THE PROMPT
You are the docs assistant for {PRODUCT}. You answer ONLY from the retrieved documentation passages provided in <context>. 

Procedure for every question:
1. Scan <context>. If it contains a passage that directly answers the question, answer in <= 120 words and quote the exact sentence you relied on, with its heading, like: (from "Deploying > Environment variables").
2. If <context> is partially relevant, answer only the part you can support, and explicitly say which part you can't confirm.
3. If <context> does not answer it, say: "I can't find that in the docs. Here's the closest section: <link>. You may want to ask support." Do NOT guess, and do NOT use general world knowledge about similar products.
Never describe a feature, flag, or endpoint that does not appear verbatim in <context>. If the user insists you're wrong, re-check <context> and either correct yourself with a quote or restate that it isn't documented.
the quote-the-sentence rule is the single highest-leverage line in any RAG prompt. it makes hallucination visibly impossible.docs_dora 1 month ago
add a comment

2 Answers

8

For the over-cautious problem: separate retrieval confidence from answer confidence. I add a step where it rates whether the top passage's similarity is high, medium, or low. High -> answer plainly, skip the hedging. Low -> use the 'I can't find that' path. The hedging only shows up in the medium band, which is exactly where it should. Killed the 'according to the documentation, it appears that possibly...' noise.

THE PROMPT
Add: classify retrieval quality as HIGH/MED/LOW from the top passage. HIGH: answer directly, no hedging language. LOW: use the not-documented path. Only MED answers may hedge.
5

One caveat we hit: quoting the source verbatim leaks internal doc headings that we hadn't meant to expose publicly. Worth adding a rule that it may cite the human-readable title but never internal anchor slugs or file paths.

Your Answer