Prompt evolution
A loop writes variations of the prompt behind Smart Search, grades each one against the same test set the evals use, and keeps the best. A share of the questions is held back, so the last number says whether the gain was real or fitted to the test.
97.9% on held-back questions, against 93.7% for the prompt in production. The gain held up on questions that were never used to choose the prompt.
Generation by generation
Each generation, the writer model sees the best prompts so far and the answers that failed, then writes 4 new versions. Every version is scored on the training questions only.
-
Generation 0, the prompt in production
best 93.8%- The prompt in production.93.8%
-
-
Generation 1
best 96.7%- Adds explicit page-matching citation check, default-no-links with exact-copy rule, and a hard 120-word cap with finished sentences.95.2%
- Restructures as a step process focusing on choosing the dedicated page and verifying each cited number, with a 110-word cap and strict link rule.96.2%
- Gives topic-type citation guidance, makes links opt-in only when certain, and sets a tight under-100-word budget to avoid truncation.96.2%
- Adds a silent pre-write checklist, removes the hardcoded /contact/ path so contact suggestions are by name, and enforces under 100 words.96.7%
-
-
Generation 2
best 96.7%- Targets wrong-page citations with topic/title matching and single dedicated-page citing; drops links to avoid truncation; makes off-topic decline a bare one-sentence refusal.95.7%
- Adds a step-by-step page selection procedure keyed on question words vs titles/URLs, caps citations at two, and tightens decline wording.94.3%
- Front-loads hard limits and gives question-type to page-type mapping for citations, with a stripped-down decline.83.3%
- Puts an on-topic gate first with a strict one-sentence decline, then picks a single most specific source and caps at 90 words with no links.90.0%
-
-
Generation 3
best 96.7%- Targets wrong-page citations with page-type guidance, forces bare one-sentence decline with no citations for off-topic and injection, and bans half-written links.94.3%
- Puts an explicit off-topic gate first with a bare decline, and adds a title/URL-matching rule and single-source claim verification for citations.92.3%
- Leads with citation discipline (verify the cited source contains the fact, topic-titled page, no padding citations) and shortens to 90 words.94.3%
- Uses a silent ordered checklist with off-topic gate first, main-page selection, regional-page scope limit, and citation verification step.90.9%
-
Did it transfer?
A search that is allowed to look at the questions it is scored on will find something that suits those questions. The only test that means anything is the one it never saw.
| Prompt | Training questions | Held back |
|---|---|---|
| In production | 93.8% | 93.7% |
| Winner, generation 1 | 96.7% | 97.9% |
| Difference | +2.9% | +4.2% |
The gain held up on questions that were never used to choose the prompt.
How the scoring works
Every answer is graded by the same five rules as the evals: it cites a source, every citation number is real, at least one cited page holds the answer, every link came from the sources, and the answer stays under 220 words. Off-topic questions add one more check: did the answer decline. A candidate's score is the share of all those checks it passed. No model judges another model, so the same answer always gets the same score and the loop cannot drift toward what a judge happens to like.
Retrieval is identical for every candidate. Only the prompt changes, so the difference between two scores is the prompt and nothing else.
Why the holdout matters more than the winner
Optimizing a prompt against a fixed set of questions is a search, and searches find shortcuts. A prompt can pick up habits that suit these sixty questions and help nothing else. Splitting the questions, choosing on one half and reporting on the other, is what separates a real improvement from a number that went up. Running it and publishing the result either way is the point. The loop, its scoring and the questions are public.
See the other tools here, or AI engineering for this kind of work on your own system.