Ask evals
Every time this site changes, a script asks the Ask feature 60 test questions, grades the results with fixed rules and publishes them here. Failures included.
Retrieval: does the right page get found?
Each question lists the pages that hold its answer. Keyword search (BM25) runs over 731 passages, and the test checks where the first of those pages lands.
| Check | Result | Rate |
|---|---|---|
| Right page ranked first | 51 of 60 | 85% |
| Right page in the top 3 | 56 of 60 | 93% |
| Right page reaches the model | 60 of 60 | 100% |
| Off-topic questions stopped before the model | 4 of 8 | 50% |
Keywords alone vs. keywords plus meaning
The live site blends the keyword ranking with search by meaning: each passage and each question is turned into an embedding, and the closest passages are merged in. Both versions ran on the same 60 questions against the deployed site on November 13, 2026. The table above tracks the keyword half on every change.
| Check | Keywords | Hybrid |
|---|---|---|
| Right page ranked first | 51 of 60 | 56 of 60 |
| Right page in the top 3 | 56 of 60 | 59 of 60 |
| Right page reaches the model | 60 of 60 | 60 of 60 |
| Mean reciprocal rank | 0.898 | 0.957 |
Hybrid moved 8 questions up and 1 down. The one that got worse: "Does Joe have ecommerce SEO experience?" went from #1 to #4.
Answers: does the model stay inside the sources?
Answer grading has not run yet. Retrieval results above are current. Answer results appear here after the next site change.
Every question
Rank is where the first page that holds the answer landed in the search results.
How this works
A GitHub Actions workflow runs on every push. It builds the search index from the site's content, runs the test set, writes the results to a JSON file and commits it. That commit rebuilds this page. The questions, the grading rules and the workflow are public.
The rules check what a script can verify: that an answer cites a source, that each citation points at a passage the model was given, that at least one cited page holds the answer, that every link came from the sources, and that the answer stays short. They do not judge writing quality. Read the case study for how the feature is built.