A better benchmark starts with better questions
A short note on what an evaluation measures—and what it can miss.
The question behind the score
A retrieval score depends on the questions used to measure it. If every question repeats the vocabulary of its source document, the test may reward matching familiar words more than finding useful information.
A practical starting point
Write down what someone needs to know before inspecting the system’s answer. Include straightforward questions, ambiguous questions, and questions the corpus cannot answer. Keep those categories visible when reading the results.
Make the experiment interpretable
Freeze a baseline and change one part of the system at a time. Keep a separate set of questions out of tuning. A single headline score is useful, but examples of failures often explain more about what to build next.
Where this connects to my work
Arbol makes source structure and traceable context part of retrieval. Nullius explores the evaluation side: how to inspect the instrument used to judge a system. The common question is whether the evidence supports the conclusion.