Last Friday I went to an interesting seminar by William Webber (blog), the basic premise of which was that IR researchers should consider constructing their own test collections, and outlined how to go about that. Here's the abstract.

I thought one slide pointing out how reusing test collections can lead to an unhealthily narrow focus was especially pithy, and I reproduce it here with William's permission.

Methodology section before TREC:

We identify as experimental variables: user characteristics;
problem statement; question statement; question characteristics;
search strategy; search characteristics …

Methodology section after TREC:

We take the TREC 8 AdHoc track collection. Our evaluation metrics
are P@10 and MAP.

Much of the discussion in 1966 has continued to revolve around methodological issues and has consisted largely of a repetitious dissection of a very limited amount of experimental activity with little theoretical basis. Refutations of rebuttals are often interesting but they typically generate little additional knowledge. Thus, the dialogue between those who accept the Cranfield methodology and those who, for a variety of reasons, are critical of it, has not been particularly productive from the point of view of advancing the state of the art. There is, and can be, no one way to test and evaluate retrieval systems, and it is absurd to imagine that any particular testing technique, or set of measures, will solve the problem of evaluation. Rather than engage only in carping criticism of the deficiencies of any one research project (thus giving rise to a new round of justifications of the procedures employed), it would be more desirable to devise and test alternative methodologies. Unfortunately, only scattered instances of this more positive approach can be found.

This is Alan Rees in 1967, as quoted by Cyril Cleverdon, one of the founders of modern information retrieval, in 1968. A common problem in many areas of research, especially in their youth. I love that Cleverdon doesn't directly go after his (or at least his methodology's) detractors, but quotes someone else doing it.