Almost ten years ago, Keith van Rijsbergen delivered a Salton Award keynote with the intriguing title “Quantum Haystacks”. In this talk he put forward the idea that quantum theory could inspire a “design language for IR”. As a consequence, this design language relies on quantum logic and quantum probabilities, rather than the classical probabilities that underpin much of IR theory. The aim of this talk is to revisit quantum haystacks. It will put forward the idea that quantum theory can inspire a “design language for the IR user”. By drawing on developments in the emerging field of quantum cognition, theory and empirical evidence will be provided that quantum probabilities more naturally express human conceptual processing and decision making. As both of these pertain to the information seeking user, implications for the development of formal IR user models will be presented.
A major access divide has opened up between the academic and commercial Web search communities over the last 10-15 years due to the availability and importance of large scale search log user data in understanding certain types of search behavior. At the same time, the success and simplicity of Cranfield-inspired IR test collection experimentation methods has encouraged our research publication community to chase marginal gains of the latest effectiveness metric. However, experimentation based on test collections remains fundamentally essential in carrying out cost-effective development of search system algorithms. The real challenge and opportunity for all IR researchers, be they academic or industrial, is to improve the external validity of our experimental outcomes. The users of search systems, and their unfailing and surprising variability, have too often been overlooked when developing test collections. Based on a range of research that sits at the intersection of academic and industrial investigations, I hope to persuade you of the centrality of users in search evaluation, and the opportunities to integrate them more within our familiar experimental frameworks.