Comparison of the performance of different information retrieval (IR) systems is typically solved by calculating evaluation measures for the systems using a standard test IR collection. To reach a desired level of reliability, the systems need to be usually compared using the same collection. However, this assumption requires that all the systems are run on this test collection, which is not always feasible. Comparison of the systems over different collections then requires some system normalization [1].
The performance of the IR system on the test collection depends on the characteristics of this collection, namely the queries, documents and relevance judgements. The dependence of the performance on the queries is studied by the query performance prediction [2]. The goal of this work is to explore the dependence of the performance on the features of the documents. The features might be mined using different neural and non-neural NLP approaches. An example might be a calculation of the similarity between two collections using lexical measures or embeddings or detecting the domain of the documents.
The goals of this work is thus following:
Formulate a list of the features which might affect the performance of the IR system applied to them.
Create collections which would be suitable for studying these effects.
Export the features on the created collections
Explore whether and how does the performance of the IR system depend on the features,
Finally, the features might be used to determine the performance of the IR system on the new unknown collection. The features might thus serve as the pivots to compare the IR systems across different collections.
Supervisors : Gabriela Gonzalez-Saez (PhD), Petra Galuscakova (Post-Doc), Lorraine Goeuriot, Philippe Mulhem.
[1] William Webber, Alistair Moffat and Justin Zobel: Score Standardization for Inter-Collection Comparison of Retrieval Systems, SIGIR 2008.
[2] David Carmel and Elad Yom-Tov: Estimating the Query Difficulty for Information Retrieval, Synthesis Lectures on Information Concepts, Retrieval, and Services, 2010.