TREC 2017 Core Track
- Evangelos Kanoulas (firstname.lastname@example.org), University of Amsterdam
- James Allan (email@example.com), University of Massachusetts
- Donna Harman (firstname.lastname@example.org), NIST
The primary goals of the proposed core track are three-fold: (a) to bring together the community in a common track that could lead to a diverse set of participating runs, (b) to build one or more new test collections using more recently created documents, and (c) to establish a (new) test collection construction methodology that avoids the pitfalls of depth-k pooling.
As a side goal the track intends to: 1. study the shortcomings of test collections constructed in the past 2. experiment with new ideas for constructing test collections 3. expand existing test collections in a number of dimensions: * new participant tasks (ad-hoc/interactive) * new relevance judgments (binary/multilevel) * new pooling methods * new assessment resources (NIST / Crowd+NIST) * new retrieval systems contributing documents (Manual/Neural/Strong baselines)
The participants task will be ad-hoc search. Automatic and manual runs are encouraged. The organizers of the track will provide participants with a task (title/description/narrative of TREC topics) and allow participating sites to run the experiment as they wish as long as they contribute a ranked list of documents as an output.
DATA AND RESOURCES
Collection: NYTimes corpus (https://catalog.ldc.upenn.edu/ldc2008t19) – Cost: $300
Queries: A subset of the TREC Robust queries from the Robust track.
Relevance Assessments: 1. NIST assessors (~50 queries) 2. Crowdsourcing (~200 queries)
ASSESSMENT AND EVALUATION
Evaluation: Participating runs will be evaluated in terms of unique relevant documents they contribute to the pool. Contributing irrelevant documents will be penalized. A suitable evaluation measure will be determined.