TREC 2017 Core Track

Coordinators

Mailing list

INTRODUCTION

The primary goals of the proposed core track are three-fold: (a) to bring together the community in a common track that could lead to a diverse set of participating runs, (b) to build one or more new test collections using more recently created documents, and (c) to establish a (new) test collection construction methodology that avoids the pitfalls of depth-k pooling.

As a side goal the track intends to: 1. study the shortcomings of test collections constructed in the past 2. experiment with new ideas for constructing test collections 3. expand existing test collections in a number of dimensions: * new participant tasks (ad-hoc/interactive) * new relevance judgments (binary/multilevel) * new pooling methods * new assessment resources (NIST / Crowd+NIST) * new retrieval systems contributing documents (Manual/Neural/Strong baselines)

TRACK TASK

The participants task will be ad-hoc search. Automatic and manual runs are encouraged. The organizers of the track will provide participants with a task (title/description/narrative of TREC topics) and allow participating sites to run the experiment as they wish as long as they contribute a ranked list of documents as an output.

DATA AND RESOURCES

Collection: NYTimes corpus (https://catalog.ldc.upenn.edu/ldc2008t19) – Cost: $300

Queries: A subset of the TREC Robust queries from the Robust track.

Relevance Assessments: 1. NIST assessors (~50 queries) 2. Crowdsourcing (~200 queries)

ASSESSMENT AND EVALUATION

Evaluation: Participating runs will be evaluated in terms of unique relevant documents they contribute to the pool. Contributing irrelevant documents will be penalized. A suitable evaluation measure will be determined.

TIMETABLE

TBD