TREC (Text REtrieval Conference)

The Text REtrieval Conference (TREC) is an evaluation campaign for information retrieval run by the U.S. National Institute of Standards and Technology (NIST). According to NIST’s TREC overview, it began in 1992 as part of the TIPSTER Text program and is co-sponsored by NIST and the U.S. Department of Defense.

What it provides

NIST describes TREC’s purpose as supporting research in the information retrieval community by providing the infrastructure needed for large-scale evaluation of text retrieval methods. Its stated goals include encouraging research based on large test collections, improving communication among industry, academia, and government, speeding the transfer of technology into products, and developing evaluation techniques.

The mechanism is shared: NIST provides a test set of documents and questions, participants run their systems on it, and NIST pools the individual results, judges the retrieved documents for correctness, and evaluates the results. The reusable product is a test collection made of documents, questions, and relevance judgments.

Why pooling matters

Only the pooled documents are judged, so the labels cover what the participating systems retrieved. A document outside the pool is unjudged, and the evaluation protocol has to decide how to treat it. A system that retrieves many unjudged documents can be disadvantaged by a protocol that counts unjudged documents as not relevant. This is one concrete case of the general problem described in Relevance in Recommender Systems: a metric can only reward what the labels say.

Limits of a fixed test collection

A test collection fixes the documents, the questions, and the judgments, so a score says how a system performed on that collection. It does not measure how satisfied users are with a live product, and a system tuned to one collection can do worse on another.

Relevance to ranking and recommendation

Many ranking metrics used to evaluate recommenders, including DCG, NDCG, and MAP, come from this information retrieval tradition of shared tasks and relevance judgments. Recommender data usually replaces assessor judgments with behavior such as clicks, ratings, or held-out interactions, which brings the selection problems discussed in that entry.

NIST also reports that retrieval system effectiveness approximately doubled in the first six years of TREC and that most of today’s commercial search engines include technology first developed in TREC. These are NIST’s statements about its own program.

References:

• NIST, Text REtrieval Conference (TREC) overview