The National Institutes of Standards and Technology (NIST) — which recently posted a solicitation containing opportunities for computing researchers — is now out with a call for submissions to its 21st annual Text REtrieval Conference (TREC), “the premier experimental effort in the field to encourage research in information retrieval and related applications” by providing a large test collection, uniform scoring procedures, and a forum for organizations interested in comparing their results. TREC has the following goals:
- to encourage research in information retrieval based on large test collections;
- to increase communication among industry, academia, and government by creating an open forum for the exchange of research ideas;
- to speed the transfer of technology from research labs into commercial products by demonstrating substantial improvements in retrieval methodologies on real-world problems; and
- to increase the availability of appropriate evaluation techniques for use by industry and academia, including development of new evaluation techniques more applicable to current systems.
According to a NIST press release:
Finding valuable information rapidly is much more than a game for people with a high-tech phone. Text retrieval is a field of research that can save lives by helping medical researchers locate key patient information or aid lawyers seeking important data in large digital data collections — both modern-day examples of needles in haystacks…
TREC is a rather unusual conference in that it starts months ahead of the actual meeting (Nov. 6-9, 2012) with the distribution of test data sets and challenges that TREC participants will use to develop and test advanced text retrieval techniques…
Scientists from academia and public and private-sector organizations … focus on improving information retrieval in specific areas. The groups develop algorithms to find information from [the] large, challenging datasets … provided by NIST. They work throughout the year and come to NIST’s headquarters in Gaithersburg, MD, to discuss their findings at the November meeting.
TREC is overseen by a program committee consisting of representatives from government, industry, and academia. For each TREC, NIST provides a test set of documents and questions. Participants run their own retrieval systems on the data, and return to NIST a list of the retrieved top-ranked documents. NIST pools the individual results, judges the retrieved documents for correctness, and evaluates the results…
A recent economic impact study prepared for NIST found that the NIST-led TREC project has significantly improved the ability to retrieve digital data. The report notes that TREC-related improvements are responsible for about one-third of the web-search advances between 1999 and 2009 and that the improvements may have saved up to 3 billion hours of web-search time.
This year’s TREC features 8 tracks:
Contextual Suggestions Track — investigates search techniques for complex information needs that are highly dependent on context and user interests.
Crowdsourcing Track — investigates emerging crowd-based methods for search evaluation and/or developing hybrid automation+crowd search systems.
Knowledge Base Acceleration Track — looks to develop techniques to dramatically improve the efficiency of (human) knowledge base curators by having the system suggest modifications/extensions to the KB based on its monitoring of the data streams.
Legal Track — to develop search technology that meets the needs of lawyers to engage in effective discovery in digital document collections.
Medical Records Track — to foster research on providing content-based access to the free-text fields of electronic medical records.
Microblog Track — examines search tasks and evaluation methodologies for information seeking behaviors in microblogging environments.
Session Track — to provide the necessary resources in the form of test collections to simulate user interaction and help evaluate the utility of an IR system over a sequence of queries and user interactions, rather than for a single “one-shot” query.
Web Track — explores Web-specific retrieval tasks, including diversity and efficiency tasks, over collections of up to one billion Web pages.
And here’s a timeline for those interested in participating this year:
As soon as possible — submit your application to participate in TREC 2012:
Submitting an application will add you to the active participants’ mailing list. On Feb 23, NIST will announce a new password for the “active participants” portion of the TREC web site.
Beginning March 1
Document disks used in some existing TREC collections distributed to participants who have returned the required forms. Please note that no disks will be shipped before March 1.
July–August:
Results submission deadline for most tracks. Specific deadlines for each track will be included in the track guidelines, which will be finalized in the spring.
September 30 (estimated):
Relevance judgments and individual evaluation scores due back to participants.
November 6-9:
TREC 2012 conference at NIST in Gaithersburg, MD.
To learn more, and to apply to participate, check out the TREC 2012 website.
(Contributed by Erwin Gianchandani, CCC Director)
