creators_name: Bennett, Paul N. creators_name: Maxwell Chickering, David creators_name: Mityagin, Anton type: conference_item datestamp: 2009-04-06 19:08:48 lastmod: 2009-04-07 14:02:12 metadata_visibility: show title: Learning Consensus Opinion: Mining Data from a Labeling Game ispublished: pub full_text_status: public pres_type: paper abstract: We consider the problem of identifying the consensus rank- ing for the results of a query, given preferences among those results from a set of individual users. Once consensus rank- ings are identified for a set of queries, these rankings can serve for both evaluation and training of retrieval and learn- ing systems. We present a novel approach to collecting the individual user preferences over image-search results: we use a collaborative game in which players are rewarded for agree- ing on which image result is best for a query. Our approach is distinct from other labeling games because we are able to elicit directly the preferences of interest with respect to image queries extracted from query logs. As a source of rel- evance judgments, this data provides a useful complement to click data. Furthermore, the data is free of positional biases and is collected by the game without the risk of frus- trating users with non-relevant results; this risk is prevalent in standard mechanisms for debiasing clicks. We describe data collected over 34 days from a deployed version of this game that amounts to about 18 million expressed prefer- ences between pairs. Finally, we present several approaches to modeling this data in order to extract the consensus rank- ings from the preferences and better sort the search results for targeted queries. date: 2009-04 pagerange: 121-121 event_title: 18th International World Wide Web Conference event_location: Madrid, Spain event_dates: April 20th-24th, 2009 event_type: conference refereed: TRUE citation: Bennett, Paul N. and Maxwell Chickering, David and Mityagin, Anton (2009) Learning Consensus Opinion: Mining Data from a Labeling Game. In: 18th International World Wide Web Conference, April 20th-24th, 2009, Madrid, Spain. document_url: http://www2009.eprints.org/13/1/p121.pdf