Orchid Project

Human Computation Conference 2013

Overview

Report from the first AAAI – Human computation and crowdsourcing conference (HCOMP). This is one of the main conferences in the crowdsourcing area and this year received papers from diverse communities including machine learning, HCI, psychology, information retrieval, speech recognition, etc. Papers are mostly application-centric. Technical papers (e.g. machine learning) were accepted and presented though their reception was not generally great due to the diversity of the participant’s background.

Notable papers relevant to ORCHID WAs:

• Incentive engineering:

• Jon Kleinberg’s keynote (Cornel), Steering the Crowd: Rewards and Incentives for Collective Behavior:

• Incentives for Privacy Tradeoff in Community Sensing.

• Accountable infrastructures:

• Depth-Workload Tradeoffs for Workforce Organization:

1. Hannes Heikinheimo and Antti Ukkonen (Finland)The Crowd-Median Algorithm

• Crowdsourcing Multi-Label Classification for Taxonomy Creation (Best paper)
• Community Clustering: Leveraging an Academic Crowd to Form Coherent Conference Sessions

• Notable demos and posters (work in progress):
• HiveMind: Tuning Crowd Response with a Single Value
• Towards a Language for Non-Expert Specification of POMDPs for Crowdsourcing

All papers are available online.

Industrial presence: MSR, Google, Xerox, oDesk, Crowdflower, MicroWorks.

A common message from the industry was they are focusing now on hiring (and training) skilled workers and developing sustainable relationships with them rather than following the traditional model of one-off micro-tasks (like those on AMT).

Notable: Matteo’s algorithm and his joint Soton / MSR team wins the CrowdScale-Shared Task challenge. A more detailed post to follow.