I spent some time last week working with the “Geeklist Corps of Developers” on their Typhoon Haiyan hackfest. I tried to bring simple machine learning to bear on categorising tweets into different classes, ranging from the general “Infrastructure” down to the more specific “Need Water”, “Food Offered”, and others. This (perhaps unsurprisingly) did not work very well.
Classifying short text and extracting its meaning is tough job, and even humans are not completely reliable. Reading through some of the MicroMapper’s (see Joel’s piece earlier today) tagged tweets which I was using to train the classifiers, it became clear that rushed, stressed and tired humans are not a ground truth. Even if my simple classifier was able to extract enough meaning from the text to work, it would only be as good as its training set.
The groups of volunteers who worked to make new web applications and classify the short text messages provide an incredibly useful but flawed service. Even with AIDR finding relevant tweets and motivated volunteers working tirelessly, the signal to noise ratio was still high – and unquantified. ORCHID led work on crowd-sourcing should be deployed to reduce stress of the volunteers and increase the quality of their output, as well as make sure we collect the right data and give our results to the right people. 
Where ORCHID can really make a difference is in working across all three institutions to provide turnkey tools to enhance the crowd sourcing that is already going on- through the use of provenance, principled classifier combination and novel techniques to take the load off the human volunteers, along with other methods.