The purpose of the workshop was to provide a forum for energy disaggregation enthusiasts to learn about recent developments in the field, as well as network and discuss projects for future collaboration. The workshop was attended by academics, employees of energy disaggregation companies, multinational utility companies and a few hobbyists.

Prof Mario Bergés, Assistant Professor at Carnegie Mellon University gave the keynote speech which focussed on the relevance of NILM within the emerging domain of the Internet of Things (IoT). Mario covered recent trends in energy disaggregation, as well as his projection of the field’s relevance into the future. His talk also proposed four ‘million dollar’ ideas which he believes will have significant impact on the domain of energy disaggregation. You can watch Mario’s full talk on YouTube.

Workshop attendees also enjoyed talks from both academic and industry aspects, with speakers including Mingjun Zhong from the University of Edinburgh and Stephen Makonin from Simon Fraser University representing academic findings, and focussed on models for energy disaggregation, socioeconomic concerns and accuracy evaluation. From an industry perspective the workshop welcomed Bruno Charbonnier from EDF R&D, and Hjalmar Nilsonne from Watty who cemented the importance and benefits of disaggregating electricity and announced the release of a new dataset.

Delegates were invited to bring a poster to present at a dedicated ‘lightning talk’ session, giving each presenter a chance to disseminate and discuss a NILM related topic of their choice for 1 minute. During the lunch and break sessions, posters were displayed on the walls, giving the presenter an opportunity to engage in one-to-one discussions with other attendees. The posters are available via a dropbox folder, while the lightning talk session is also available on YouTube.
An MSc group from Imperial College London presented a tool for evaluating NILM algorithms without requiring the NILM algorithm’s code to be released. There was a real buzz of excitement around such an initiative, and a number of improvements were suggested around the need for a real-world private data set.

As the NILM Workshop came to a close, an agenda was decided for topics to discuss the following morning at an informal user group designed to encourage collaboration and potential projects among attendees, which included funding applications and data sharing. The afternoon session explored NILMTK; an open source toolkit for non-intrusive load monitoring and included an overview of the toolkit as well as discussions on how to encourage contributions from the community. The need for a collaborative knowledge base, where items such as public data sets can be described in an easily comparable and searchable format was also discussed, with the result being a web based wiki which will be available on the nilm.eu website soon.

The most obvious learning from the workshop was the increasing momentum in this domain. The 2014 European Workshop was attended by around 20 people while this year saw nearly 70 attendees from around the globe. In addition, the diversity of the problems being studied by each of the attendees was also clear from the poster session, as each start up or academic project has a subtle but significantly different take on the problem of energy disaggregation. Lastly, the problem of evaluation accuracy cropped up regularly throughout the workshop. The need for standard data sets, metrics and methodologies is now more important than ever.

The workshop was streamed live on YouTube, and videos of all talks can be seen via our YouTube playlist, while each presenter’s slides can be downloaded from our dropbox folder.

The two day workshop finished with a discussion of plans for the 2016 European workshop. While the location of the workshop is yet to be decided, it was clear that there was sufficient demand for a future meeting.

http://spectrum.ieee.org/tech-talk/consumer-electronics/gadgets/crowdsourcing-smartphone-gps-could-improve-earthquake-warnings

Simpson, Edwin, Venanzi, Matteo, Reece, Steven, Kohli, Pushmeet, Guiver, John, Roberts, Stephen and Jennings, Nicholas R. (2015) Language Understanding in the Wild: Combining Crowdsourcing and Machine Learning. In, 24th International World Wide Web Conference (WWW 2015)

This problem involves classifying the sentiment of a large corpus, i.e., hundreds of thousands, of tweets using only a small set of crowdsourced sentiment labels provided by human annotators. In particular, this problem is relevant to various text mining tasks such as weather sentiment classification from twitter [1] and disaster response applications, e.g., the Ushahidi-Haiti project where 40,000 emergency reports were received in the first week from victims of the 2010 Haiti earthquake [2].

To build such a system that classifies tweets based on crowdsourced judgments, we must deal with three key challenges. Firstly, each annotator may have different reliabilities of labelling tweets correctly depending on the content of the tweet. In fact, interpreting sentiment or relevance of a piece of text is highly subjective and, along with variations in annotators’ skill levels, it can result in disagreement amongst the annotators. Secondly, typically there are so many tweets that a small number of dedicated expert labellers will be overwhelmed as was the case during the Haiti earthquake. As a result, the human labels may or may not cover the whole set of tweets, so we may have tweets with only one label or multiple, perhaps conflicting, labels, or none. Thirdly, each distinct term of the dictionary has different probabilities to appear in tweets of different sentiment classes. For example, the terms “Good” and “Nice” are more likely to be used for tweets with a positive sentiment. Thus, we must be able to provide reliable classifications of each tweet by leveraging the language model inferred from the aggregated crowdsourced labels to classify the entire corpus (i.e., tweet set).

To solve this problem, BCCWords builds upon the core structure the Bayesian Classifier Combination model (BCC) to add a new feature relating to modelling human language in addition to aggregating crowdsourced labels. In detail, BCC can learn the reliability of each annotator through a confusion matrix expressing the labelling probabilities for each possible sentiment class. Here are examples of confusion matrices for two annotators rating tweets in five sentiment classes [neutral, positive, not related, unknown] from the CrowdFlower (CF) dataset described in the paper:

Thus, by combining the aggregation mechanism of BCC with language model, we are able to simultaneously inferring the confusion matrix of each worker, the true label of each tweet and the word probabilities of each sentiment class.

We have applied BCCWords to the CF dataset containing up to five weather sentiment annotations for tens of thousands of tweets annotated by thousands of workers. The model correctly found the correlation between positive, negative, neutral words in the related sentiment class. We can see this in the word clouds below. These show the most probable words in each class with word size proportional to the estimated probability of the word conditioned on the true label:

*The black boxes hide some swear words that were inferred by BCCWords within the feature set for the tweets with negative sentiment.

We can also identify the most discriminative words in each class by applying a standard normalisation of the estimated word probabilities (details in the paper):

These word clouds clearly show that the words such are “beautiful” and “perfect” are more discriminative for positive tweets, while the words “stayinghometweet” and “dammit” are more likely to occur in negative tweets. We also found that some words like “complain”, “snowstorm” and “warm” do not necessarily imply a particular positive or negative sentiment as their interpretation is highly context dependent and therefore most of the annotators classified the relative tweets as “unknown”. You can find out more about the classification accuracy of BCCWords and its ability to exploit the language model to predict labels for the entire set of tweets in the paper.

More trials of this technology is in progress. We are currently working with RescueGlobal (A UK-NGO specialised in professional disaster response) and the ORCHID project to use the developed model to analyse live streams of emergency tweets received during recent environmental disasters in the Philippines.

[1] See http://www.crowdflower.com/blog/2013/12/crowdsourcing-at-scale-shared-task-challenge-winners

[2] See https://www.linkedin.com/pulse/how-social-media-can-inform-un-assessments-during-major-patrick-meier

At one point Garry mentioned humans and computers playing together, rather than in opposition. One human plus one computer interact to devise the next move, and on the other side of the chessboard sits another human/computer pair. This setup, which made some of us think of HACs, enhances rather then detracts from player creativity and has led to innovations in game strategy.

The second edition of the Gamification course taught by Prof. Kevin Werback (University of Pennsylvania) is live on Coursera. The course was very successful in his first edition (2012). It is currently half way through its programme but all the videos and assignments are available online and the lectures organise regular hangout meetings with the students. I find it quite engaging and fun to attend and presents a lot of interesting material.

-Matteo

      This message and any attachment are intended solely for the addressee  and may contain confidential information. If you have received this  message in error, please send it back to me, and immediately delete it.     Please do not use, copy or disclose the information contained in this  message or in any attachment.  Any views or opinions expressed by the  author of this email do not necessarily reflect the views of the  University of Nottingham.    This message has been checked for viruses but the contents of an  attachment may still contain software viruses which could damage your  computer system, you are advised to perform your own checks. Email  communications with the University of Nottingham may be monitored as  permitted by UK legislation.