Sentiment Analysis Symposium: Adventures in Emotion Recognition

The following post explores just one of the many talks at the Sentiment Analysis Symposium. To find out more about the Sentiment Analysis Symposium, check out our recap.

In a talk that was a clear crowd favorite of the day, MIT Media Lab’s Director of Affective Computer Research Rosalind Picard demonstrated “Adventures in Emotion Recognition.”

Her team aims to help people on the autism spectrum learn about emotional expressions. She’s received interesting requests for this technology, with adult participants who have Asperger’s disorder asking if she can give them something “that tells us when to back off.”

To do so, however, Picard’s team needs access to millions of videos tracking standard emotions, which would have cost big money to collect. The sheer volume of videos required would have made this experiment hard to scale.

To get around this, Picard and her colleague Rana el Kaliouby founded a company that created Affdex, an innovative external site that shows ads to volunteers and measures their smiles and other emotional reactions as they watch the videos. The site has a commercial application too, in that it measures the emotional connection people have with advertising, brands, and media. It turns out that people who have seen these ads before are smiling before the punch line—a good sign for ad creators!

In advertising, smiles denote happiness and bemusement. In reality, however, smiles could mean so much more. Picard played recordings of users struggling with broken technology and pointed out that some of the users were smiling even through their frustration. It turns out that there’s a difference between “delight” and “frustration” smiles, but the difference is very hard to pinpoint if you’re only viewing a static image. Picard emphasized that it’s important to look at the long-term dynamic (a recorded live session) rather than just a one-second reaction. As in many areas, context is key.

Facial recognition is all well and good, but there is more to emotion than just expressions. One innovative way to measure body ‘sentiment’ is via Electodermal Activity (EDA) sensors. By measuring the autonomic nervous system, researchers can get a read on a user’s current emotional state and brain activity. Use of these sensors helps in Picard’s expressed goal of helping those with autism. After all, Picard says, “We all live in our little bubbles. We all think that people think and feel like we do.”

Tracking emotions via expressions, body language, and skin sensors allows researchers to objectively compare participants’ reactions to those of others—or in the case of subjects with autism, tracking emotions means researchers can objectively compare normative emotional reactions and sentiment states to those of the people being studied. EDA sensors can measure a user’s engagement, non-engagement, frustration, or calm demeanor, which is good for autism and consumer studies alike.

In addition, turns out that seizures directly stimulate sensors and send a response to the skin. With this knowledge, Picard’s team can track epilepsy via EDA sensors. After all, she says, “emotion is the fourth vital sign.” The team has also noticed that EDA sensors measure an increase in brain activity during sleep. This, Picard said, is due to the memory consolidation and performance improvement that occurs while resting. It turns out that your brain is hard at work, even while you’re sleeping!

Picard’s work is not constrained to just video work with facial recognition or body sensors meant to track physiological emotional response. She is also currently working on identification and patterning of cyber bullying using text analysis. As it turns out, “adventures in emotion recognition” span all types of media, including face (analyzing smiles), physiology (emotional response judged by EDA), and now text analytics (for cyber bullying). The future of sentiment analysis is bright, and with Picard’s help, it will have applications in everything from medicine to commercial marketing.

Want more from the Sentiment Analysis Symposium? Check out our other posts: