Our visualization system for emotion-oriented video summarization with three major views. The summary view consists of two parts: the emotion archives (a) display the emotion distribution of each person in the video, and the emotion flow (b) shows the overall and individual evolution. The black dashed rectangles and thumbnails of faces in the emotion flow are added manually for illustration. The character view (c) describes a selected person’s emotions with different factors, such as face size and occlusion. The video view (d) provides the original video and highlights the faces detected in each frame. Children’s eyes are covered with black rectangles for privacy reasons.


Analyzing students’ emotions from classroom videos can help both teachers and parents quickly know the engagement of students in class. The availability of high-definition cameras creates opportunities to record class scenes. However, watching videos is time-consuming, and it is challenging to gain a quick overview of the emotion distribution and find abnormal emotions. In this paper, we propose EmotionCues, a visual analytics system to easily analyze classroom videos from the perspective of emotion summary and detailed analysis, which integrates emotion recognition algorithms with visualizations. It consists of three coordinated views: a summary view depicting the overall emotions and their dynamic evolution, a character view presenting the detailed emotion status of an individual, and a video view enhancing the video analysis with further details. Considering the possible inaccuracy of emotion recognition, we also explore several factors affecting the emotion analysis, such as face size and occlusion. They provide hints for inferring the possible inaccuracy and the corresponding reasons. Two use cases and interviews with end users and domain experts are conducted to show that the proposed system could be useful and effective for analyzing emotions in the classroom videos.