ReflectLive

Real-time Analysis and Feedback of Communication Behaviors in Video Meetings to Support Better Communication.

ReflectLive is a real-time, WebRTC-based communication system designed to help participants improve the way they present themselves and communicate to others. To accomplish this, the system analyzes a meeting’s audio and video streams to provide real-time feedback to participants about their own non-verbal communication behaviors.

Video teleconferencing allows people to connect and communicate across distance, but users are not always aware of their own non-verbal behaviors (such as how much they speak, interrupt others, look away from others, etc.). These types of behaviors can have a large influence on communication, and the limited view of speakers and their environment in video meetings makes these non-verbal behaviors particularly salient.

We designed and built ReflectLive, a system that analyzes the audio and video streams of a WebRTC-based video conference and provides real-time feedback (in the form dynamic data visualizations) about a speaker’s own speaking amount, eye gaze, interruptions, face position, and other non-verbal behaviors important for effective communication. ReflectLive runs in each user’s browser, maintaining privacy and reducing additional use of bandwidth. With ReflectLive, we investigate whether real-time feedback presented live to each user during the video meeting can help them adjust their communication behaviors to match their own expectations or goals, while at the same time minimizing distractions from the feedback itself.

Technical Contact

Related Publications

2019
Publication Details
  • IEEE 2nd International Conference on Multimedia Information Processing and Retrieval
  • Mar 14, 2019

Abstract

Close
We present an approach to detect speech impairments from video of people with aphasia, a neurological condition that affects the ability to comprehend and produce speech. To counter inherent privacy issues, we propose a cross-media approach using only visual facial features to detect speech properties without listening to the audio content of speech. Our method uses facial landmark detections to measure facial motion over time. We show how to detect speech and pause instances based on temporal mouth shape analysis and identify repeating mouth patterns using a dynamic warping mechanism. We relate our developed features for pause frequency, mouth pattern repetitions, and pattern variety to actual symptoms of people with aphasia in the AphasiaBank dataset. Our evaluation shows that our developed features are able to reliably differentiate dysfluent speech production of people with aphasia from those without aphasia with an accuracy of 0.86. A combination of these handcrafted features and further statistical measures on talking and repetition improves classification performance to an accuracy of 0.88.
2017
Publication Details
  • Computer-Supported Cooperative Work and Social Computing
  • Nov 1, 2017

Abstract

Close
Video telehealth is growing to allow more clinicians to see patients from afar. As a result, clinicians, typically trained for in-person visits, must learn to communicate both health information and non-verbal affective signals to patients through a digital medium. We introduce a system called ReflectLive that senses and provides real-time feedback about non-verbal communication behaviors to clinicians so they can improve their communication behaviors. A user evaluation with 10 clinicians showed that the real-time feedback helped clinicians maintain better eye contact with patients and was not overly distracting. Clinicians reported being more aware of their non-verbal communication behaviors and reacted positively to summaries of their conversational metrics, motivating them to want to improve. Using ReflectLive as a probe, we also discuss the benefits and concerns around automatically quantifying the “soft skills” and complexities of clinician-patient communication, the controllability of behaviors, and the design considerations for how to present real-time and summative feedback to clinicians.