Real-time Sociometrics is a system that can provide meaningful, real-time feedback to people on their social behaviour to facilitate communication. The key elements of this system are:
Tests indicate that the current algorithms can correctly understand social behaviour of the speaker in around 85% of cases. The feedback can be provided through various easy-to-use platforms like smartphone, VoIP platforms like Skype or smart glasses. Also, different social situations require different types of feedback. Verbal keywords detected from user speech, can play an important role in recognizing such a situation. These keywords will be integrated with existing non-verbal signals to better comprehend the social context and adapt the nature or frequency of feedback.
The technology comprises of:
1. Lapel microphones to record audio on separate channels
2. Microsoft Kinect to record video
3. PC or smartglass or smartphone to process the audio and/or video and deliver the feedback
Feedback can be provided through two technologies, VoIP or Android platform.
To integrate our social analysis with VoIP we choose the service Skype. We use appropriate settings on a Skype recorder to capture the audio-video data of each participant of the Skype call on two separate files. Subsequently, we separate the audio from the video data and fuse the audio data on a single 2-channel audio file at a sampling frequency of 8 kHz, each channel containing the audio data of only one person. The recordings are automatically stored in segments of 1 minute. We interface Matlab with the recorded audio through a file event handler which analyses the data as each file is saved. The recorded files are processed, including speech detection, feature extraction, and sociometric analysis. Feedback messages may be communicated to the users through the Skype API.
For Android implementation, we send messages to the user whenever the non-verbal cues and social indicators are in an abnormal range. Since Android devices often have small displays, only limited information can effectively be displayed. The analysis is conducted in Matlab environment; we use machine learning algorithms to infer various social indicators. The interface between Matlab and the Android device is achieved through the TCP server and client approach, where the TCP server operates on the same computer where the sociometric analysis is performed in Matlab, and listens to the incoming client connection, specifically, the Android device. Once the application is launched on the Android device, it connects to the TCP server and feedback from the sociometric system is sent to the Android device.
In this advanced digital era, valuable social and behavioural information is readily available via smartphones, apps and sociometric badges. This opens up a tremendous potential to develop tools to transform the way people communicate. Our proposed technology can be implemented on smartphones, PC/laptops, smartglasses, humanoid robots, VoIP platforms and is applicable in various industries. Hence, the market size can be approximated to be at least S$1 billion.
The system is a quick, non-invasive and effective way of receiving feedback on speaker's social behavior and speaking mannerisms. The solution is also language independent. Feedback is easily understandable as it is conveyed through graphical images.