Mental Overload Detection with Respiration Variability Spectrogram(aqibsaeed.github.io)
aqibsaeed.github.io
Mental Overload Detection with Respiration Variability Spectrogram
http://aqibsaeed.github.io/2018-01-19-mental-overload-detection-with-respiration-variability-spectrogram/
17 comments
TTS, STT?
Text to speech, speech to text, is how I understand the two acronyms put together.
Study: DeepBreath: Deep Learning of Breathing Patterns for Automatic Stress Recognition using Low-Cost Thermal Imaging in Unconstrained Settings
Citation: Youngjun Cho, Nadia Bianchi-Berthouze, Simon J. Julier. arXiv:1708.06026 [cs.HC]
Link: https://arxiv.org/abs/1708.06026
Abstract: We propose DeepBreath, a deep learning model which automatically recognises people's psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person's breathing patterns as temperature changes around his/her nostril. The paper's technical contribution is threefold. First of all, instead of creating hand-crafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community.
Citation: Youngjun Cho, Nadia Bianchi-Berthouze, Simon J. Julier. arXiv:1708.06026 [cs.HC]
Link: https://arxiv.org/abs/1708.06026
Abstract: We propose DeepBreath, a deep learning model which automatically recognises people's psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person's breathing patterns as temperature changes around his/her nostril. The paper's technical contribution is threefold. First of all, instead of creating hand-crafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community.
Huh? Isn't "hypertension" high blood pressure? I get that "extreme tension", as in "extreme stress", also makes sense. But is this recognized usage?
From the linked study, those spectrograms seem to represent a frequency of nostril breathing versus... mouth breathing?
The automatic analysis of the source data’s thermal video capture seems to look for hot nostril flares, from breathing through the nose only, as a reliable measure of relaxed respiratory rate? The respiratory rate is either assessed as a measurable number, ranked as a percentile, high or low, or indeterminate/chaotic which is then presumed as correlated with elevated mental activity?
Note that this spectrogram is not chemical analysis of respiratory exhaust from exhaled gases. It’s not mass spectrometry, searching for CO2 and water vapor content as an indicator of metabolic respiration, which is what the title almost sounds like it could be about, without careful reading.
The study then goes on to state that other sensors (air flow meters, pressure sensitive chest straps) could be used to extract respiratory rate over time, and construct the same data interchangably. I’m not so sure that’s true, but clearly the goal here is passive reads, probably for retrospective lie detector types of analysis. Seems like a presuptive effort at precrime.
The automatic analysis of the source data’s thermal video capture seems to look for hot nostril flares, from breathing through the nose only, as a reliable measure of relaxed respiratory rate? The respiratory rate is either assessed as a measurable number, ranked as a percentile, high or low, or indeterminate/chaotic which is then presumed as correlated with elevated mental activity?
Note that this spectrogram is not chemical analysis of respiratory exhaust from exhaled gases. It’s not mass spectrometry, searching for CO2 and water vapor content as an indicator of metabolic respiration, which is what the title almost sounds like it could be about, without careful reading.
The study then goes on to state that other sensors (air flow meters, pressure sensitive chest straps) could be used to extract respiratory rate over time, and construct the same data interchangably. I’m not so sure that’s true, but clearly the goal here is passive reads, probably for retrospective lie detector types of analysis. Seems like a presuptive effort at precrime.
It would be interesting to see if this could have a positive application for pilots and situations of loss of situational awareness or pilot load factor.
Heart rate variability (https://en.wikipedia.org/wiki/Heart_rate_variability) may also be an interesting measure here.
When I did biofeedback therapy a couple of years ago, heart rate variability was the main proxy they used for relaxation. The goal for session was to obtain a steady heart rate.
Counterintuitively, it's actually the other way around -- at rest, more variability in your heart rate (over a roughly 20 second period) is correlated with lower levels of stress. Deep, slow, regular breathing tends to increase HRV, since heart rate is linked to respiration -- your heart beats a little faster when you inhale, a little slower when you exhale. It's desirable to have big, regular swings in heart rate, synchronized with your respiration.
Thankfully, biofeedback protocols will typically hide these details, and just calculate a single metric for you to attempt to optimize.
Thankfully, biofeedback protocols will typically hide these details, and just calculate a single metric for you to attempt to optimize.
No, I actually just remembered it wrong.
I remember what the chart looks like now, and yes the goal was in fact to have your heart rate vary with breath.
When I started the session, my heart rate would jump around almost arbitrarily. By the end, it would fall into a steady rhythm of rising and falling as I breathed.
I remember what the chart looks like now, and yes the goal was in fact to have your heart rate vary with breath.
When I started the session, my heart rate would jump around almost arbitrarily. By the end, it would fall into a steady rhythm of rising and falling as I breathed.
Why do you need a neural net for this? isn't it just a simple increase in variance?
I've been looking for an excuse to get a FitBit or equivalent (hint: I'm not particularly active other than a lot of walking). This would be awesome. I think I'm one of those "hides it well" people when it comes to stress. Would love something like this on my phone.
This reminds me of the professional gamers with collapsed lungs.
https://news.ycombinator.com/item?id=15425256
https://news.ycombinator.com/item?id=15425256
Cool project.
BTW, is this line by line `x = Something()(x)` style of writing code common in Tensorflow?
BTW, is this line by line `x = Something()(x)` style of writing code common in Tensorflow?
For Keras, yes. It's known as the "functional model".
Black Mirror episode idea: Luxury computer company starts including thermal front-facing cameras, as well as heartbeat monitors in laptop palm rest areas, in their hardware. Releases StressKit, an API which is promptly used by companies to optimize their experiences. At first, it's great: Slack silences notifications when someone's "in the zone," and Netflix has its greatest ever season of hyper-optimized horror movies.
But then our protagonist, a programmer at a large software company, is told to code a feature that subtly influences the behavior of users to drive them towards microtransactions in their moments of greatest stress. Protagonist finds this horrific and threatens to go public, at which point her colleagues, fearful for their own careers, start using the prototype system to manipulate her towards paranoia and madness. In true Black Mirror form, this escalates to neverending existential horror by the end.
*mic drop
But then our protagonist, a programmer at a large software company, is told to code a feature that subtly influences the behavior of users to drive them towards microtransactions in their moments of greatest stress. Protagonist finds this horrific and threatens to go public, at which point her colleagues, fearful for their own careers, start using the prototype system to manipulate her towards paranoia and madness. In true Black Mirror form, this escalates to neverending existential horror by the end.
*mic drop
The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over, but it can't. Not without your help. But you're not helping.