Research
COMPUTATIONAL SLEEP SCIENCE
The importance of a good night’s sleep is paramount to quality of life.
Insufficient sleep can impede physical, emotional, and mental well-being, and lead to a variety of health problems such as insulin resistance, cardiovascular disease, mood disorders (e.g., depression or anxiety), and decreased cognitive function for memory and judgement. Moreover, the rapid pace of modern existence has resulted in an increasing prevalence of poor sleep quality and boosted interest in studying sleep behaviours and their contributing factors.
​
The following algorithms transform the traditional sleep science clinical process, into an automated, robust and pro-active data-driven approach.
Human activity recognition is the understanding of human behaviour from data captured by pervasive sensors, such as cameras or wearable devices. The analysis of complex health behaviours traditionally requires time-consuming manual interpretation by experts. While current state-of-the-art human activity recognition algorithms attempt to label the type of activity on simulated lab behaviour, we attempt to label the intensity on natural behaviours monitored by only a wearable device.
Deep learning models have achieved state-of-the-art results in a wide variety of tasks in computer vision, natural language processing and speech recognition. The fact that deep learning models automatically learn abstract feature representations from raw features, while also optimizing on the target prediction tasks, makes them an attractive solution for analyzing wearable device data.
DEEP LEARNING
Deep learning models have achieved state-of-the-art results in a wide variety of tasks in computer vision, natural language processing and speech recognition. The fact that deep learning models automatically learn abstract feature representations from raw features, while also optimizing on the target prediction tasks, makes them an attractive solution for analyzing wearable device data.
A data-driven activity recommendation system can take wearable device output, and provide personalized coaching to improve an individual's quality of life. A unique characteristic of activity, is its dynamic trajectory. An individual's behaviour is continuously changing. By evaluating an individuals daily activities using human activity recognition, we can identify behavioural recipes that lead to good or poor quality sleep. These recipes are "archetype behaviours for good sleep", which will guide the recommendations given to users throughout the day.
IEEE Computer Magazine released a special issue on quality of life technologies. Our work in the space of sleep science was a cover feature, and gave a high level overview of the deep learning models we created. In addition to the article, I was also interviewed by Dr. Katarzyna Wac, an associate professor at University of Geneva. You can find the interview below.
HUMAN ACTIVITY RECOGNITION
Human activity recognition is the understanding of human behaviour from data captured by pervasive sensors, such as cameras or wearable devices. It is a powerful tool. Wearable devices provide an unobtrusive platform for continuous monitoring, and due to their increasing market penetration, feel intrinsic to the user.
The analysis of complex health behaviours traditionally requires time-consuming manual interpretation by experts. While current state-of-the-art human activity recognition algorithms attempt to label the type of activity on simulated lab behaviour, we attempt to label the intensity on natural behaviours monitored by only a wearable device.
​
However, continuous sensing generates large amounts of big data that requires advanced computational analysis. Current algorithms either require a rich high-dimensional dataset, which is not available from most wearable devices alone, or are evaluated on supervised behavior, where the data is collected by asking a user to follow a set of commands observed within a clinic.
RAHAR is a robust automated human activity recognition algorithm developed to detect and label exertion levels from an activity signal. RAHAR works on natural, non-simulated, behaviors of wearable device users as they follow their daily routines. Moreover, it measures their exertion levels rather than activity type.
The RAHAR methodology has 4 key steps. Sleep period annotation and time series segmentation to identify consecutive periods of time when an individual is awake, change point detection to identify the statistically significant changes in behavior, and an activity classification which labels each behavior with an exertion label.
For more details, please refer to the following papers for more info:
​
Sathyanarayana, Aarti, Ferda Ofli, Luis Fernandes-Luque, Jaideep Srivastava, Teresa Arora, Shahrad Taheri. “Robust Automated Human Activity Recognition and its Application to Sleep Research”. IEEE International Conference of Data Mining DMHAA Workshop (2016) (link)
​
Sathyanarayana, Aarti, Luis Fernandez-Luque, Jaideep Srivastava. “The Science of Sweet Dreams: Wearable Devices and Sleep Medicine”. Published in IEEE Computer Magazine (March 2017) (link)
​​
Sathyanarayana, Aarti, Ferda Ofli, Luis Fernandes-Luque, Jaideep Srivastava, Teresa Arora, Shahrad Taheri. “Robust Automated Human Activity Recognition”. arXiv preprint (2016) (link)
DEEP LEARNING
Deep learning models have achieved state-of-the-art results in a wide variety of tasks in computer vision, natural language processing and speech recognition. The fact that deep learning models automatically learn abstract feature representations from raw features, while also optimizing on the target prediction tasks, makes them an attractive solution for analyzing wearable device data.
The importance of this approach is two-fold:
​
-
First, since the approach can be used in cases where sensory data during sleep is not available, the models can be used in the early detection of potential low sleep efficiency. This is a common problem with consumer-grade wearable devices, as users might not wear them during the night.
​
-
Second, this study was focused on advanced deep learning methods. Traditional prediction models applied to raw accelerometer data (e.g. logistic regression) suffer from at least 2 key limitations: (i) They are not robust enough to learn useful patterns from noisy raw accelerometer output. As a result, existing methods for classification and analysis of physical activity rely on extracting higher-level features that can be fed into prediction models. This process often requires domain expertise and can be time consuming. (ii) Traditional methods do not exploit task labels for feature construction, and thus can be limited in their ability to learn task-specific features. Deep learning has the advantage that it is robust to raw noisy data, and can learn, automatically, higher level abstract features by passing raw input signals through non-linear hidden layers while also optimizing on the target prediction tasks. This characteristic was leveraged by building models using a range of deep learning methods on raw accelerometer data. This reduced the need for data preprocessing and feature space construction and simplified the overall workflow for clinical practice and sleep researchers.
For more details, please refer to the following papers for more info:
​
Sathyanarayana, Aarti, Shafiq Joty, Luis Fernandez-Luque, Ferda Ofli, Jaideep Srivastava, Ahmed Elmagarmid, Teresa Arora, and Shahrad Taheri. “Sleep Quality Prediction From Wearable Data Using Deep Learning”. JMIR mHealth and uHealth 4, no. 4 (2016) (link)
Sathyanarayana, Aarti, Shafiq Joty, Luis Fernandez-Luque, Ferda Ofli, Jaideep Srivastava, Ahmed Elmagarmid, Teresa Arora, and Shahrad Taheri. “Impact of Physical Activity on Sleep:A Deep Learning Based Exploration”. arXiv preprint. (link)
​
Sathyanarayana, Aarti, Luis Fernandez-Luque, Jaideep Srivastava. “The Science of Sweet Dreams: Wearable Devices and Sleep Medicine”. Published in IEEE Computer Magazine (March 2017) (link)
RECOMMENDATION SYSTEMS
A data-driven activity recommendation system can take wearable device output, and provide personalized coaching to improve an individual's quality of life. A unique characteristic of activity, is its dynamic trajectory. An individual's behaviour is continuously changing. The only other dynamic recommendation systems that are deployed today, are traffic appli- cations such as Google Maps, Waze, Apple Maps etc. These applications frequently update the recommended driving route based on ongoing traffic, as well as the frequent change in a person’s location.
By evaluating an individuals daily activities using human activity recognition, we can identify behavioural recipes that lead to good or poor quality sleep. These recipes are "archetype behaviours for good sleep", which will guide the recommendations given to users throughout the day. To identify the behavioural recipes, we use time series clustering constructed from a human activity recognition alphabet (see RAHAR).
Many recommendation systems rely on evaluation via intervention and clinical trials. This is costly and time consuming. As an alternative we provide a framework for evaluating the validity of a recommendation retrospectively. This framework will be of particular use for population datasets such as the All of Us Research Project, where intervention is not possible.
The implications of integrating this system can lead to consumers developing a deeper understanding of their own personal sleep patterns, and lead to proactive involvement in their own sleep health. Moreover, the recommendations provide concrete actionable steps towards behaviour change. A successful recommendation engine for sleep quality has the potential to reduce fatigue in consumers, patients, and society as a whole.
For more details, please refer to the following patent for more info:
​​
Sathyanarayana, Aarti, Jaideep Srivastava. "Dynamic Activity Recommendation System". US Patent 20200075167A1 (link)