In-conversation: How do you feel about your sleep tracker?

A new research project is examining user perceptions of sleep tracking technology. We spoke to doctoral researcher Ruth Ravichandran about sleep tracking and her research that takes in the worlds of medicine, engineering and psychology.

You can become a research participant yourself, by clicking this link to complete a 10 minute research questionnaire. As a small bonus, upon completion of the survey you may opt in to a raffle for one of five 20$ Amazon gift cards.

If you’re concerned about your sleep health and you happen to be a technology fan too, it’s likely you may have used or been tempted to try one of the many sleep tracking apps or gadgets that have exploded onto the consumer digital health market in the past few years.

However, as much of a sleep geek you might be, in the world of personal sleep tracking technology, there’s a big white elephant in the room that no-one seems to be talking about.

That is, do sleep tracking devices actually provide any real, (rather than perceived) and lasting benefits? And how accurate and/or reliable are the proprietary ‘sleep score’ algorithms each manufacturer uses?

The gold standard of sleep tracking, polysomnography  (PSG) requires trained staff, a mass of wires and expensive monitoring equipment, and dozens of electrodes hooked up to your body and brain.

In contrast, most sleep tracking apps use just one metric – your body movement – to measure the quality of your sleep. Whilst more consumer devices are starting to introduce other sensors to measure sleep variables such as heart rate, breathing and temperature, it’s clear that personal sleep tracking is light years away from getting anywhere near to the accuracy and reliability of PSG.

Someone who has in interest in both the consumer and medical aspects of sleep tracking is Ruth Ravichandran a 3rd year Ph.D. student in Electrical Engineering working at the UbiComp (Ubiquitous Computing) Lab at the University of Washington, Seattle.

Ruth got in touch recently to raise awareness of her research project which looks at how users interact with sleep tracking technology and how they perceive the data their devices reveals.

Part of the research program is a survey that’s open to anyone who has previously used a sleep tracking app, wearable or sleep monitor. So if you’re interested in contributing to the scientific advancement of our understanding of the intersection of sleep and technology, you can be a research participant by following the link below:

And as a small bonus, upon completion of the survey you may opt in to a raffle for one of five 20$ Amazon gift cards.

We think it’s a great initiative, so we asked Ruth in more depth about her work and the background and motivation for the research:

JM (Jeff Mann): Hi Ruth, thanks for taking the time to speak to us. What’s your scientific background and what kind of work goes on at the UbiComp Lab?

RR (Ruth Ravichandran): I’m a electrical engineer and a third year PhD student at the University of Washington, Seattle. UbiComp stands for Ubiquitous Computing and we are a group of computer science and electrical engineering majors who work on designing novel sensing applications using sensors that are all around us.

A lot of our works revolves around health sensing devices like detecting infant jaundice, anemia or asthma using sensors built into our smart phones. My work is focussed on continuous tracking of vital signals like breathing rate and heart rate using Radio Frequency signals.

JM: How did you first get interested in sleep and technology?

RR: My first research project was building a system to detect people’s breathing rate using wireless signals in the Wi-Fi frequency range. While working on the system we found that while continuous breathing rate estimation could be indicative of a person’s stress levels, doctors were particularly interested in using this application to be able to detect sleep apnea episodes.

That’s how I got interested in sleep sensing technology. Sleep was a prefect application scenario for non-contact and unobtrusive vital signal monitoring since it has the potential of identifying sleep related breathing disorders which a lot of people are unaware of.

In order to validate the system against the medical gold standard, we’ve been conducting studies in a sleep lab and monitoring patients with sleep disorders. Now I’m well aware of sleep staging from a technologist point of view and also from the medical perspective.

JM:  Manufacturers often make vague or misleading claims about ‘sleep scores’. Do you think we need more scientific rigour/standardisation in how the features and benefits are marketed and described?

RR: This is a really good question and this is what got me interested in conducting this research study. My husband started using the Microsoft band for activity tracking and tried out the sleep tracking functionality as a curiosity. But now it’s become a part of his daily routine and he’s begun paying a lot of attention to the sleep data from the device.

The sleep score is more often than not the first thing that he sees when he wakes up. And I’ve noticed that a lot of our friends are interested in discussing even comparing scores. Since sleep is an unconscious activity and we usually have little control over what happens when we sleep, these scores and data are very intriguing to users. As a scientist, I want to get a better idea of how users perceive the data they get from these devices, how they make sense of the data (which usually doesn’t come with any explanation) and how it affects them. That’s the purpose of this survey.

Now before I go ahead and talk about what these scores actually mean or whether or not they should be standardized, it is paramount that the audience reading this take the survey with an unbiased mind. Its an important part of our data collection process. I do not want any opinion (mine or experts’) to cloud their judgement.

In addition to survey responses, we have also been mining customer reviews for popular sleep tracking smart phone apps and wearable devices. We are also conducting interviews with sleep experts and sleep researchers and physicians. Once we have the data from a diverse audience, we will publish our findings in a research paper and talk about the implications of these scores and metrics. We will discuss design recommendations and explore ways of making sleep tracking more beneficial to people.

JM: Sleep staging is another grey area in the world of sleep tracking technology. For example, some gadgets claim to be able to distinguish REM sleep just by measuring your breathing and heart rate. What are your thoughts on these claims?

RR: Sleep staging is an integral part of sleep tracking. People not only want to see how long they slept but they want to know what’s the quality of their sleep. Did they have a restless night or not, how much time did they spend in REM or deep stages. I’ve seen that come up in reviews over and over again.

Again, I don’t want to say anything here that might get the readers biased before they take the survey. I don’t want it to appear as if being ambiguous or elusive. As a scientist my goal is to make sure that our work would some day create a well-informed society but in order to get there we first need to identify what our roadblocks are and in this case whether the data from these devices is mis-leading and feeding peoples’ misconceptions.

Now that I’ve emphasized my point, I’m well aware that there’s a disconnect between what people think these trackers are capable of sensing and what they actually sense. As people cycle through different stages of sleep, several physiological changes happen. While electrical signals from the brain and eye movement are the most telling and indicative physiological signals that help classify these stages, heart rate and breathing rate also go through characteristic changes corresponding to the different sleep stages.

So commercial trackers are not completely off when they say they can detect the different stages. But whether or not they are accurate or validated against a gold standard is a completely different question. Companies don’t publish results in a peer reviewed journal or they don’t in any way make their results public.

The same goes for their algorithms and sleep score calculations. It’s all proprietary information. Once we start talking about accuracy the immediate question that pops up is “Well how accurate does a sleep tracking device need to be?” I think a lot of sleep tracking devices can get away with inaccurate results because

a) sleep is an unconscious activity and users have no way of testing the validity of the data they’re getting. Unlike activity trackers where people can to some extent test if they did walk X amount of steps. I’ve read about sleep tracking enthusiasts trying on different brands at the same time in a lab sleep study and compare the results to medical ground truth (PSG). But again it’s one person doing a study for one night. And sleeping in a sleep lab is not very representative or normal sleep behaviors.

b) even if these devices consistently underestimate or overestimate these values, what, if any effect does it have on people?

JM: Tell us about your PhD research and how we can get involved.
RR: I’m passionate about the quantified self and health sensing devices. I firmly believe that technological advances facilitate people to become more aware of their bodies motivate them towards making behavioral changes to move towards healthier lifestyles.

As engineers and researchers we need to be wary of the information that is being presented to people. As these tracking devices are becoming prevalent, it’s time we take a step back and understand what sleep tracking means in the context of an everyday routine. One of the experts we interviewed said:

“The thing about sleep is there’s trait variability and the susceptibility to the effects of a poor night sleep. Some people can sleep deprive themselves or have poor quality sleep and not really feel many ill effects from it. Other people can have just minor decrements in those sleep factors can have a pretty big impact on sleep quality.”

I feel that commercial sleep tracking devices are ignoring this big fact and trying to normalize people’s sleep quality on a scale of 1-100. This survey is a big part of understanding what sleep quality means to each and every individual and whether or not their sleep tracking device is helping them understand and become more aware of their sleep.

So I urge the audience to give us their honest opinions on what they think about the data they get from sleep tracking devices and how it relates to their own perception of their sleep quality. Hopefully we will get a better understanding of the disconnect and open up discussions and inform technologists on what we need to focus on how we can make sleep tracking more beneficial to people.

Here’s the survey link again for anyone that wants to take part in Ruth’s research:

Leave a Comment