In the past few years, I’ve worked quite a bit on sleep stage detection, first with Oura, to help develop their latest sleep staging algorithm, and then with a group of experts, to devise guidelines for using wearables in the context of sleep tracking. These efforts are documented in the papers below:
2023. M. de Zambotti; C. Goldstein; J. Cook; L. Menghini; M. Altini; P. Cheng; R. Robillard. “Practical Guidelines for Using Wearable Technology in Sleep and Circadian Research: A Position Statement from the Sleep Research Society”. See here.
2021. M. Altini, H. Kinnunen, "The Promise of Sleep: A Multi-Sensor Approach for Accurate Sleep Stage Detection Using the Oura Ring". Sensors. Open access here.
While I do think we made great progress in using movement and autonomic activity (e.g. pulse rate and pulse rate variability, temperature) to estimate wake and sleep time as well as to estimate sleep stages at the population level, it is quite clear to me that the data is not particularly accurate (or useful) at the individual level. I would never personally rely on it to determine the effect of an intervention, for example.
In the context of sleep time, regardless of the technology used (Oura ring, Apple Watch, Coros or Garmin watches), the same problems arise: e.g. when reading my Kindle or even just watching a movie - in another room - I am often detected as sleeping, leading to inaccurate estimates of sleep time. I am not even going into stages here, as sleep time is something we can be aware of.
In the words of Brian X. Chen, who writes for the new york times (see here), “They struggle to tell the difference between when you are asleep and sitting very still … It’s surprising to me that this is the fourth Oura Ring and that these problems … haven’t already been solved.” - you can replace Oura Ring with any other wearable for that matter.
At this point, someone in the back will say that “the trend is what matters” - but where does the trend come from?
The trend is simply an average over a longer timeframe, and unfortunately, automatically collected data is not even good for long-term trends in my case, as the rather frequent misdetections, giving me 1-2 extra hours of sleep here and there, make the trend also wildly inaccurate.
That’s just my experience. Someone else in the back will call it anecdata.
What does the research say?
A recent meta-analysis (i.e. a research method that combines data from multiple studies on a topic to draw broader conclusions) looking at the ability of sleep trackers to actually detect sleep time reports that “wrist-worn sleep tracking devices, while popular, are not as reliable as polysomnography in measuring key sleep parameters like total sleep time, sleep efficiency, and sleep latency“.
While it is quite clear that sleep stages cannot be detected reliably with wearables, most people believe sleep time works well. I’ve argued the opposite a few times, as my experience is also in line with the paper.
This doesn’t mean we all have the same experience of course, as the accuracy is highly dependent on our behavior and I’m sure in some cases it can work well. However, bear in mind that as it often happens, there can be large errors and high individual variability in estimates (nothing is measured here, the wearable can only try to guess when you are actually sleeping, as it is unable to measure what it should be measuring for this application to work reliably: your brain waves), and therefore data should not be trusted blindly.
In our practical guidelines paper, we argue for looking at the actual physiology instead - which is measured - as opposed to relying on sleep-related estimates which might be inaccurate at the individual level. This is something I also discuss more in detail in the blog below, and is in my view, the most useful way to use a wearable:
A framework to make better use of Wearables data
When it comes to wearables, I often see either blindly embracing a device (i.e. fanboy kind of attitude, or sponsored athlete), or dismissing it entirely because of e.g. an inaccuracy in a metric provided (e.g. skeptical coach or scientist).
Personally, I track my sleep time in the only way I found reliable, i.e. using the questionnaire in HRV4Training.
I glance at the time before closing my eyes in the evening, and that’s more accurate than using any wearable technology, in many cases.
Go figure!
Marco holds a PhD cum laude in applied machine learning, a M.Sc. cum laude in computer science engineering, and a M.Sc. cum laude in human movement sciences and high-performance coaching.
He has published more than 50 papers and patents at the intersection between physiology, health, technology, and human performance.
He is co-founder of HRV4Training, advisor at Oura, guest lecturer at VU Amsterdam, and editor for IEEE Pervasive Computing Magazine. He loves running.
Social:
Hey Marco, I found your 2022 deep dive into wearable data comparisons and have been working through your subsequent work. I love it! Where are you at right now in terms of using a wearable for sleep tracking only? Ex. wear an Oura only at night. My main goal is to improve my sleep quality.
Hi! This is a really interesting article, and I’d like to share a few thoughts. If sleep detection algorithms are biased in a consistent way, why would they be such a bad option for tracking trends? For example, if I want to see if I’m sleeping better or worse than last week, it doesn’t matter too much if the algorithm mistakenly detects me as sleeping while I’m reading before bed, as long as it happens consistently, right? (Of course, this assumes my habits are mostly the same over time.)
In my case, I’ve found that self-questionnaires can be problematic (though I understand they may not be for everyone). I struggle with sleep issues and try not to look at the time when I have insomnia. Over the last year with my Coros watch, I’ve found that the algorithm is quite accurate. When I have a bad night, the metrics reflect that, and when I sleep well, they show that too.