The study protocol has been described in detail in a previous publication [21]. We recruited 20 voluntary participants (mean age 24.5 years, SD 3.5, range 20-37 years; 10 [50%] women) by word of mouth in Helsinki, Finland. Participants were recruited from the research team’s circle of acquaintances: if the acquaintance showed initial interest in participating, they received a detailed description of the procedure via email. After reading the description, if the potential participant was still interested in taking part in the sleep study, they were screened for suitability. Their sleep was then measured for two consecutive nights using PSG, chest-worn Firstbeat Bodyguard 2, and a wrist-worn Geneactiv actigraph (Activinsights Ltd, Kimbolton, United Kingdom). The inclusion criteria were as follows: aged between 20 and 45 years, and a relatively stable sleep schedule (eg, no shift work or jet lag). Exclusion criteria were any diagnosed sleep disorder, the use of any medication that could affect sleep, acute sickness (eg, the flu), and gold allergy (as the electrodes used for the PSG recording were gold-plated). Each participant received a compensation of 100 euros (US $115) and structured feedback on their sleep stages. Written informed consent was obtained from all participants. The study was approved by the Ethical Committee of the Helsinki University Central Hospital. All procedures followed were in accordance with the Helsinki Declaration and its later amendments.

In our previous study, we investigated two different intervention groups within this setting, and demonstrated that these groups did not differ significantly from each other [21]. The Pittsburgh Sleep Quality Index (PSQI) scores of the participants, ranging from 2 to 12 points, indicate some variation in sleep quality. For the purpose of this study, all nights from all participants were pooled together for comparisons of PSG and Firstbeat sleep metrics.

A research assistant visited participants at their homes on two consecutive nights. Participants had been asked not to consume alcohol or caffeine after 4 PM on the measurement nights. The evening visit started between 6 and 10 PM, depending on the participant’s current sleeping schedule. The research assistant attached the measurement electrodes to the participant during the house call and began the recording. Before the research assistant left, participants were given instructions for the night; the participants were instructed to spend the evening as usual but to refrain from vigorous activities. They were also asked to keep their phones and any other electrical devices with transmitters at least 2 meters away from the bed so they would not interfere with the PSG recording. Participants were instructed to sleep normally, and the visit for the following morning was scheduled according to the participant’s expected wake-up time. The research assistant arrived the following morning approximately 0 to 30 minutes after the wake-up time.

We measured HRV with Firstbeat Bodyguard 2, including two chest electrodes and 3-axis acceleration data obtained from the wrist with a Geneactiv actigraph. The Firstbeat sleep analysis method evaluates the physiological state of the person as being awake or asleep based on HRV and acceleration data, and scores sleep as N1+N2 (light), N3, or REM. The method uses a neural network–based algorithm with HRV, HRV-derived respiration rate, movement, and time of day data for sleep and wake detection and for sleep classification. To align the measurement modes, we combined PSG-measured sleep stages N1 and N2 to correspond to “light” sleep of the Firstbeat method.

We used overnight PSG to measure sleep at home (SOMNOscreen plus, SOMNOmedics GmbH, Germany) with the following recorded parameters: EEG (left and right for F, C, O), left and right electrooculogram (EOG), left and right EMG, and ECG. The setup for the PSG and the removal in the morning were carried out by a trained research assistant. EEG measurements were recorded with gold cup electrodes at 6 EEG locations (F3, F4, C3, C4, O1, and O2) and 2 channels for the mastoids (A1, A2) according to the standardized 10/20 system. The ECG, EOG, and EMG were measured using disposable adhesive electrodes (Ambu Cardiology Blue Sensor M; Ambu Neuroline 715, Ambu A/S, Denmark) with two locations for ECG and EOG, and three locations for EMG. In addition, an online reference Cz and a ground electrode in the middle of the forehead were used. The sampling rate was 256 Hz. All signals were filtered with a pass band of 0.5-40 Hz (Hamming windowed sinc zero-phase FIR filter, cutoff [−6 dB] 0.25 Hz and 44.3 Hz, respectively) and rereferenced to the average signal of A1 and A2 electrodes. Sleep stages from PSG data were scored manually with the DOMINO program version 2.7 (SOMNOmedics GmbH, Germany) by two experienced researchers in 30-second epochs. The scoring was completed with both researchers visually inspecting the data together and agreeing over each epoch. The scoring was paused if any disagreement emerged and continued after agreement was found based on careful inspection of all channels, in accordance with the rules published by the American Academy of Sleep Medicine (AASM) [1].

Following standard sleep score practices in the AASM manual [1], we used 30-second epochs for sleep stage comparisons. The entire data were compared side by side after lights off; following AASM scoring rules, sleep onset was defined as the first epoch of any sleep stage as detected by the PSG measurement. We compared how Firstbeat was able to detect the actual sleep onset by calculating the difference between the two time points, which were statistically evaluated using a paired t test. All comparisons of sleep staging between PSG and Firstbeat were performed from the PSG-measured actual sleep onset onward.

First, we used paired-sample t tests to compare sleep metrics for evaluating differences between PSG and Firstbeat in sleep onset, and minutes spent in wake, light sleep, SWS, and REM sleep. Second, we conducted epoch-by-epoch comparisons between Firstbeat and PSG to calculate the sensitivity (ability of Firstbeat to detect true sleep), specificity (ability of Firstbeat to detect true wake), and accuracy (ability of Firstbeat to detect both sleep and wake). This comparison was performed across all sleep stages, as well as for overall sleep-wake comparisons between PSG and Firstbeat.

Third, we used a confusion matrix to compare epoch-by-epoch measures of true positives, true negatives, false positives, and false negatives between Firstbeat and PSG across all measured nights for sleep versus wake as well as for light sleep, SWS, and REM sleep stages. True positives arise when both the PSG and Firstbeat score the 30-second epoch as sleep. True negatives arise when both the PSG and Firstbeat score the epoch as awake. False positives arise when the PSG scores the epoch as sleep but Firstbeat scores it as wake. False negatives arise when the PSG scores the epoch as wake but Firstbeat scores it as sleep.

Fourth, we evaluated the differences between the amount of sleep scored as wake, light sleep, SWS, or REM sleep when comparing Firstbeat and PSG using minute-based Bland-Altman plots, and visually demonstrate how many observations remained within a 30-minute window of the PSG measure.

Finally, we used t tests to compare whether specificity, sensitivity, and accuracy differed based on sex, measurement night, or the intervention we reported previously [21].

Wearable devices have gained a significant share of the health and well-being consumer market, and new wearable devices and algorithms emerge frequently. Although a great majority of this research aims to detect sleep quality and duration based on data derived from accelerometer sensors [23], other measures such as respiratory signals have also been utilized [24]. Several reviews have evaluated the accuracy of accelerometer-based sleep wearables [23,25,26], and a recent review summarized an overall evaluation of wearables utilizing other sensors [27]. They concluded that detecting sleep from wake is relatively successful in many devices, but when wearables aim to classify sleep stages as opposed to simply distinguish between sleep and wake, there is a challenge in distinguishing four choices (wake, light, deep, and REM sleep) [27], which makes the result more inaccurate.

Commercial accelerometers typically yield accuracy between 0.81 and 0.91, sensitivity values between 0.87 and 0.99, and specificity values between 0.10 and 0.52 in distinguishing sleep from wake [26]. However, when attempting to detect sleep stages, the results are less consistent. A recent review focusing on commercial accelerometers identifying sleep stages found great variation in accuracy depending on the study [26]. For instance, accuracy in detecting light sleep varied between 69% and 81%, accuracy in detecting SWS was between 36% and 89%, and that for REM sleep ranged between 62% and 89%. Such variation suggests that acceleration itself may not be sufficient in reliably identifying sleep stages.

Previous studies have implied that HRV may be a useful marker for detecting sleep stages [10,11]. One study reported an accuracy of up to 89% in detecting SWS, but their method included respiratory signals alongside HRV [28]. When detecting sleep stages by utilizing both HRV and accelerometer data, one study managed to identify 75% of SWS correctly [29]. In that study, REM sleep was identified correctly in over 70% of epochs, whereas light sleep detection was the weakest with correct identification varying between 42% and 52%. Our findings are of similar accuracy, which further supports the notion of combining accelerometer and HRV-based measures for reproducible sleep staging.

This study was performed to evaluate the ability of HRV- and acceleration-based Firstbeat sleep analysis methods to detect sleep and different sleep stages. In pairwise comparisons, the Firstbeat method detected light sleep and SWS with no statistically significant difference to the gold-standard PSG method. There were two measures that differed significantly between the Firstbeat method and PSG: Firstbeat underestimated REM sleep (mean 18 minutes) and overestimated wake (mean 14 minutes). Considering the number of minutes in the context of a typical night’s sleep, the differences are not alarmingly high in practice, especially when measuring sleep over repeated nights. Sleep onset detection was very accurate, which is in accordance with a review published earlier this year [30].

Sleep stages can only be detected using PSG, as the stages are, by definition, separated by different patterns in ECG, EOG, and ECM. REM sleep is particularly difficult to detect without measuring activity from EOG and EMG channels. Thus, relying on other physiological measures as a means for separating sleep stages is always based on secondhand information. Although HRV has both previously [10,11] and in this study reflected sleep stages relatively well, it cannot detect the immediate changes in EEG, EOG, and EMG. However, this study suggests that HRV-assisted sleep stage detection can serve as a good estimate of sleep architecture despite being less accurate in detecting specific sleep stages.

When observing the comparisons in more analytical detail, we found that comparing the Firstbeat method against PSG yielded good specificity, and excellent sensitivity and accuracy in detecting wake. Regarding light sleep, the measures of specificity, sensitivity, and accuracy were less convincing. SWS detection had excellent specificity, adequate sensitivity, and good accuracy, while REM sleep was detected with similarly excellent specificity, adequate sensitivity, and good accuracy. These results suggest that the Firstbeat method is best at detecting sleep stages that have strong parasympathetic cardiac markers; however, light sleep is typically not significantly differentiated based its physiological fingerprint [12,14].

Our study was fully balanced in sex distribution and we were able to evaluate the Firstbeat method across two different nights in two different settings in the participants’ own homes. Thus, the ecological validity in this study can be considered excellent.

As a limitation, even though our sample had some variation in PSQI-measured sleep quality, this study did not include any participants with diagnosed sleep disorders. Our study included only healthy participants, and the results are likely to be different if any health issues, particularly cardiovascular, or any sleep disorders are present. This is a question to solve before utilizing the Firstbeat method in clinical contexts.

Combining HRV with accelerometer measurements can be considered a feasible method with sufficient validity to measure nocturnal sleep stage variation. We found that the specificity, sensitivity, and accuracy were the weakest in detecting light sleep. Nevertheless, considering its availability, affordability, and ease of administration, Firstbeat may be a useful tool in various contexts, particularly in consumer-based sleep-measuring environments to produce an overview of sleep structures.