The study was designed as a within-person comparative study where parameters estimated by the Philips health watch (DL8791, Philips, Stamford, CT, USA) were compared with measurements of reference devices. The study was performed in compliance with ISO (International Organization for Standardization) 14155 “Clinical investigation of medical devices for human subjects – Good clinical practice,” the Declaration of Helsinki, and local regulations. An independent medical ethics committee (METC Brabant) approved the study and it was registered in the Netherlands Trial Registry (NTR5552). Before participation all subjects gave written informed consent.

The primary objective of this study was to determine, in a clinical study, the accuracy of the Philips health watch regarding the estimation of total energy expenditure and resting heart rate. Secondary objectives included the assessment of accuracy of other output parameters of the monitor: heart rate, step count, activity type, and respiration rate at rest.

For this study, adult (≥18 years) participants with a body mass index between 19 and 35 kg/m² were recruited from the Dutch general population. Respondents with any of the following criteria were excluded from participation in the study: pregnancy, presence of skin conditions or wounds in the wrist area, presence of a chronic disease for which a physician had contraindicated moderate-intensity exercise without medical supervision, presence of a pacemaker or other implantable electronic device, or presence of a functional or cognitive impairment preventing compliance with the study protocol.

For each participant, the study started with an evaluation of his or her eligibility based on the inclusion and exclusion criteria and, if positive, giving informed consent. Then participants completed the Fitzpatrick skin type questionnaire [52] and received detailed information about the study procedure and the health watch that they received. Subsequently, a participant wore the health watch at home for 3 days during which he or she could carry out his or her normal daily life activities to gather free-living data; participants did not wear reference devices during the free-living period. The participants then performed a laboratory test comprising a variety of daily life activities, during which data were collected using both the health watch and various reference devices. The purpose of this laboratory test was to assess the accuracy of multiple health watch parameters compared with reference measurements.

The standardized laboratory protocol comprised the following activities: (1) indoor activities including rest (watching television) for 15 minutes and treadmill (4.5 km/h), treadmill uphill 5% (3 km/h), ergometer bike (60 rpm), cross trainer (60 W), household activities (mixture), desk work, lying down, and standing for 3 minutes each and (2) outdoor activities including walking, cycling, and running for 3 minutes each.

After each activity, there was at least 3 minutes of rest. The 15 minutes of rest at the start of the protocol was included for measuring resting heart rate in the laboratory and respiration rate at rest. The data from this activity were included in the cumulative energy expenditure that was analyzed for the primary objective. During this period, participants were sitting on a chair while watching an emotionally neutral documentary on a television. The mixture of household activities consisted of three 1-minute subactivities: washing dishes, folding towels and handkerchiefs, and vacuum cleaning.

If the outside temperature was less than 10°C (to remain compliant with the K4b² instructions for use) or when it was raining, the outdoor activities were performed indoors. That is, participants walked in the corridors at their own pace, cycled at their own pace on the ergometer bike, and ran on the treadmill at a pace that they set themselves. This occurred for 8 participants. Participants were asked to eat a light breakfast or lunch before the test, to not take caffeine or smoke in the 2 hours before their appointment, and to not carry out intense physical activity in the period before the test.

The investigational device for this study was the Philips health watch, a wrist-worn, PPG-based, heart rate and activity monitor (Figure 1). The watch measures the health parameters at a 1-Hz sampling rate and displays real-time heart rate values and daily cumulative values for steps, active energy expenditure, and total energy expenditure. The 1-minute average values for heart rate, and cumulative steps and energy expenditure over 1 minute, are logged in internal memory and transmitted via Bluetooth to a phone running the companion app for 24/7 monitoring. The companion app displays the parameters over time to provide insights to the user by, for instance, color coding optimal or suboptimal parameter values based on personalized settings that are automatically determined from international standards (i.a. WHO and ACSM) and based on user input and input from the health watch. Additionally, based on a user’s personal program for achieving, for instance, a certain daily energy expenditure, coaching cards pushed via the app provide further insight and motivation to promote behavior change toward a healthier lifestyle.

For assessing the accuracy of heart rate, the health watch heart rate was evaluated over the whole duration of the laboratory protocol. For this purpose, based on the 1-Hz sampled values, mean heart rate values were calculated for 10-second nonoverlapping epochs for the duration of the laboratory protocol for both the reference device (Actiwave Cardio) and the Philips health watch. These data were then compared between devices to determine error values and coverage values, which were defined as the percentage of time the difference between both devices (either absolute in beats per minute, bpm, or relative in percentage compared with the reference) was within specific limits (10 bpm and 10%, respectively).

As a reference device for the respiration rate during rest, the K4b² was used, which has been validated for this parameter [56]. The respiration rate was evaluated over the rest part of the laboratory protocol during which the participants were watching television. The mean respiration rate from the K4b² during this activity was compared with the mean respiration rate of the Philips health watch.

The accuracy of the step counting algorithm of the Philips health watch was determined by comparing with a Fitbit One (Fitbit Inc, San Francisco, CA, USA) device that was clipped onto participants’ trouser pockets as per manufacturer’s instruction. This waist-mounted step counter has been shown to be highly accurate for step counting during walking [57-59]. As a measure of the accuracy of the step counting algorithm for walking and running activities, the total number of steps for all these activities in the protocol was compared between the Philips health watch and the waist-mounted reference.

Activity type recognition was compared against the (reference) list of activities from the protocol. The health watch classifies measurement data into 4 different types of activities (walking, running, cycling, and other), where changes between activity types are registered with a corresponding time stamp. Each activity from the laboratory protocol was timed using markers at the beginning and end of each activity that were set by the researcher using a Garmin Forerunner 620 (Garmin International Inc, Olathe, KS, USA). The activity type classifications for the laboratory activities were defined as follows: both treadmill exercises and outdoor walking were defined as walking, stationary cycling and outdoor cycling were defined as cycling, outdoor running was defined as running, and, except for the cross-trainer activity, the remaining activities were defined as other. The cross-trainer activity was not taken into account for determining the accuracy of health watch activity type recognition, as it could be classified as walking or running depending on the intensity at which the participant performed the task. The accuracy of activity type recognition was determined by calculating the average percentage of correct classifications of the consecutive activity type outputs of the device compared with the reference activity type on a second-by-second basis.

A total of 31 participants were recruited, of whom 2 were excluded before data collection. This left 29 participants who took part in the trial, 14 male and 15 female. We observed the following distribution of the 6 Fitzpatrick skin types (1-6): n=0, 7, 18, 4, 0, 0. Demographics are presented in Table 3.

The TOST evaluation was applied at a significance level of .05, for equivalence margins of ±15%, leading to rejection of both null hypotheses [62] and therefore the conclusion that cumulative total energy expenditure as measured with the health watch and the K4b² were equivalent. Results of the cumulative total energy expenditure comparison between the Philips health watch and the COSMED K4b² ambulatory metabolic system are shown in Figure 2. As can be seen in Figure 2, the mean error (97% ratio, 95% CI 92%-101%) in total energy expenditure estimation (indicated by the thick black line) was well within the predefined 15% range of equivalence (indicated by the red dashed lines). There was a mean underestimation of 10.0 kcal, SD 38.9 kcal, which in relative terms was 2.9%, SD 13.1%, of the average reference value. The mean absolute error amounted to 27.5 kcal, SD 28.7 (MAPE 10.0%, SD 8.7%).

The TOST evaluation was applied at a significance level of .05, for equivalence margins of ±10%, leading to rejection of both null hypotheses [62] and therefore the conclusion that resting heart rate as derived from the health watch and resting heart rate as derived from the Actiwave ECG during the 15 minutes of rest in the laboratory test were equivalent. Results regarding the comparison are shown in Figure 3. The mean ratio was 100%, and the 95% CI was 99.5%-100.5%. In absolute terms, the mean absolute error was 0.2 bpm as most values were exactly equal to the Actiwave reference, with the maximum deviation being 3 bpm.

Additionally, for each participant separately, we visually assessed whether the resting heart rate from the health watch coincided with the heart rate values at rest during the free-living part of the protocol. Figure 4 shows an example of a heart rate trace measured by the Philips health watch over 3 days of free-living conditions, together with the resting heart rate that was reported over time by the health watch (top), and the total energy expenditure estimation for the same time period (bottom). It can be seen that the resting heart rate corresponds with a low segment of the heart rate trace. In addition, the values sampled for each participant seem to correspond to time periods where subjects were awake and inactive, as can be deduced from the total energy expenditure graph, where sleep can be recognized as periods of low total energy expenditure with relatively low fluctuation.

Table 4 presents the results regarding heart rate measurement accuracy. The mean error was −1.7 bpm and the mean absolute error was 3.1 bpm. This corresponds to a mean percentage error of −1.3% and a MAPE of 3.1%. Of the available comparative data, the health watch measured heart rate within a difference of 10 bpm with the Actiwave ECG-based reference heart rate 94% of the time and within 10% of the ECG value 93% of the time (see Table 5).

Compared with the step count reported by the waist-mounted reference, an overall (average) underestimation of 21 steps was observed corresponding to an overall error of −1.6% (see Table 6). Again, the TOST evaluation was applied at a significance level of .05, for equivalence margins of ±10%, leading to rejection of both null hypotheses and therefore the conclusion that cumulative steps as estimated by the health watch and by the waist-mounted counter were equivalent. The calculated 95% CI boundaries for all walking activities combined were at 97.1% and 99.6% (Figure 5).

Overall, more than 90% of the time the correct activity type was identified by the Phillips health watch during the annotated laboratory activities (Table 7).

The TOST evaluation indicated that respiration rate at rest as measured by the health watch and the K4b² were equivalent (.05 significance level, ±10% equivalence margins). Figure 6 shows the boxplot of the ratio between the health watch and the K4b² respiration rate, both averaged over the 15-minute rest period at the beginning of the laboratory protocol, with the calculated 95% CI boundaries of 93.4% and 99.5%. These data indicate a slight underestimation of respiration rate compared with the reference by -0.7, SD 1.1, breaths per minute. In percentages this amounted to an error of -3.8, SD 8.1, percent. The absolute mean error was 1.2, SD 1.0, breaths per minute (MAPE 8.3%, SD 7.0%).

In this study the measurement accuracy of the Philips health watch, a wrist-worn heart rate and activity monitor, was evaluated against (medical) reference instruments. Resting heart rate was determined on heart rate sampled over a 15-minute resting protocol in sitting position and fell within 3 bpm of the Actiwave ECG comparator. During a protocol covering a variety of activities of daily life, the health watch measured total energy expenditure on average within the predefined 15% accuracy compared with a K4b² mobile metabolic system. These results indicate that the watch can provide valuable information that can help in the prevention and management of lifestyle-related chronic diseases by measuring and tracking resting heart rate and energy expenditure over time and interpreting these data in the context of a user’s personalized range of a parameter value or goal based on international standards (i.a. WHO, ACSM). The (automated) coaching that the companion app provides uses the information to offer further support to making a lifestyle change.

Energy expenditure estimation by means of wearable devices is not a new concept. There are many commercially available activity monitors that provide energy expenditure estimates; however, the reported accuracy of consumer-grade devices is highly variable [30,63-66]. Comparison across studies is hampered by differences in the type of reference measure (eg, doubly labeled water, metabolic chambers, or mobile metabolic systems) and differences in the type, intensity, and duration of activities performed during the validation (eg, standardized treadmill walking or free-living evaluation). Similar to our study, Lee et al [30] evaluated several consumer-grade physical activity monitors against a portable metabolic system (Oxycon Mobile) over a 69-minute standardized protocol of various activities. They reported MAPEs of 9.3% (BodyMedia FIT) up to 23.5% (Basis B1 band). Another study that resembled our design was performed by Bai and colleagues [67]. They evaluated activity monitors against an Oxycon Mobile metabolic system over an 80-minute standardized protocol. MAPE values ranged from 15.3% (BodyMedia Core) to 30.4% (Misfit Shine) in this study. Most recently, Nelson et al [68] evaluated several activity monitors over a 65-minute protocol covering 10 minutes of rest and a selection of 11 different activities using a COSMED K4b² as reference. They reported MAPEs ranging from 13% to 35% for energy expenditure prediction over the different activities. In this study, the Philips health watch MAPE for total energy expenditure was 10%, which is highly accurate for this type of device when compared with the performance reported in the aforementioned studies. It is important to realize that the heart rate and acceleration measurements will give an estimation of total energy expenditure, which is less accurate than objective measurement techniques such as doubly labeled water or indirect calorimetry by means of ambulatory metabolic systems. However, these measurement methods are not feasible for long-term 24-hour monitoring of total energy expenditure in daily life and are not readily accessible to consumers [22]. In comparison with self-report questionnaires for physical activity, the Philips health watch provides a more objective measurement of total energy expenditure that is well suited for long-term, noninvasive monitoring. Similar to other validation studies of energy expenditure estimation, our study was limited to an evaluation of participants for a limited time frame at our test facility, as home testing was not practically feasible with regard to obtaining within-person reference measurements [63]. A strength of the study was that the protocol included activities of daily life, such as desk work, household activities, and activities performed outdoors, in addition to more traditional treadmill-based protocols. This will provide a better reflection of daily life performance [69].

In this study the resting heart rate value determined with the health watch was found to be equivalent to that of the Actiwave reference (Figure 2). With the mean bias centered around 0 and a maximum deviation of 3 bpm, the health watch is suited for inspection of resting heart rate as a risk factor, as dose-response investigations often report increments in hazards for 5-10 bpm increments of resting heart rate [31,32,34,70]. A limitation of the study was that it was not possible to evaluate the resting heart rate produced by the health watch against a gold standard because there is currently no international consensus on a standardized manner to obtain resting heart rate values. We did, however, follow current recommendations by assessing resting heart rate based on multiple heart rate samples during a 15-minute resting protocol [50,55]. Furthermore, inspection of individual free-living heart rate traces indicated that the resting heart rate estimates of the health watch correspond to low heart rate levels during nonactive, although nonsleep, periods throughout the day. A strength of this method of obtaining resting heart rate by means of continuous heart rate monitoring is that it is much less influenced by circadian or temporary factors such as the “white coat effect,” which can confound the measurement [51,71].

Continuous heart rate logging of the Philips health watch was evaluated against Actiwave measurements over the complete duration of the protocol. Values sampled at 1 Hz were averaged over 10-second nonoverlapping windows. Parak and Korhonen [29] performed a similar comparison over a 50-minute protocol of various activities with a Mio Alpha, a wrist-worn device using a predecessor sensor module to that of the Philips health watch. They found a mean absolute error of 4.43 bpm and MAPE of 5.23% using 5-second nonoverlapping windows. In comparison, the Philips health watch performed better in this study with a mean absolute error of 3.1 bpm and MAPE of 3.1% (Table 4). Additionally, the coverage of the health watch within 10% of the reference device was higher in this study with 93.1% (Table 5) versus 87.5% reported in the study by Parak and Korhonen. For some activities, we observed lower accuracy and coverage presumably owing to the relative short duration of the activities. Our hypothesis was that the short duration of the activities resulted in relatively steep rising and dropping of heart rates, thus negatively affecting the estimation accuracy for only these activities compared with a protocol with longer activity durations resulting in more stable heart rates. Furthermore, during the months of testing (February and March) a temperature shift from the warmer indoor temperature to the lower outside temperature may have caused temporary localized vasoconstriction leading to lower coverage values for the outdoor activities walking and bicycling (Table 5). This phenomenon has been observed in an experimental setup by Maeda and colleagues [72], who demonstrated that the pulsatile AC (alternating current) component of the PPG signal is significantly lower at skin temperatures below 20°C compared with normal skin temperatures. This results in a significantly lower AC/DC (direct current) component ratio and reduces the correlation with ECG-based heart rate measurements. Although the accuracy of the health watch is equivalent to an ECG-based comparator for a high percentage of time, deviations due to, for example, a poor sensor-skin contact or movement artifacts are still possible. Attention to correct wearing of PPG-based devices is therefore important. Furthermore, for the purpose of this device, a 24/7 heart rate and activity monitor, a 93.1% coverage within 10% can provide a good overall representation of a user’s heart rate over the course of a day.

Step counting of the health watch was compared against a waist-worn step counter over walking activities at different speeds, indoors on a treadmill and outdoors. From prior research, it is known that waist-worn devices generally have less error in step counts when compared with observed counts than wrist-worn devices [57-59,73]. Compared with the waist-worn counter, the health watch had a slight average overestimation of 0.3% for treadmill walking at 3 km/h at a 5% inclination and a small underestimation of −3.7% for treadmill walking at 4.5 km/h at 0% inclination (Table 6). These errors were smaller than those reported by Diaz et al [58], who found a mean underestimation of 16.3% and 10.6% when comparing a wrist-worn step counting device with observer counts over comparable slow and moderate speeds.

Activity type recognition is useful for physical activity monitors as it can give insight to users into the duration of different types of activities that were performed over the course of a day and what amount of energy expenditure was associated with this. Furthermore, activity type classification provides the potential to enhance energy expenditure estimation [22,74-78]. Overall, more than 90% of the time the correct activity type was identified by the health watch during the annotated laboratory activities. This is a good result when comparing with other studies of automatic activity type recognition, where overall correct classifications range from 42% to 96% depending on the types of activities classified [79-82]. In Table 7 it can be seen that the least accurate activity type was running at 89.9%. Running was classified as walking approximately 6% of the time. One reason that the running recognition was least accurate could be the fact that 2 participants were actually walking during the running part of the protocol.

This study showed that the health watch can estimate total energy expenditure with 85% accuracy during daily life activities and measure resting heart with ±3 bpm accuracy during rest compared with medical device reference instruments. In addition, the secondary outcome parameters, heart rate, step counts, resting respiration rate, and activity type classification, showed high levels of accuracy. On the basis of these results the health watch can serve its medical purpose of measuring resting heart rate and total energy expenditure over time in an unobtrusive manner, thereby providing valuable data for the prevention and management of lifestyle-related chronic diseases.