We aimed to recruit 40 members of the university’s staff and students through department-approved internal emails for a pilot study assessing the feasibility of wearable-based observational studies examining physical activity, nutrition, and mental well-being. Out of this pilot study group, a convenience sample of 10 participants would be invited to participate in an additional validation component to examine the accuracy of heart rate measures provided by Fitbit Charge HR (Fitbit, San Francisco CA, USA). Participants could be included if they owned a compatible mobile phone with a data plan, were aged between 18 and 65 years, and were unlikely to travel abroad over the subsequent 1 month. The following criteria excluded an interested participant from the study: having a severe medical condition that would prevent participation in physical activity, discomfort, or unwillingness to wear multiple devices concurrently and participation in activities or work that would restrict the use of the devices.

We compiled baseline characteristics for all our participants by means of a self-administered questionnaire. Measures of height and weight were taken using a SECA stadiometer (SECA GmbH, Hamburg, Germany) at our study site.

Each participant was provided a new Fitbit Charge HR (Fitbit) tracker to be worn on the nondominant hand throughout the pilot study. Participants were instructed in the use of the tracker device and the installation of mobile phone apps according to manufacturer’s specifications. They were also taught to synchronize the Fitbit tracker periodically.

Each of the 10 participants in the heart rate validation series were also fitted with one Polar H6 heart rate monitor (Polar Electro Oy, Kempele, Finland) worn across the chest. To record the Polar H6 heart rate monitor (Polar) data, these participants were provided with an Actigraph GT3X+ logger (Actigraph) on Bluetooth receiver mode set to sample measures at 10-s intervals and worn on the same wrist as the Fitbit device. The 10 participants were asked to wear all 3 devices for at least 3 and at most 6 continuous hours of nonsleeping activities. They were encouraged to continue pursuing their usual activities, excluding water sports. The Polar and wrist-worn Actigraph devices could be removed before bedtime and returned to the study site over the following days. Participants would continue to use their Fitbit trackers for the remaining 1 month of free-living study. Participants used their personal data plans to run synchronizations with the Fitbit server.

Fitbit heart rate measures were downloaded directly from the Web server using a developer’s application programming interface (API) issued by Fitbit. Polar measures were collated from the wrist-worn Actigraph devices. Common wear time for the validation study was defined as every 1-min epoch, which reflected a nonzero heart rate on both devices. For the 1-month period of continuous monitoring, any nonzero heart rate registered by the Fitbit was defined as valid 1-min epoch of wear time. A valid day of wear time was defined as having at least 600 1-min epochs of nonzero counts within 1 calendar day.

Our first dataset comprised one Fitbit heart rate measure for each discreet 1-min epoch, whereas six 10-s measures for the same epoch were available from the Polar device. While the Fitbit API allowed us to review heart rate measures at intervals less than 1 min, the time differences between measures were irregular. Given that Fitbit users would only have access to data logs recorded by minute, we did not attempt to generate our own summary measures for within-minute heart rates. It is worth noting that the literature recommend that photoplethysmographic readings should be averaged over a 60-s duration to obtain a reliable measure [10]. To appreciate the data contributed by individual participants, we generated time series plots of discreet 1-min epochs where Fitbit measures were superimposed onto ranges of Polar 10-s measures. Thereafter, 10-s Polar heart rate measures were averaged for each 1-min epoch. Subsequently, we rank-ordered aggregated discreet epochs by their average Polar measure, divided these epochs by deciles, and constructed box plots to compare average Polar and Fitbit measures. Box plots for the width of within-epoch ranges of 10-s Polar measures were included in this descriptive plot.

Intraclass correlation coefficients (ICCs) were calculated using a mixed effects model assessing for absolute agreement between average 10-s Polar measures, and 1-min Fitbit measures. ICCs were calculated first for overall measures and then for measures stratified by physical activity heart rate zone cut-offs proposed by the Centers for Disease Control and Prevention [25] and by individual participants. Given that the Polar device was chosen as reference, heart rate zones were assigned according to each participant’s age and average 10-s Polar measure within discreet epochs. Two Bland-Altman plots were constructed to visually evaluate the overall differences in absolute measures within heart rate zones. A two-by-two table was constructed to estimate the sensitivity, specificity, and positive and negative predictive values for Fitbit devices, correctly identifying MVPA heart rate zones where average 10-s Polar values were considered as reference.

In our second dataset, we compiled all Fitbit heart rate measures obtained in the 1-month free-living study period. The aggregated valid days and epochs of wear time were compiled in a bar chart with superimposed dot and whiskers plots. We tabulated summary statistics for each participant, detailing the number of 1-min epochs spent in respective heart rate zones under free-living conditions.

All statistical analyses were conducted using STATA (Version 13.1, StataCorp LP). Given the exploratory nature of this study, P<.05 was chosen as a level of statistical significance. The strength of ICC coefficients was interpreted based on the following definitions: weak (r<.5), moderate (.5-.7), and strong (r>.7). This study was approved by the institutional review board of the National University of Singapore.

From our pilot study group of 20 males and 20 females, we recruited 3 females and 7 males to participate in the heart rate validation segment. Recruitment and data collection began on November 4, 2015, and the last day of assessment was January 7, 2016. Nine out of 10 participants were students, and their average body mass index was 22.9 kg/m² (standard deviation [SD] 3.8). Table 1 describes the characteristics of the final sample of 10 study participants.

Of a total of 2769 possible 1-min epochs, 2509 valid paired readings were identified, with each participant contributing on average 250 (SD 95) epochs. Unpaired readings were treated as missing data and omitted from further analyses. For illustrative purpose, 2 of the 10 time series plots are shown in Figure 1, representing the strongest and weakest measures of intraclass correlation.

The graphical comparison of aggregated epochs in Figure 2 shows how heart rate measures from Fitbit were consistently lower than Polar, whereas the width of 10-s Polar value ranges remained consistent.

Table 2 shows the ICCs and differences between Fitbit and Polar measures. The overall ICC between both devices was strong (.83; 95% CI 0.63-0.91) and ranged from .40 to .97 across participants. The ICC was markedly weaker at MVPA heart rate zones as compared with the low heart rate zone. On average, Fitbit devices measured heart rates that were −5.96 bpm (95% CI −6.33 to −5.60) lower than Polar. Reviewing the differences between participants, we noted that the underestimate was statistically significant in all but 1 participant, who also demonstrated the strongest ICC. Again, the difference between both devices was greater in MVPA heart rate zones. This finding was reproduced in the Bland-Altman plots of measures (Figure 3).

Within the aggregate 2509 1-min epochs shown in Table 3, only 363 were spent in the MVPA heart rate zone according to Polar measures.

Figure 4 shows that wear time on valid days was consistent, although the number of participants providing valid days of device usage declined over the course of the study period.

On valid days of device usage as shown in Table 4, 24.4% of epochs were classified as nonwear time, 71.7% of epochs fell within the low intensity heart rate zone, and 3.9% of epochs were classified as MVPA. Thus, on average, participants spent 55 min (SD 34) per day in the MVPA heart rate zone.

In this free-living validation study, we have compiled rich data by relatively convenient means, capturing 2509 1-min epochs of paired data and tens of thousands of unpaired 1-min epochs in the follow-up period. Visual inspection of within-participant heart rate plots showed that there were differences in how well Fitbit readings coincided with Polar ranges. We ascertained an overall strong ICC for absolute agreement between Fitbit and Polar measures that varied markedly between participants and diminished at heart rates that represent moderate to vigorous intensity physical activity. The Fitbit devices identified just over half of MVPA heart rate zone readings correctly.

Our summary measure of a 6 bpm or 7% underestimate of heart rates measured by Fitbit was in keeping with current literature where error estimates have been established under laboratory conditions [18,26,27]. In MVPA heart rate zones, we found an average underestimate of 16 bpm, which was shown to markedly impact Fitbit’s ability to correctly identify time spent in MVPA under free-living conditions. This finding of a larger error at higher heart rates is consistent with other studies [26,28]. However, our findings suggest that the systematic underestimation of heart rates might partially be accounted for by differences between participants.

Applying our own measures of sensitivity and specificity to the data obtained during the 1-month follow-up period, we could surmise that close to an hour’s worth of MVPA epochs were not captured on any given day. This underestimate of daily MVPA time contrasts with the overestimate observed in a free-living validation study of a wrist-worn tracker of the same brand that measured bouts of activity based on accelerometry [29]. It is also important to note that Fitbit wear time and the number of participants wearing it sufficiently long had decreased considerably even over the 1-month monitoring period. This is consistent with findings from other studies [30] that have also reported considerable drops in compliance with wearable device use over time.

Our free-living validation study into the accuracy of wrist-worn heart rate monitors has several implications on their potential usefulness in monitoring relevant physiological parameters over time and tracking compliance with exercise prescriptions. The results are in keeping with the past studies, which concluded that the device would fare poorly in the calibration of intensity of activity owing to insufficient accuracy. Concerning activity tracking, we found that the devices would fail to recognize one in two MVPA heart rate zone epochs, thus diminishing their value as a means of assessing activity levels objectively. Our follow-up data suggest that device use declined over the course of the study, further complicating potential uses as a compliance monitoring tool.

Overall, our findings have demonstrated that more emphasis should be placed on eliminating systematic error in the tracker measures. Our data showed that errors might be explained in part by putative between-participant differences that would include device fit and skin surface characteristics. Additional mathematical calibration might be appropriate for the trackers to more reliably detect MVPA heart rate zones. As these sensors become increasingly ubiquitous, their potential role in exercise prescription and health promotion merits further evaluation.

Our study design was limited to a small sample of mostly male, young adults who did not report significant health issues, thus limiting the generalizability of our findings. The recruitment of female participants was affected by expressed discomfort wearing the Polar H6 chest strap beneath undergarments. In addition, the measurement period was restricted over a few hours of one day where participants would engage in their normal activities. Although this provided us the necessary information on usual day-to-day life, it resulted in a limited number of paired measures, particularly in the moderate to vigorous heart rate zones. We did not include any form of activity diary in the 1-month period of follow-up, thus limiting our ability to verify the total duration of MVPA accrued. The sensitivity of the photoplethysmographic probe is strongly affected by placement and skin condition [10]. Although we provided advice on proper fit, it is plausible that participants may have foregone scientific accuracy in favor of personal comfort by loosening wrist straps. Due to the logistics of the study, we were unable to ascertain proper fit at the end of the observation period, which could have provided further insights into the apparent interpersonal differences in Fitbit accuracy. Finally, it is important to note that the optical sensors provide a measure of microvascular perfusion, whereas our reference device registers myocardial electric potentials. In practice the arterial pulse rate is often synonymous with the rate of cardiac contractions, but this level of equivalence might be considered inappropriate in the evaluation of photoplethysmographic devices, which are known to be affected by movement and other artifacts that contribute to error rates of up to 8% [10].

The nature of this study was part validation and part exploration. While the overall ICC for absolute agreement appears strong, our data suggest that under free-living conditions, Fitbit Charge HR trackers overall compared poorly against the reference device, especially at higher heart rate zones. Our findings are in line with findings of past studies, which have expressed concern that such devices might not provide adequate information to guide exercise intensity or detect MVPA. Given the nature of our small pilot study with a limited period of observation, further research with a larger sample is warranted to confirm our results.