Wearable fitness trackers have gained popularity worldwide, and their annual sales continue to grow [1,2]. These trackers were listed as the No. 1 fitness trends in the years 2016, 2017, 2019, and 2020 in a worldwide survey conducted by the American College of Sports and Medicine [3-6]. The advantages of these wearable devices are that they are convenient to use and measure various parameters noninvasively. In addition, they allow the users to monitor their daily physical activities in a free-living environment instead of controlled laboratory settings.

Earlier versions of fitness trackers, equipped with triaxial accelerometers and a gyroscope, could sense motions made by the users, monitor their activity metrics, and provide estimated information such as walking and running in terms of steps or distance, energy expenditure, sedentary time, sleep patterns, and activity routes (with GPS function). Most of these fitness trackers were placed on the wrist. The users obtained the real-time information from the display on the trackers or received feedback through connected mobile phone apps.

The recent application of photoplethysmography (PPG) in wrist-based wearable fitness trackers has enabled newer versions of fitness trackers to detect heart rates. This breakthrough provides several benefits. First, heart rate is a vital component in cardiovascular fitness assessments and an important parameter in exercise training programs [7]. Second, resting heart rate is also a widely used parameter for general health assessments to detect cardiovascular diseases [8]. Thus, the development of fitness trackers that have heart rate detection technologies has brought about several additional benefits that were absent in older models.

PPG measures heart rates based on the changes in vascular blood flow during the cardiac cycle [9]. It has previously been applied in medical devices such as oximeters [10]. This technology has since been integrated and commercialized as optical heart rate monitors by companies such as Mio and Omron. The number of commercial companies producing such devices has gradually grown in the last 5 years (ie, Apple Watch, Fitbit, and Garmin), along with the design and development of such products and research [1,2,10-16].

Despite the growing popularity and functions of these fitness trackers and substantial investments in commercial advertisements, many users have expressed concerns regarding the data accuracy of these trackers [17]. Inaccurate and inconsistent readings are major reasons for negative user experiences, which discourage the continued use of these devices [17-20]. The concerns regarding the data accuracy of these trackers influence the users in terms of their perceptions of personal health and program interventions or research evaluations that adopt these devices.

Most commercially available fitness trackers use step counts as a parameter to indicate the level of physical activity. The step-count function of these devices has been widely scrutinized in studies examining their accuracy [21-23]. Importantly, while generally producing accurate results, these devices did not report reliable step-count readings in certain conditions, such as slow walking or while performing unnatural hand movements [21-24]. A systematic review investigated the validity and reliability of Fitbit and Jawbone trackers. The results revealed that most studies validated the tracker accuracy and indicated that it had a higher accuracy for step counts, followed by that for distance and physical activity and finally for energy consumption and sleep [23]. Nevertheless, most studies recommend caution when deriving energy expenditure estimations directly using these readings [11,13,25,26]. In addition, studies have started to examine the validity and reliability of the fitness trackers among older adults instead of young adults because they might present different movements such as gait patterns or speeds [27,28].

The accuracy of heart rate displayed on the fitness trackers with optical heart rate monitors has also been investigated [11,14,16,29-31]. Common research methods for the development of these optical heart rate monitors involve fitness assessments using basic indoor training equipment such as treadmills, stationary cycles, and sometimes elliptical machines. This type of study allows researchers to evaluate the feasibility of implementing optical heart rate monitors in aerobic training for the general population [1,2,10-16].

Previous studies have reported that, generally, optical sensing fitness trackers have acceptable accuracy. However, the accuracy might vary across brands [16,31] in terms of activity patterns or speed, exercise intensities [10,14,31], skin tone [10], room temperature [32], placement of sensors [29], or compression-induced and motion-induced artifacts [13,32-34]. For example, in a study conducted by Boudreaux et al [13], participants wore 8 different fitness trackers, and an increase in exercise intensity reduced the accuracy of heart rate measurement. In another validation study, the measured heart rate showed a minor deviation compared with the actual heart rate in participants with a dark skin tone [30].

Although the adoption of heart rate fitness trackers with optical heart rate sensors in the medical field is still debatable [12,35], there have been several lawsuits regarding the accuracy of heart rate information [36,37]. Assessing the reliability and validity of the heart rate readings provided by these trackers is essential because they are vital in clinical settings, and these trackers have been increasingly accepted by consumers as a tool for self-monitoring or in many intervention programs for health management [11,14].

Owing to the limitations on raw data acquisition in commercial fitness trackers, previous studies have only used average heart rate data [14] or manually recorded the heart rate at certain intervals [11]. However, averaging the heart rate or recording it at a certain time point is problematic because both fail to represent any change or variability [38]. Studies that have compared continuous heart rate in more detail revealed that evaluating the accuracy of these test devices at a second-by-second level is difficult [2]. One study used video recording to manually determine the second-by-second heart rate, which was a labor-intensive and time-consuming method [12]. Moreover, potential variables such as age, ethnicity, and gender were not considered in earlier studies [2,14]. For example, a majority of the participants of several studies that have been conducted in the US-European regions were white (Fitzpatrick skin type I or II) [2,12,16]. PPG technology uses an optical sensor that illuminates light and measures the change in light absorption by the skin, which varies with change in blood volume; thus, the accuracy of heart rate monitoring using PPG is subject to skin structures [39]. Typically, the skin changes with age, that is, “fine wrinkles, roughness, mottled hyperpigmentation, dilated blood vessels, and loss of skin tone” are observed [40]. In addition, age-related changes such as arterial stiffness can influence the pulse shape in PPG [32]. Therefore, appropriate validation of these devices for different age groups among non-white participants is imperative.

This study evaluated the heart rate reading performances of 2 commercially available fitness trackers in various settings using a second-by-second data acquisition approach. Moreover, to determine whether age would generate discrepancies in the readouts, young and senior participants were characterized separately. This study was conducted in Taiwan to validate 2 trackers used by the yellow skin tone population (Fitzpatrick skin type III or IV) [41,42].

To determine a credible sample size for achieving statistical power in the intraclass coefficient correlation (ICC) test, this study used R package (ICC.Sample. Size, GPL-3; 2015, R core team, R Foundation for Statistical Computing). Based on the formula proposed by Zou [43], the number of participants (n) required for achieving a target power of 0.90 was 8. Therefore, this study involved 20 adults aged 65 years and above (Senior) and 20 adults aged between 20 years and 26 years (Young). All participants had no clinical history of cardiovascular diseases, neurological disorders, lower limb injuries, or any other factors that would render them unfit to perform the exercise. To ensure consistency, individuals with tattoos or birthmarks on the position where the device was to be worn were not included in the study. To minimize possible sex-driven discrepancies, the sex ratio in both the Senior and Young groups was kept identical (20:20).

This study used the Polar heart rate strap (H7, Polar Electro Oy), widely used as the criterion for measuring heart rate in sports science studies [2,44]. The optical fitness trackers selected for this study were Xiaomi Mi Band 2 (Xiaomi Cooperation) and Garmin Vivosmart HR+ (Garmin International Inc) because these 2 fitness trackers share a significant market share in the Asia Pacific region, which is expected to grow. Mi Band 2 was equipped with a PPG module (with 2 LED lights) and an accelerometer to detect heart rate and sense motion. Vivosmart HR+ was also equipped with a PPG module (with 3 LED lights) and an accelerometer. In addition, GPS chips are embedded in the Vivosmart HR+ for measuring the travel distance during outdoor exercises.

Both the devices provided information regarding step counts, energy expenditure, notification for breaking up the prolonged sedentary time, and smart notifications, and both claimed accurate heart rate detection. In addition, the 2 devices had the broadcast heart rate mode, a feature that enables the transmission of second-by-second heart rate data through Bluetooth or ANT+ to the paired receiving device, and served a similar function of the conventional heart rate strap. Moreover, wrist-based fitness trackers were easy to wear and remove and, thus, eased the discomfort of wearing chest straps for monitoring the real-time heart rate during traditional exercise and fitness training programs or interventions [10,45]. Specifically, PPG fitness trackers provide pulse rate data that are obtained with an increase or decrease in blood pressure in the arteries because of the contraction and relaxation of the heart, thus leading to a noticeable pulse. Although the signals of pulse waveforms are different from those of heartbeat waveforms, the pulse rate can be analyzed to represent the heart rate [32]. The term heart rate has been used in this study in line with many studies on heart rate fitness trackers [2,10-12,15,29,30,38,46]. Hence, in this study, the heart rate will be used in its broadest sense to refer to the readings from the optical fitness trackers.

The second-by-second heart rate data-receiving app Cardio Training (Angelfmarcos) used in this study was acquired from the Android platform. The equipment adopted in this study included 3 indoor aerobic fitness equipment: treadmill, upright stationary bike, and elliptical machine (Figure 1). These types of equipment were widely demonstrated in the previous exercise protocols and proved to be ideal and safe for aerobic training [2,10,47-49].

The results of MAE, MAPE, and correlation tests from both the Young and Senior groups are shown in Tables 1 and 2. In the Young group, the Garmin device achieved MAPE values of less than 10% in all the conditions tested (Table 1), indicating that overall, the heart rate readings produced by the Garmin device were reliable [2,13]. By contrast, whereas the Xiaomi device generally achieved MAPE values of less than 10%, it did not do so during cycling and elliptical phases (Table 1), suggesting that the reliability of the Xiaomi device was likely influenced by the types of activities performed.

In the Senior group, the performances of both test devices during different activities were reliable (MAPE values below 10%, Table 1). Notably, the MAPE values achieved by the Xiaomi device were, on average, higher than those produced by the Garmin device, indicating that the Xiaomi product was overall less reliable than the Garmin one. However, the standard deviation of MAPE achieved by the Garmin device was higher in the Senior group (SD_Senior=10.49%) than in the Young group (SD_Young=6.9%; Table 1), suggesting that the reliability of the Garmin device was likely affected by age differences and that it became less reliable in the older population.

The data revealed that the Garmin device achieved CCC values above the designated threshold (0.80) in both age groups (Table 2), suggesting that it was generally accurate. By contrast, the Xiaomi device failed to achieve overall CCC values above the designated threshold in both age groups (Table 2), indicating that it exhibited suboptimal accuracy in heart rate sensing. Notably, similar to the MAPE values described earlier, whereas the Xiaomi device achieved identical CCC values in both age groups (CCC_Young=0.73; CCC_Senior=0.73), the Garmin device’s CCC values fluctuated between the 2 age groups (CCC_Young=0.93; CCC_Senior=0.80; Table 2), indicating that its accuracy was also likely influenced by age differences. Taken together, these data suggest that the Garmin device, in general, produced more reliable and accurate heart rate readings than the Xiaomi one.

To observe the overall trends and identify any apparent discrepancies in the correlation in different situations, each phase within the exercise sequence was plotted separately and color coded. The overlaid datasets of the different groups are represented in the scatter gram in Figure 2. Notably, the correlation of certain activities, such as cycling, was found to deviate from the criterion measurements much more frequently than activities such as walking. This was further confirmed using the Bland-Altman analysis (Table 1; see Bland-Altman Analysis).

Bland-Altman plots indicating the mean difference in heart rate detection between Garmin or Xiaomi and Polar H7 criterion measure and levels of agreement with 95% CIs for the Young and Senior groups are illustrated in Figure 3. The complete Bland-Altman analysis dataset is presented in Table 1 (the Bland-Altman plot for each activity phase is provided in Multimedia Appendix 1). The data indicated that both test devices achieved relatively higher variations during cycling phases compared with other activities (Table 1). These results suggest that both devices tended to underreport heart rates in certain situations, consistent with previous observations [16,24]. Notably, the Xiaomi device significantly underestimated heart rates during cycling and elliptical phases in the Young group (−13.4 bpm and −13.3 bpm, respectively). Moreover, the differences between the upper and lower limits during the recovery phase (rest sitting between active phases) were greater than those during the resting phase (rest sitting in the beginning; Table 1). This implies that the variation of differences was greater at the transitional phases in which participants changed their activities from dynamic exercise to recovery, and thus, the degree of errors might decrease gradually if the participants stay in the rest position.

Various combinations of correlation tests are frequently adopted in evaluating the reliability or validity of examined devices [35]. As such, 3 independent statistical tests were employed in this study to compare whether the results from different correlation tests would deviate.

The obtained results (Table 2) revealed that the PPMCC test might compute a higher correlation coefficient than the CCC and ICC tests. The results of all the phases were quite identical; for example, the maximum difference was less than 0.01 (0.7258 and 0.7341 for Mi Band 2 in the Senior group). However, the difference between CCC or ICC and PPMCC was more obvious for activities; for example, a higher deviation was noted for activities such as cycling and elliptical exercise.

In line with previous studies [2,11,16,53], the combined results from this study indicated that both the Garmin and Xiaomi devices generally provided accurate heart rate readings. Both devices were also considered reliable in heart rate measurements with overall MAPE values below the 10% threshold. Notably, even though both devices achieved acceptable overall correlations in both age groups, they showed a tendency to modestly underestimate heart rates in many situations, as revealed by the Bland-Altman analysis. Similar findings were also reported in previous studies [11,12] and could represent a general characteristic of optical heart rate fitness trackers.

However, it is worth noting that significant discrepancies in device accuracy remained apparent between different physical activities. In general, these devices would be more accurate during sedentary behaviors such as sitting compared with active exercise [2]. Indeed, a previous study on a number of commercial wearable activity monitors have found that most devices exhibited low ICC values (r<0.5) when the activity intensity exceeded 100 watts in graded cycling exercise [13]. Similarly, our data revealed that the test devices generally had lower correlation coefficients and higher degrees of deviation during cycling and elliptical exercises compared with other activities.

In addition to activity intensity, several other studies have identified that motion artifacts during exercise were negatively correlated with the accuracy of PPG heart rate–monitoring systems [32,38,46,54-56]. For example, in an experiment conducted by Gillinov et al [2], the optical devices exhibited more accuracy for exercise with fewer arm motion artifacts (cycling and elliptical exercise with no arms movement). It is somewhat surprising that the data collected in this study indicated the opposite (as cycling produced less motion artifact than running). Nevertheless, Benedetto et al [12] found that the Fitbit charge 2 had poor ICC values (r=0.21) and underestimated the actual heart rate values when performing stationary cycling. Without further conclusions, users should be cautious when relying on optical heart rate readouts during various physical activities. Taken together, this study provides supporting evidence for a negative correlation between activity type and the accuracy of optical heart rate sensors but not between motion artifacts and the accuracy of optical heart rate sensors [2,13,24]. The precise mechanisms for such correlations currently remain unclear.

The profoundly expanding aging population worldwide is creating challenges for all sectors in the society. Promoting health condition of the older adult population and motivating them to engage in regular physical activity have become essential [57]. The adoption of new technology such as using health-related informatics technology (such as apps) or wearable fitness trackers is increasing [20,58,59], and the benefits are also observed in the senior population [20,60]. The fitness trackers validated in this study appear to exhibit similar accuracy for heart rate detection among different age groups.

Given its more thorough data acquisition method, this study had identified certain unexpected outliers. As shown in Figure 4, these extreme readings were unexpected, unpredictable, and transient. It is likely that these extreme readings did not represent the true heart rate values and that their displays were technical faults of the devices or the detection approach. Nonetheless, these random (or untrue) readings can skew the overall dataset and falsely represent the heart rate of an individual. Because these extreme heart rate readings were only observed for a short period, detecting these deviations while examining the heart rate readings every 15 seconds, every minute, or only at the end of the exercise, as in earlier studies, is difficult [11,15,16]. Given the transiency of such extreme readings, it is therefore recommended that future studies on optical heart rate sensors adopt a second-by-second approach demonstrated here and previously [12] to identify the outliers.

Previous studies have proposed the use of different statistical methods to analyze the data correlation. These include the MAPE test, the Bland-Altman analysis, the correlation PPMCC, ICC, and CCC tests [2,12,13,15,53]. To minimize the insufficiencies of individual statistical tests, this study examined the second-by-second heart rate readings using all of the mentioned correlation tests. Our results showed that when given the same dataset, PPMCC tests would typically derive higher values than ICC or CCC tests. Although all of the correlation coefficients have previously been adopted in other studies on optical devices, future research should exercise caution when selecting correlation tests and interpreting test results. That said, ICC and CCC should nonetheless be the preferred tests, as they were initially used to assess the interrater reliability in related validation studies [61,62]. Sartor et al [38] also supported the use of the CCC test for validating wrist-based heart rate monitors. Another study has proposed standardization of exercise protocols to ensure that the aggregate data were reproducible [51]. Thus, a standard set of examining methods and statistical analyses should be developed and adopted in future validation studies of optical heart rate sensors.

In conclusion, this study revealed that both the Garmin and Xiaomi optical heart rate sensors were capable of producing fairly accurate heart rate readings for both young and older adults. In particular, these devices achieved better accuracy during sedentary behaviors compared with physical activities. The heart rate reading accuracy of both devices was influenced by different types of physical activities. Consistently, the results echoed the previously reported tendency for heart rate underestimation during cycling and elliptical training in both of the devices. Notably, both devices exhibited the tendency to transiently display erroneous extreme readings. Thus, cautions should be exercised when using wrist-strapped fitness trackers to monitor the real-time heart rate during aerobic exercises.

This study was limited by several factors. First, the test devices were chosen because of their popularity in Asia and the availability of the broadcasting heart rate mode on these devices. However, different brands would usually be integrated with different PPG modules or algorithms, which could lead to discrepancies among the different optical heart rate devices [2,11]. This makes direct interpretations of findings on other optical heart rate devices using the current results more difficult. Although this study strived to retrieve the second-by-second data, the heart rate signals derived from various devices were complex, and the time lag problem existed between the investigational and reference devices [38]; in addition, owing to the trade secrets pertaining to the PPG signal-processing algorithms and the receiving apps, we could only assume that the second-by-second data are from the nearest previous beat-to-beat waveform signal to represent the heart rate readings. Nevertheless, the PPG sensor provided satisfactory readings when it was worn on the wrist than on other body parts. Second, the exercise intensity in this study was set at a submaximal level because of the various physical conditions of the participants. Thus, performance of these examining devices during more vigorous intensity exercises remains to be examined. In addition, this study only selected healthy participants, that is, participants without any cardiovascular diseases (eg, coronary artery disease or abnormal heart rhythms) or neurological disorders (eg, Parkinson disease or essential tremor) because the abnormal heart rate might interfere in the accuracy of comparison [63,64]. Hence, the results cannot be generalized to the overall older adult population. The validity of PPG fitness trackers for a population with major disorders, such as patients with cardiac disorders, requires further investigation.

Future research on these topics should benefit from the standardization of the exercise protocol, selected statistical methods, and the threshold of acceptable accuracy. This will allow for better cross-study comparisons and more accurate interpretations [51]. Second, future studies can incorporate more participants with various health conditions to increase the representativeness of the cohort. Conducting multiple trials for the same cohort will control variability. This will also help identify erroneous readings, especially when they fall within the physiological range. For similar reasons, the second-by-second data acquisition method presented in this study should be adopted in all future studies. This will also help address the mechanisms of those conceivably erroneous displays. Third, future testing should include more contextual activities, such as outdoor walking, running, and cycling, to better mimic real-life events. This will allow for better comparisons of device performances under different settings.

Overall, the results of this study indicate that both the Garmin and Xiaomi optical heart rate sensors exhibit acceptable heart rate–sensing accuracy for yellow skin tone population (Fitzpatrick skin type III or IV). Both devices perform similar to the Polar H7 chest-strapped heart rate monitor. The results also indicate that the sensing reliability of both the Garmin and Xiaomi devices can be influenced by different types of physical activities and that the Garmin device generally outperformed the Xiaomi device. The accuracy of both devices was not significantly affected by the age of users which implies that both devices are suitable for use in older adults. This has significant implications for the increasing aging population because PPG fitness trackers are inexpensive and use a noninvasive technology to provide information regarding various parameters and they have a great potential for telemedicine use considering remote or home health monitoring, assisting the older adult population to monitor their health [32].

The accuracy levels of both devices were negatively correlated with the level of activity intensity. For both devices, the measurement accuracy deteriorated in individuals while cycling. For unknown reasons, this study also reports the occurrence of extreme errors in these heart rate–sensing devices. These relevant findings imply that users or exercise practitioners should be cautious when using wrist-strapped fitness trackers to monitor exercise performance.