Health wearables are a key component of remote patient monitoring. These devices have the capability to continuously collect, process, and display health data automatically in real time, which may improve patients’ awareness of their health outside the clinical setting and enhance self-management [3].

Health wearables in the market are of 2 types: medical grade and consumer wearables. Medical wearables are designed to collect data relevant to specific health conditions, such as continuous glucose monitoring (CGM), to monitor blood glucose levels in people with diabetes. However, consumer wearables are often designed to collect general wellness data, such as fitness trackers that are used by patients to be aware of their physical activity level, sleep quality, mood, and heart rate. Some types of wearables connect to an associated mobile app to display real-time processed data, as well as to a patient portal where both patients and health care providers access passive data. Depending on the wearable platform, patients may also be required to manually enter other data types into the mobile app, in addition to the automatic data captured by the sensors in the wearable device.

Data produced by wearables are usually called patient-generated health data (PGHD), as the data collection relies on patients’ (rather than clinicians’) control and occurs outside the clinical environment [4]—a highly complex process. Large volumes of data collected from disparate wearables on an ongoing basis and under various situations bring possibilities for PGHD to be impacted by various technical, behavioral, operational, and organizational issues at the time of storage, transmission, analysis, or access for shared decision making that can lead to poor quality data. For example, wearable dysfunctionality [5-8], inaccurate calibration [6], incorrect manual inputs [5-9], complex data visualization [6,10], or not collecting data for a period of time [9-12] result in lack of trust in PGHD for clinical decision making. During the PGHD transmission stage, patients may share their data through a home network, their mobile wireless network, or a public wireless network, all of which might affect data transmission [5]. In addition, some of the platforms used during PGHD flow operate outside health information systems and may not be trusted because of concerns about their security and data privacy [6,9]. It is still challenging to define and implement solutions for integrating existing data in electronic medical records (EMRs) with PGHD from patients’ own wearables and develop widely accepted interoperability standards for PGHD exchange [6-11]. Moreover, concerns exist regarding consumer wearables, as they are not yet regulated, and there is uncertainty about their promise in achieving the quality required in optimal PGHD collection [12,13]. Regarding human factors, an individual’s intention to use wearables for general wellness tracking or for monitoring a specific health condition can strongly affect the quality of PGHD that these devices collect [8,13-15].

Health care providers may face large volumes of inaccurate, incomplete, irrelevant, and nonunderstandable PGHD, without solid principles for dealing with these data. The result is concerns about reliability and no confidence in using PGHD in routine clinical practice [16,17].

These issues of PGHD quality make it difficult to understand PGHD’s usefulness for health care, and these hinder adoption of these data as a systematic part of clinical practices [18]. Owing to this, remote patient monitoring is often initiated on a small scale for a short time and focuses on a single disease. A 2018 survey of over 20,000 health care consumers in 28 countries revealed nonsignificant adoption of PGHD [19].

There is little evidence in the research and industry literature that the quality of PGHD from wearables is sufficiently well managed to enable them to be trusted as health data or that such data can be used safely and effectively in clinical care. Furthermore, health care organizations lack awareness about PGHD quality from wearable devices [13,20].

Assuring PGHD quality requires efforts not only from health care providers but also from patients, their caregivers, and health consumers in general. Consumers, for example, have different needs from health care providers; therefore, their attitudes toward PGHD will differ, as well as the impact on data quality [21]. However, PGHD stakeholders are not limited to consumers and providers; other stakeholders, such as wearable manufacturers, contribute to PGHD transmission from outside the clinical setting, and their activities may similarly impact PGHD quality [16].

Therefore, it is critical to enhance general awareness of the quality of PGHD from wearables.

The objective of this study was to identify current challenges stakeholders face with various attributes of PGHD quality, define potential solutions to overcome those challenges, and facilitate a process where they could workshop their data quality needs and expectations with other stakeholders.

We conducted a half-day workshop in May 2019 on the campus of an Australian University located in a major health care precinct. This was part of a larger project using a range of inputs to develop PGHD quality assurance guidance. The workshop received human research ethics approval from the University of Melbourne (Ethics ID: 18532521).

Participants were eligible to attend the workshop, who were at least 18 years old and had experience of PGHD use in clinical care, including people who identified as health care consumers, clinical care providers, wearables suppliers, health information specialists, PGHD integration service providers, data analysts, and health service managers. Expressions of interest were sought via an open call and a Web-based registration form distributed via internal and public news and media channels, as well as professional organizations for digital health in Australia. At the point of recruitment or before, as needed, gaps were filled with personal invitations and via snowball sampling, for example, via key contacts in specific organizations where the researchers had existing relationships.

In total, 18 participants took part in the workshop. The participants included 8 health consumers and consumer advocates, 5 clinicians, 3 health information professionals, and 2 wearable and data integration company representatives.

Data collection was done by facilitators capturing written notes of the proceedings as attendees engaged in participatory design activities in written and oral formats, using a range of whole-group and small-group interactive methods. The group facilitators’ summing up presentations on boards and sticky notes were photographed. Deductive analysis based on 7 PGHD quality aspects (accessibility, accuracy, completeness, consistency, interpretability, relevancy, and timeliness) and inductive thematic content analysis were used as the primary data analysis methods. The initial data analysis was conducted by the PhD researcher, which was then reviewed by 3 supervisors. Thereafter, all of the 4 researchers discussed the results together in several meetings until consensus was achieved.

Table 1 illustrates the current problems and potential solutions that mixed-stakeholder small groups raised for each data quality aspect.

The findings in Table 1 show the dynamic situations in which patients collect PGHD through wearables and associated components, including mobile apps and portals. Situations are also subject to various technical, environmental, operational, and behavioral factors. These factors largely involve the design and performance of the wearable platforms, the fragmentation in clinical setting infrastructure to utilize PGHD from disparate wearables, and the stakeholders’ literacy levels in their engagement with PGHD collection and use.

The participants offered solutions for each PGHD quality attribute, which can be categorized into 5 overall ideas, namely (1) concerning redesign and development of wearables in a way to allow meeting PGHD quality requirements; (2) creation of clinical and administrative policies to address data quality assurance, such as policies on PGHD ownership, standard measurements, and data management policies; (3) stakeholders’ engagement and education; (4) integration of PGHD with clinical workflows and electronic health record systems, which demands creation of terminologies, data exchange protocols, and data analytics; and (5) defining who in the workforce is responsible for PGHD quality assurance procedures.

During the second group discussion, participants from each of the 4 stakeholder groups—consumers, clinicians, health information professionals, and wearable manufacturers—stated their expectations of what each of the other PGHD stakeholder groups could do to assure PGHD quality. The groups’ expectations are explained in Tables 2-5. There were no expectations of health information professionals expressed by consumers and wearable manufacturers.

Current health data quality assurance approaches are now disrupted by PGHD challenges. Given the potential value of PGHD in health care decision support, their adoption requires an in-depth understanding of data quality.

The investigation of PGHD quality is in its infancy, and few studies have explored it [6,13,18]. Therefore, our workshop provided the participants with an opportunity to carefully investigate PGHD quality in future remote patient monitoring interventions. It was acknowledged by the participants that the process of thinking about PGHD quality and translating into solutions could enhance collaboration among PGHD stakeholders, within and outside clinical settings.

Mixed-stakeholder participant groups raised divergent problems of PGHD quality. Technical factors related to wearable design and functionality affect most of the aforementioned aspects of data quality at the PGHD collection stage. The participants recognized that a cohesive platform to integrate PGHD with current EMR systems has the potential to collect automatic and manual data entries from various wearables and associated apps, to ideally facilitate dynamic data transmission. However, more practically, operational and organizational problems are likely to occur during data transmission from the patient to the clinical setting, such as internet access, time zone, distinct data transfer protocols, and lack of data integration. Behavioral habits of consumers, as well as digital health literacy among key PGHD stakeholders, such as consumers and clinicians, are among the factors that participants identified, which can influence PGHD quality. The participant discussions further highlighted the need to create a patient-centered care model that includes defined technical standards, policies, rights, and responsibilities to ensure PGHD quality and make these data useful for clinical care.

The 4 groups of PGHD stakeholders in our workshop raised different expectations about each of the other stakeholders, whereas all participants reached consensus on the importance of promoting collaboration among stakeholders. Furthermore, participants emphasized the need for transparency on the redistribution of tasks and responsibilities given to consumers, as well as to the clinicians. Participants also stated that wearables should be designed in a way to reconcile the needs of both patients and clinicians. Another key discussion point focused on hearing the consumer’s voice in improving PGHD collection. Without this voice, it may result in the consumer’s lack of understanding on how to optimally contribute to PGHD collection and management processes, thus leading to poor quality data.

The scope of remote monitoring services is still limited to single programs for single diseases, and it has not yet broadened to multiple health conditions. Moreover, in current remote monitoring scenarios, PGHD flow mainly occurs among consumers, clinicians, and wearable manufacturers, without further actions entering EMR systems or without the involvement of other specialist clinical or nonclinical staff. Presently, health information professionals within clinical settings are not full participants in the PGHD process, primarily because of the lack of integrated infrastructure, which could optimally incorporate PGHD into the health care system. This particular gap points to the need to broaden the team of stakeholders involved in the process and better define tasks for ensuring PGHD quality across the flow.

Critically, there was consensus by the stakeholder groups that PGHD management should occur in a standardized way. Different types of PGHD could become part of clinical terminology standards, made possible through national efforts. For example, specific metadata could be standardized to codify patients as source data generators.

As stated, there is a paucity of research investigating PGHD quality for use in routine clinical practice [6,13,18]. This study included a wider range of PGHD stakeholders: consumers, clinicians, and wearable vendors, as well as health information professionals whose roles in PGHD management and quality assurance have not been investigated in previous research.

This is the first study of its kind, which brings various groups of PGHD stakeholders together to share their concerns and expectations with each other, with regard to ensuring PGHD quality. This process helped to reach a consensus among participants on clear responsibilities they could take to effectively collaborate for better patient care.

The workshop participants were not representing an exhaustive range of PGHD stakeholders to explore quality issues from other perspectives. Further stages of this project will involve more groups of stakeholders. The workshop was focused on PGHD quality assurance for primary use of data, which falls within clinical care. Therefore, quality issues around secondary use of PGHD, such as utilization of PGHD in biomedical research, shared care plans, and mobile health surveillance, were not discussed.

In addition, the full spectrum of consumer and medical wearables was not represented among the participants. We purposely chose CGM devices as a medical wearable example, and we chose fitness trackers as a consumer wearable example in this workshop. CGM wearables are complex devices that require both automatic and manual data entry, calibration by using conventional glucometers, and a reliable connection to mobile apps and patient portals. Patients may also need to collect lifestyle data from consumer wearables, beside the CGM data, to provide a more comprehensive picture of their health status to their clinicians.

This study used a participatory research method to identify (1) the problems of PGHD quality, (2) potential solutions to overcome the challenges, and (3) the PGHD stakeholders’ expectations toward PGHD quality assurance. Our approach to codevelop challenges, solutions, and expectations enabled us to engage various stakeholders to jointly share experience and concerns. Active and continuous collaboration among PGHD stakeholders, from wearable development to data production and use in patient care, is vital not only to ensure the quality of the data but also the quality of the consumer’s health care experience and clinical outcomes. The findings from this workshop will contribute to the development of practical recommendations toward PGHD quality assurance, which can be adopted by a wide range of PGHD stakeholders.