Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 22.01.20 in Vol 8, No 1 (2020): January

Preprints (earlier versions) of this paper are available at, first published Jul 01, 2019.

This paper is in the following e-collection/theme issue:

    Original Paper

    Quality Assurance of Health Wearables Data: Participatory Workshop on Barriers, Solutions, and Expectations

    Health and Biomedical Informatics Centre, The University of Melbourne, Melbourne, Victoria, Australia

    Corresponding Author:

    Robab Abdolkhani, MSc

    Health and Biomedical Informatics Centre

    The University of Melbourne

    Level 13

    305 Grattan St

    Melbourne, Victoria, 3000


    Phone: 61 390358703



    Background: The ubiquity of health wearables and the consequent production of patient-generated health data (PGHD) are rapidly escalating. However, the utilization of PGHD in routine clinical practices is still low because of data quality issues. There is no agreed approach to PGHD quality assurance; therefore, realizing the promise of PGHD requires in-depth discussion among diverse stakeholders to identify the data quality assurance challenges they face and understand their needs for PGHD quality assurance.

    Objective: This paper reports findings from a workshop aimed to explore stakeholders’ data quality challenges, identify their needs and expectations, and offer practical solutions.

    Methods: A qualitative multi-stakeholder workshop was conducted as a half-day event on the campus of an Australian University located in a major health care precinct, namely the Melbourne Parkville Precinct. The 18 participants had experience of PGHD use in clinical care, including people who identified as health care consumers, clinical care providers, wearables suppliers, and health information specialists. Data collection was done by facilitators capturing written notes of the proceedings as attendees engaged in participatory design activities in written and oral formats, using a range of whole-group and small-group interactive methods. The collected data were analyzed thematically, using deductive and inductive coding.

    Results: The participants’ discussions revealed a range of technical, behavioral, operational, and organizational challenges surrounding PGHD, from the time when data are collected by patients to the time data are used by health care providers for clinical decision making. PGHD stakeholders found consensus on training and engagement needs, continuous collaboration among stakeholders, and development of technical and policy standards to assure PGHD quality.

    Conclusions: Assuring PGHD quality is a complex process that requires the contribution of all PGHD stakeholders. The variety and depth of inputs in our workshop highlighted the importance of co-designing guidance for PGHD quality guidance.

    JMIR Mhealth Uhealth 2020;8(1):e15329




    The health care industry is rapidly moving toward patient-centered care, in which patients actively contribute to their health care [1]. An example of a patient-centered care model, remote patient monitoring keeps patients outside the clinical setting while monitoring their health status. In these interventions, patients actively engaged in their health care and played an increasingly important role in sharing responsibilities with health care providers [2].


    Health wearables are a key component of remote patient monitoring. These devices have the capability to continuously collect, process, and display health data automatically in real time, which may improve patients’ awareness of their health outside the clinical setting and enhance self-management [3].

    Health wearables in the market are of 2 types: medical grade and consumer wearables. Medical wearables are designed to collect data relevant to specific health conditions, such as continuous glucose monitoring (CGM), to monitor blood glucose levels in people with diabetes. However, consumer wearables are often designed to collect general wellness data, such as fitness trackers that are used by patients to be aware of their physical activity level, sleep quality, mood, and heart rate. Some types of wearables connect to an associated mobile app to display real-time processed data, as well as to a patient portal where both patients and health care providers access passive data. Depending on the wearable platform, patients may also be required to manually enter other data types into the mobile app, in addition to the automatic data captured by the sensors in the wearable device.

    Data produced by wearables are usually called patient-generated health data (PGHD), as the data collection relies on patients’ (rather than clinicians’) control and occurs outside the clinical environment [4]—a highly complex process. Large volumes of data collected from disparate wearables on an ongoing basis and under various situations bring possibilities for PGHD to be impacted by various technical, behavioral, operational, and organizational issues at the time of storage, transmission, analysis, or access for shared decision making that can lead to poor quality data. For example, wearable dysfunctionality [5-8], inaccurate calibration [6], incorrect manual inputs [5-9], complex data visualization [6,10], or not collecting data for a period of time [9-12] result in lack of trust in PGHD for clinical decision making. During the PGHD transmission stage, patients may share their data through a home network, their mobile wireless network, or a public wireless network, all of which might affect data transmission [5]. In addition, some of the platforms used during PGHD flow operate outside health information systems and may not be trusted because of concerns about their security and data privacy [6,9]. It is still challenging to define and implement solutions for integrating existing data in electronic medical records (EMRs) with PGHD from patients’ own wearables and develop widely accepted interoperability standards for PGHD exchange [6-11]. Moreover, concerns exist regarding consumer wearables, as they are not yet regulated, and there is uncertainty about their promise in achieving the quality required in optimal PGHD collection [12,13]. Regarding human factors, an individual’s intention to use wearables for general wellness tracking or for monitoring a specific health condition can strongly affect the quality of PGHD that these devices collect [8,13-15].

    Health care providers may face large volumes of inaccurate, incomplete, irrelevant, and nonunderstandable PGHD, without solid principles for dealing with these data. The result is concerns about reliability and no confidence in using PGHD in routine clinical practice [16,17].

    These issues of PGHD quality make it difficult to understand PGHD’s usefulness for health care, and these hinder adoption of these data as a systematic part of clinical practices [18]. Owing to this, remote patient monitoring is often initiated on a small scale for a short time and focuses on a single disease. A 2018 survey of over 20,000 health care consumers in 28 countries revealed nonsignificant adoption of PGHD [19].

    There is little evidence in the research and industry literature that the quality of PGHD from wearables is sufficiently well managed to enable them to be trusted as health data or that such data can be used safely and effectively in clinical care. Furthermore, health care organizations lack awareness about PGHD quality from wearable devices [13,20].


    Assuring PGHD quality requires efforts not only from health care providers but also from patients, their caregivers, and health consumers in general. Consumers, for example, have different needs from health care providers; therefore, their attitudes toward PGHD will differ, as well as the impact on data quality [21]. However, PGHD stakeholders are not limited to consumers and providers; other stakeholders, such as wearable manufacturers, contribute to PGHD transmission from outside the clinical setting, and their activities may similarly impact PGHD quality [16].

    Therefore, it is critical to enhance general awareness of the quality of PGHD from wearables.

    The objective of this study was to identify current challenges stakeholders face with various attributes of PGHD quality, define potential solutions to overcome those challenges, and facilitate a process where they could workshop their data quality needs and expectations with other stakeholders.


    Study Design

    We conducted a half-day workshop in May 2019 on the campus of an Australian University located in a major health care precinct. This was part of a larger project using a range of inputs to develop PGHD quality assurance guidance. The workshop received human research ethics approval from the University of Melbourne (Ethics ID: 18532521).

    Participant Recruitment

    Participants were eligible to attend the workshop, who were at least 18 years old and had experience of PGHD use in clinical care, including people who identified as health care consumers, clinical care providers, wearables suppliers, health information specialists, PGHD integration service providers, data analysts, and health service managers. Expressions of interest were sought via an open call and a Web-based registration form distributed via internal and public news and media channels, as well as professional organizations for digital health in Australia. At the point of recruitment or before, as needed, gaps were filled with personal invitations and via snowball sampling, for example, via key contacts in specific organizations where the researchers had existing relationships.


    In total, 18 participants took part in the workshop. The participants included 8 health consumers and consumer advocates, 5 clinicians, 3 health information professionals, and 2 wearable and data integration company representatives.

    Workshop Structure

    On-Site Registration

    Each participant was given a package of stationery materials, the consent form, the Plain Language Statement, and a copy of the presentation slides.

    Researcher’s Presentation

    Participants were informed about the workshop purpose and its structure. In addition, findings from previous parts of the project [6,14] were elucidated. The concept of PGHD quality and its 7 aspects—accessibility, accuracy, completeness, consistency, interpretability, relevancy, and timeliness—adapted from a comprehensive data quality framework developed for clinical data quality assurance by the Australian Capital Territory [22] were explained through scenarios (Multimedia Appendix 1) to enable the participants to identify the various circumstances that might impact each PGHD quality aspect and identify the data management stage in which the quality aspect might require consideration. We used CGM devices as a medical wearable example, and we used fitness trackers as a consumer wearable example in this workshop.

    Group Discussion 1

    The facilitators directed participants into 4 groups, each with a cross section of different stakeholders (Figure 1). Groups discussed problems related to PGHD quality. Participants contributed verbally, wrote ideas on sticky notes, and placed these on their group worksheet (Figure 2). A total of 1 facilitator managed each group to direct discussions and assist in timekeeping. Small group discussions ran for 30 min. Thereafter, in a 15-min session, each facilitator took turns presenting to the whole room the consolidated deliberations of the facilitator’s small group. The facilitators wrote the key findings on whiteboards to be available during the break.

    Figure 1. Participants’ discussions in 4 groups of patient-generated health data stakeholders.
    View this figure
    Figure 2. Worksheets of patient-generated health data quality problems and ideas.
    View this figure

    Participants walked around the workshop room to view the discussion findings on whiteboards and exchange further ideas with participants, in preparation for the next part.

    Group Discussion 2

    The facilitators directed participants into 4 differently configured groups. Each group took on a role of a key stakeholder—health care consumer, health care provider, health information professional, or wearables manufacturer. Groups discussed and translated their ideas into a set of their expectations of other stakeholder groups, on worksheets (Figure 3). This task took 30 min, followed by 15 min in which each facilitator set out and spoke to the respective group’s expectations on whiteboards.

    Figure 3. Worksheets of patient-generated health data stakeholder’s expectations.
    View this figure

    The researchers pointed out the outcomes and future work implications. Participants were informed about the next steps of the research project and how they may stay engaged if they are interested.

    Data Collection and Analysis

    Data collection was done by facilitators capturing written notes of the proceedings as attendees engaged in participatory design activities in written and oral formats, using a range of whole-group and small-group interactive methods. The group facilitators’ summing up presentations on boards and sticky notes were photographed. Deductive analysis based on 7 PGHD quality aspects (accessibility, accuracy, completeness, consistency, interpretability, relevancy, and timeliness) and inductive thematic content analysis were used as the primary data analysis methods. The initial data analysis was conducted by the PhD researcher, which was then reviewed by 3 supervisors. Thereafter, all of the 4 researchers discussed the results together in several meetings until consensus was achieved.


    The workshop findings are organized thematically as problems and solutions related to each health data quality dimension, as well as stakeholders’ expectations of each other, to assure PGHD quality.

    Data Quality Challenges and Solutions

    Table 1 illustrates the current problems and potential solutions that mixed-stakeholder small groups raised for each data quality aspect.

    Table 1. Patient-generated health data quality problems and potential solutions.
    View this table

    The findings in Table 1 show the dynamic situations in which patients collect PGHD through wearables and associated components, including mobile apps and portals. Situations are also subject to various technical, environmental, operational, and behavioral factors. These factors largely involve the design and performance of the wearable platforms, the fragmentation in clinical setting infrastructure to utilize PGHD from disparate wearables, and the stakeholders’ literacy levels in their engagement with PGHD collection and use.

    The participants offered solutions for each PGHD quality attribute, which can be categorized into 5 overall ideas, namely (1) concerning redesign and development of wearables in a way to allow meeting PGHD quality requirements; (2) creation of clinical and administrative policies to address data quality assurance, such as policies on PGHD ownership, standard measurements, and data management policies; (3) stakeholders’ engagement and education; (4) integration of PGHD with clinical workflows and electronic health record systems, which demands creation of terminologies, data exchange protocols, and data analytics; and (5) defining who in the workforce is responsible for PGHD quality assurance procedures.

    Patient-Generated Health Data Stakeholders’ Expectations

    During the second group discussion, participants from each of the 4 stakeholder groups—consumers, clinicians, health information professionals, and wearable manufacturers—stated their expectations of what each of the other PGHD stakeholder groups could do to assure PGHD quality. The groups’ expectations are explained in Tables 2-5. There were no expectations of health information professionals expressed by consumers and wearable manufacturers.

    Table 2. Consumers’ expectations of other patient-generated health data stakeholder groups.
    View this table
    Table 3. Clinicians’ expectations of other patient-generated health data stakeholder groups.
    View this table
    Table 4. Health information professionals’ expectations of other patient-generated health data stakeholder groups.
    View this table
    Table 5. Health wearables’ manufacturers’ expectations of other patient-generated health data stakeholder groups.
    View this table


    Principal Findings

    Current health data quality assurance approaches are now disrupted by PGHD challenges. Given the potential value of PGHD in health care decision support, their adoption requires an in-depth understanding of data quality.

    The investigation of PGHD quality is in its infancy, and few studies have explored it [6,13,18]. Therefore, our workshop provided the participants with an opportunity to carefully investigate PGHD quality in future remote patient monitoring interventions. It was acknowledged by the participants that the process of thinking about PGHD quality and translating into solutions could enhance collaboration among PGHD stakeholders, within and outside clinical settings.

    Diversity of Patient-Generated Health Data Quality Challenges and Potential Solutions

    Mixed-stakeholder participant groups raised divergent problems of PGHD quality. Technical factors related to wearable design and functionality affect most of the aforementioned aspects of data quality at the PGHD collection stage. The participants recognized that a cohesive platform to integrate PGHD with current EMR systems has the potential to collect automatic and manual data entries from various wearables and associated apps, to ideally facilitate dynamic data transmission. However, more practically, operational and organizational problems are likely to occur during data transmission from the patient to the clinical setting, such as internet access, time zone, distinct data transfer protocols, and lack of data integration. Behavioral habits of consumers, as well as digital health literacy among key PGHD stakeholders, such as consumers and clinicians, are among the factors that participants identified, which can influence PGHD quality. The participant discussions further highlighted the need to create a patient-centered care model that includes defined technical standards, policies, rights, and responsibilities to ensure PGHD quality and make these data useful for clinical care.

    Convergence in Patient-Generated Health Data Stakeholders’ Expectations

    The 4 groups of PGHD stakeholders in our workshop raised different expectations about each of the other stakeholders, whereas all participants reached consensus on the importance of promoting collaboration among stakeholders. Furthermore, participants emphasized the need for transparency on the redistribution of tasks and responsibilities given to consumers, as well as to the clinicians. Participants also stated that wearables should be designed in a way to reconcile the needs of both patients and clinicians. Another key discussion point focused on hearing the consumer’s voice in improving PGHD collection. Without this voice, it may result in the consumer’s lack of understanding on how to optimally contribute to PGHD collection and management processes, thus leading to poor quality data.

    The scope of remote monitoring services is still limited to single programs for single diseases, and it has not yet broadened to multiple health conditions. Moreover, in current remote monitoring scenarios, PGHD flow mainly occurs among consumers, clinicians, and wearable manufacturers, without further actions entering EMR systems or without the involvement of other specialist clinical or nonclinical staff. Presently, health information professionals within clinical settings are not full participants in the PGHD process, primarily because of the lack of integrated infrastructure, which could optimally incorporate PGHD into the health care system. This particular gap points to the need to broaden the team of stakeholders involved in the process and better define tasks for ensuring PGHD quality across the flow.

    Critically, there was consensus by the stakeholder groups that PGHD management should occur in a standardized way. Different types of PGHD could become part of clinical terminology standards, made possible through national efforts. For example, specific metadata could be standardized to codify patients as source data generators.

    Relation to Other Works

    As stated, there is a paucity of research investigating PGHD quality for use in routine clinical practice [6,13,18]. This study included a wider range of PGHD stakeholders: consumers, clinicians, and wearable vendors, as well as health information professionals whose roles in PGHD management and quality assurance have not been investigated in previous research.

    This is the first study of its kind, which brings various groups of PGHD stakeholders together to share their concerns and expectations with each other, with regard to ensuring PGHD quality. This process helped to reach a consensus among participants on clear responsibilities they could take to effectively collaborate for better patient care.

    Limitations and Future Work

    The workshop participants were not representing an exhaustive range of PGHD stakeholders to explore quality issues from other perspectives. Further stages of this project will involve more groups of stakeholders. The workshop was focused on PGHD quality assurance for primary use of data, which falls within clinical care. Therefore, quality issues around secondary use of PGHD, such as utilization of PGHD in biomedical research, shared care plans, and mobile health surveillance, were not discussed.

    In addition, the full spectrum of consumer and medical wearables was not represented among the participants. We purposely chose CGM devices as a medical wearable example, and we chose fitness trackers as a consumer wearable example in this workshop. CGM wearables are complex devices that require both automatic and manual data entry, calibration by using conventional glucometers, and a reliable connection to mobile apps and patient portals. Patients may also need to collect lifestyle data from consumer wearables, beside the CGM data, to provide a more comprehensive picture of their health status to their clinicians.


    This study used a participatory research method to identify (1) the problems of PGHD quality, (2) potential solutions to overcome the challenges, and (3) the PGHD stakeholders’ expectations toward PGHD quality assurance. Our approach to codevelop challenges, solutions, and expectations enabled us to engage various stakeholders to jointly share experience and concerns. Active and continuous collaboration among PGHD stakeholders, from wearable development to data production and use in patient care, is vital not only to ensure the quality of the data but also the quality of the consumer’s health care experience and clinical outcomes. The findings from this workshop will contribute to the development of practical recommendations toward PGHD quality assurance, which can be adopted by a wide range of PGHD stakeholders.


    The authors wish to thank all the participants for their valuable time and contribution. The authors wish to especially thank Ms Cecily Gilbert, a research assistant, and Mr Gerardo Luis Dimaguila, a PhD student, at Health and Biomedical Informatics Centre at the University of Melbourne for their help in facilitating the workshop.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Workshop presentation slides.

    PDF File (Adobe PDF File), 22245 KB


    1. Santana M, Ahmed S, Lorenzetti D, Jolley RJ, Manalili K, Zelinsky S, et al. Measuring patient-centred system performance: a scoping review of patient-centred care quality indicators. BMJ Open 2019 Jan 7;9(1):e023596 [FREE Full text] [CrossRef] [Medline]
    2. World Health Organisation. Chronic disease management and remote patient monitoring. Eurohealth 2009;15(1):1-48 [FREE Full text]
    3. Vegesna A, Tran M, Angelaccio M, Arcona S. Remote patient monitoring via non-invasive digital technologies: a systematic review. Telemed J E Health 2017 Jan;23(1):3-17 [FREE Full text] [CrossRef] [Medline]
    4. Shapiro M, Johnston D, Wald J, Mon D. HealthIT. 2012. Patient-generated Health Data: White Paper Prepared for the Office of the National Coordinator for Health IT by RTI International 2012.   URL: [accessed 2019-11-19]
    5. Sriram J, Shin M, Kotz D, Rajan A, Sastry M, Yarvis M. Challenges in data quality assurance in pervasive health monitoring systems. In: Future of Trust in Computing. Wiesbaden: Vieweg+Teubner; 2009:129-142.
    6. Abdolkhani R, Gray K, Borda A, DeSouza R. Patient-generated health data management and quality challenges in remote patient monitoring. JAMIA Open 2019 [FREE Full text] [CrossRef]
    7. Chung C, Dew K, Cole A, Zia J, Fogarty J, Kientz JA, et al. Boundary negotiating artifacts in personal informatics: patient-provider collaboration with patient-generated data. CSCW Conf Comput Support Coop Work 2016 Dec 27;2016:770-786 [FREE Full text] [CrossRef] [Medline]
    8. West P, Giordano R, Kleek M, Shadbolt N. The Quantified Patient in the Doctor's Office: Challenges & Opportunities. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2016 Presented at: CHI'16; May 7 - 12, 2016; San Jose, California, USA p. 3066-3078. [CrossRef]
    9. Zhu H, Colgan J, Reddy M, Choe E. Sharing patient-generated data in clinical practices: an interview study. AMIA Annu Symp Proc 2016;2016:1303-1312 [FREE Full text] [Medline]
    10. Cohen DJ, Keller SR, Hayes GR, Dorr DA, Ash JS, Sittig DF. Integrating patient-generated health data into clinical care settings or clinical decision-making: lessons learned from project HealthDesign. JMIR Hum Factors 2016 Oct 19;3(2):e26 [FREE Full text] [CrossRef] [Medline]
    11. Puentes J, Montagner J, Lecornu L, Lähteenmäki J. Quality analysis of sensors data for personal health records on mobile devices. In: Bali R, Troshani I, Goldberg S, Wickramasinghe N, editors. Pervasive Health Knowledge Management. New York: Springer; 2013:103-133.
    12. Hernández N, Castro L, Favela J, Michán L, Arnrich B. Data quality in mobile sensing datasets for pervasive healthcare. In: Khan S, Zomaya A, Abbas A, editors. Handbook of Large-Scale Distributed Computing in Smart Healthcare. Cham: Springer; 2017:217-238.
    13. West P, van Kleek M, Giordano R, Weal M, Shadbolt N. Information quality challenges of patient-generated data in clinical practice. Front Public Health 2017;5:284 [FREE Full text] [CrossRef] [Medline]
    14. Abdolkhani R, Borda A, Gray K. Quality management of patient generated health data in remote patient monitoring using medical wearables-a systematic review. In: Cummings E, Ryan A, Schaper L, editors. Connecting the System to Enhance the Practitioner and Consumer Experience in Healthcare. Amsterdam, Netherlands: Ios Press; 2018.
    15. Codella J, Partovian C, Chang H, Chen C. Data quality challenges for person-generated health and wellness data. IBM J Res Dev 2018;62(1):3:1-3:8 [FREE Full text] [CrossRef]
    16. HealthIT. 2018 Jan. Conceptualizing a Data Infrastructure for the Capture, Use, and Sharing of Patient-generated Health Data in Care Delivery and Research Through 2024.   URL: [accessed 2019-11-19]
    17. HIMSS Media. Healthcare Coaching: Multiplying the Value of Wearables and Patient-generated Health Data 2018.   URL: https:/​/2nwchq3a3ags2kj7bq20e3qv-wpengine.​​wp-content/​uploads/​FINAL_HIMSS_FitBit_WP_10.​01.​20181.​pdf [accessed 2019-11-19]
    18. West P, van Kleek M, Giordano R, Weal M, Shadbolt N. Common Barriers to the Use of Patient-Generated Data Across Clinical Settings. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 2018 Presented at: CHI'18; April 21 - 26, 2018; Montreal QC, Canada. [CrossRef]
    19. Ipsos Healthcare. 2018 Oct 31. Connected Health Trends 2018.   URL: [accessed 2019-11-19]
    20. Piwek L, Ellis DA, Andrews S, Joinson A. The rise of consumer health wearables: promises and barriers. PLoS Med 2016 Feb;13(2):e1001953 [FREE Full text] [CrossRef] [Medline]
    21. Andersen T, Bansler J, Kensing F, Moll J, Nielsen K. Alignment of Concerns: A Design Rationale for Patient Participation in eHealth. In: 2014 47th Hawaii International Conference on System Sciences. 2014 Presented at: HICSS'14; January 6-9, 2014; Waikoloa, HI, USA. [CrossRef]
    22. ACT Health. Data Quality Frameworknterprise Business Intelligence Solutions.   URL:


    CGM: continuous glucose monitoring
    EMR: electronic medical record
    PGHD: patient-generated health data

    Edited by G Eysenbach; submitted 01.07.19; peer-reviewed by P West, J Salisbury; comments to author 15.08.19; revised version received 04.10.19; accepted 23.10.19; published 22.01.20

    ©Robab Abdolkhani, Kathleen Gray, Ann Borda, Ruth DeSouza. Originally published in JMIR mHealth and uHealth (, 22.01.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.