This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Background
Despite the increasing use of remote measurement technologies (RMT) such as wearables or biosensors in health care programs, challenges associated with selecting and implementing these technologies persist. Many health care programs that use RMT rely on commercially available, “off-the-shelf” devices to collect patient data. However, validation of these devices is sparse, the technology landscape is constantly changing, relative benefits between device options are often unclear, and research on patient and health care provider preferences is often lacking.
Objective
To address these common challenges, we propose a novel device selection framework extrapolated from human-centered design principles, which are commonly used in de novo digital health product design. We then present a case study in which we used the framework to identify, test, select, and implement off-the-shelf devices for the Remote Assessment of Disease and Relapse-Central Nervous System (RADAR-CNS) consortium, a research program using RMT to study central nervous system disease progression.
Methods
The RADAR-CNS device selection framework describes a human-centered approach to device selection for mobile health programs. The framework guides study designers through stakeholder engagement, technology landscaping, rapid proof of concept testing, and creative problem solving to develop device selection criteria and a robust implementation strategy. It also describes a method for considering compromises when tensions between stakeholder needs occur.
Results
The framework successfully guided device selection for the RADAR-CNS study on relapse in multiple sclerosis. In the initial stage, we engaged a multidisciplinary team of patients, health care professionals, researchers, and technologists to identify our primary device-related goals. We desired regular home-based measurements of gait, balance, fatigue, heart rate, and sleep over the course of the study. However, devices and measurement methods had to be user friendly, secure, and able to produce high quality data. In the second stage, we iteratively refined our strategy and selected devices based on technological and regulatory constraints, user feedback, and research goals. At several points, we used this method to devise compromises that addressed conflicting stakeholder needs. We then implemented a feedback mechanism into the study to gather lessons about devices to improve future versions of the RADAR-CNS program.
Conclusions
The RADAR device selection framework provides a structured yet flexible approach to device selection for health care programs and can be used to systematically approach complex decisions that require teams to consider patient experiences alongside scientific priorities and logistical, technical, or regulatory constraints.
When used as part of health care programs, remote measurement technologies (RMT) such as wearables or biosensors have the potential to affect clinical decision making, provide novel health insights, and improve the standard of care in a variety of disease areas [1-4]. RMT is a subset of mobile health (mHealth) technologies, which includes “any technology that enables monitoring of a person’s health status through a remote interface, which can then be transmitted to a healthcare provider” for review or as a means of education for the user themselves [5]. Though use of RMT in health care programs has grown in recent years [1,2,6,7], its impact on health outcomes does not always live up to its supposed potential [1,7,8].
Successful utilization of RMT depends on careful consideration of the program’s scientific, technical, and usability requirements. However, many programs employ commercially available, “off-the-shelf” devices that cannot be customized according to these requirements. In such cases, program designers are challenged to select devices from hundreds of options [9] in a marketplace where validation is sparse [1,7,8], product turnover is high [10], and relative benefits between device options are often unclear. Comparative studies show either limited accuracy or low to moderate agreement between similar, widely-used devices for common measurements such as activity levels [11-14], sleep [14-16], heart rate [12,17,18], and energy expenditure [14,16,19]. Few industry-wide data standards have been established [6,9,20], and different devices may define and report measurements in ways that are not directly comparable [13]. Additionally, the experiences of potential users—including patients, caregivers, and health care professionals—affect the use of RMT heavily [21-23], but these insights are often not collected or transformed into technology requirements [24]. Unfortunately, RMT that do not cater to user needs can increase patient, caregiver, and health care provider burden in otherwise promising health care programs [6,25] and may negatively impact enrollment and retention [26].
Those designing health care programs often struggle to navigate device selection due to the technology landscape’s complexity and potential tensions between device selection criteria [4,20,27]. To date, few best practices exist to guide the selection of off-the-shelf devices. The Framework of Specifications to Consider During Mobile Technology Selection developed by the Clinical Trial Transformation Initiative lists factors to consider when selecting RMT, including technical performance, data management, safety, and human factors [28]. However, it does not provide a method to apply or prioritize these factors. The Digital Health Selection Framework by the Institute for Healthcare Improvement [29] describes a computational method for assessing the technology landscape based on high-level selection criteria. However, this framework aims to support the development of health care policy, and the method does not support the identification and ranking of sufficiently detailed requirements for use in individual program designs. Scientific publications provide only high-level commentary on device selection, suggesting that designers consider technical requirements, user experiences, data quality, safety, privacy, regulations, costs, and other factors when choosing technologies [27-30]. Such publications also discuss the need to set detailed objectives [27,31] and gather requirements from a diverse set of stakeholders [24,28,31]. However, to our knowledge, no publication describes systematic methods for gathering, prioritizing, and weighing device selection criteria within the context of the program’s users, environments, and goals.
This is problematic, as device-related factors have the potential to limit the success, reproducibility, or scalability of otherwise promising health care programs. In this study we propose a framework to guide device selection based on human-centered design (HCD) principles. We then demonstrate the use of this framework in a research program using RMT to identify and predict relapses in multiple sclerosis (MS).
MethodsHuman-Centered Design in Mobile Health
HCD is increasingly used to design novel health care programs and products [4,10,32-37]. HCD is a series of methods through which designers study a product user’s needs and environment and then design accordingly [38,39]. Designers engage or “empathize” with potential users then generate ideas, develop prototypes, and test those prototypes with the people for whom they are designing [38,39]. Designers alternate between divergent and convergent thinking, looking broadly to understand context and possible solutions, and then converging onto a final problem statement, approach, or solution [38,40]. Many methods also employ agile or lean principles, which use rapid prototyping, feedback loops, and learning cycles to drive an iterative design and implementation process [38,41]. These methods allow designers to develop a deep understanding of the contextual factors that affect design, making them well-suited to support product design in complex, ambiguous, and rapidly-changing environments. The merits of HCD in health care program design have been discussed at length elsewhere [24,33], though such methods are largely applied to de novo designs, rather than technology selection.
HCD frameworks exist for a variety of mHealth applications, including behavioral intervention design [32], implementation of patient-facing technology in interventional clinical trials [31], mHealth solution development and validation [10,33,42], stakeholder engagement [36], and requirement development [43]. Though these frameworks are inconsistent in their language, they employ a set of common methods to inform the design of digital solutions within the context of the health care system (Textbox 1).
Common human-centered design principles recommended in mobile health solution design.
• Assemble a multidisciplinary team [31,43]
• Iterate throughout the design process [10,31-34,36,42,43]
• Begin by conducting stakeholder engagement activities to understand users’ needs and environments [31-34,36,42,43]
• Conduct ideation sessions in which a variety of approaches and potential solutions are explored [10,31,32,34,42]
• Enable a variety of stakeholders, including patients, health care professionals, technical experts, and others to participate in the design process [31-34,36,42]
• Prioritize identified requirements and resolve conflicting requirements through further engagement with team members and stakeholders [43]
• Prototype and test with end users prior to scaled implementation [10,31-34,42,43]
• Consider the implementation strategy early and refine it during the design process [31-33]
• Measure the solution’s impact and efficacy [10,31,43]
• Share both positive and negative lessons learned with relevant stakeholders to improve current and future designs [31,32]
To our knowledge, no HCD framework addresses the challenges associated with selecting off-the-shelf devices for digital health care interventions. We hypothesized that HCD methods may also be useful for that purpose, because HCD methods address similar design challenges to those posed by device selection. Such challenges include understanding and navigating complicated contextual factors [31,32,34,42,43], engaging with multifunctional stakeholders [36], and prioritizing requirements while addressing diverse stakeholder needs [43].
RADAR Device Selection Framework
A novel device selection method was developed for the Remote Assessment of Disease and Relapse-Central Nervous System (RADAR-CNS) project, a collaborative research program using RMT to study central nervous system disease progression. This framework was developed empirically based on the authors’ previous experience with HCD in medical technology design. We hypothesized that HCD methods could help design teams manage the complexity inherent to device selection. Therefore, the three-stage RADAR-CNS device selection framework (Figure 1) was proposed and optimized for the RADAR-CNS program. The framework uses HCD techniques to explore the technology landscape, refine device requirements, develop an implementation strategy, and make informed decisions in parallel with program design and implementation.
RADAR device selection framework.
Stage 1: Prepare
In this stage, the team studies contextual and user-related factors that may affect device use and implementation. The goals of the program, motivations and experiences of patients, involvement of caregivers, and symptoms or sensitivities related to the target disease area will define how user-friendly, discreet, configurable, or multifunctional a device must be. These activities are analogous to the empathize, define, and ideate steps of the design thinking process [44], and similar steps have been proposed in other frameworks [32,33]. In stage 1 we highlight relevant device-related insights that can be collected through HCD methods early in the program planning process.
Engage With Patients and Other Stakeholders
Simblett et al (2018) [22] described five categories of facilitators and barriers that influence patient engagement with RMT: health status, usability, convenience and accessibility, perceived utility, and motivation. During the preparation stage, the device selection team engages with patients and other stakeholders to explore these factors, identify user needs, and draft technology requirements. These activities can be conducted alongside other engagement activities designed to inform program goals or design. Methods for engaging with these and other relevant stakeholders have been proposed, including co-design sessions, focus groups, interviews, workshops, and surveys [44-48]. Integrated patient advisory boards can also guide discussions and decisions throughout the device selection process.
Though published literature on research priorities and usability requirements may provide general insights into patient perspectives in a variety of disease areas [22,24,49], primary research with the program’s target population is critical [4,24]. RMT can increase the burden associated with giving and receiving care [4,9,25], which must be minimized to enable sustained program adoption. Direct engagement with potential users provides the nuanced insights that are necessary to minimize burden and increase the chances of program success. Patients may be the primary users of the technology; however, caregivers, health care professionals, and others should also be engaged, as they affect patients’ willingness and motivation to engage with RMT [22].
Explore Potential Approaches
The team then explores different approaches for measuring health status. Options should reflect scientific and clinical goals as well as patients’ priorities. The team should propose potential measurement schemes that list relevant variables or outcomes, surrogate measurements, data streams, required sensors, and desired frequency of measurements. In this stage, it is helpful to use good brainstorming techniques such as those described in IDEO’s Design Thinking Bootleg [44] to generate a variety of options and encourage creativity by limiting discussion of potential constraints. The team should define potential program goals, endpoints, and measurement schemes before exploring technology options and implementation strategies [20,27,31]. Delaying discussion of specific technology options forces the team to frame device selection around program and user needs, thereby preventing the design of a program around a familiar but ill-suited technology.
Define Measurement and Technology Goals
Based on the outcomes of the engagement and brainstorming activities, the team should converge on one or more promising measurement schemes and clearly define goals for the RMT. Only once these are defined should the team draft selection criteria. The team should clearly state what compromises they are and are not willing to make, as these choices will drive final device selection. Examples of relevant device selection criteria have been published elsewhere [27-29].
Milestone 1: Propose a Monitoring Plan
By the end of this stage, the team should have developed a robust understanding of stakeholder needs and priorities, a well-defined program goal, one or more potential measurement schemes, and a preliminary understanding of the technology landscape and technology selection criteria. The activities that led to this preliminary plan will provide necessary context to support device selection decisions, especially when no device meets all criteria and concessions must be made. To achieve this level of clarity, the team may need to conduct multiple iterations of the “Prepare” stage. For example, the team may need to re-engage stakeholders to confirm the acceptability of a measurement scheme and then adjust the scheme in subsequent brainstorming activities.
Stage 2: Select
In this stage, the team progresses iteratively through a series of activities to identify a suitable device and refine an implementation strategy. With each iteration, the team should identify and answer outstanding questions, refine their thinking, and add detail to their proposed implementation plan. The team should first think broadly before refining the measurement scheme and implementation plan to reflect the program’s constraints. This approach allows the team to explore multiple approaches efficiently and to pursue creative options for getting as close to an ideal solution as possible.
Explore Technology Landscape
First, the team performs an initial technology landscape assessment and compiles a list of potentially suitable technologies. Devices should then be systematically excluded from this list based on user feedback and updates to the selection criteria or measurement scheme. When appropriate, additional options should be added to reflect updates to the selection criteria and implementation strategy. A short list of candidates should be defined based on the team’s selection criteria.
Refine Selection Criteria and Implementation Strategy
Based on identified technology options and insights from user engagement, the team should begin to define how the technology will be implemented. Factors such as the necessary connections to information technology (IT) systems, device provisioning, training, frequency of device use, compliance monitoring, and data syncing methods should be considered. This strategy may change over time; however, considering these factors early in the selection process will help the team understand potential infrastructure or logistical constraints that could impact device selection. Lack of such strategic planning has been shown to hinder successful implementation of RMT [30].
Off-the-shelf devices may not fit the initial measurement scheme and selection criteria perfectly. Iterative refinement of the selection criteria, measurement scheme, implementation strategy, and technology landscape will help the team explore creative alternatives, make minor concessions, and identify a small group of candidate technologies that meet most criteria.
Conduct Proof of Concept
Throughout this process, additional questions about candidate devices’ characteristics and relative advantages are likely to emerge. In the proof of concept (PoC) phase, the team should conduct targeted tests to answer these questions. PoCs are targeted device assessments that can be conducted quickly prior to implementation in a clinical study that enable rapid learning and decision making during the technology selection process [4,31]. PoCs can test technical characteristics (eg, bench testing for data quality, connectivity, durability), assess user experience in the target population (eg, usability studies), compare candidate devices, or test aspects of a technology’s implementation strategy (eg, “dry runs” to test training protocols and technology support systems) [31]. The results of any PoC should be actionable, either in a technology selection decision or to influence refinement of the implementation strategy.
Milestone 2: Select Technology and Implementation Strategy
By the end of this stage, the team should have narrowed the landscape to a few well-defined technology options, though each is likely to require compromise. To weigh these options, the team should use a systematic method to compare candidate devices and their required compromises. The team should facilitate multifunctional conversations to develop understanding of the required compromises and consensus on a final decision. The team should also finalize an implementation strategy, validating it through PoC testing and additional user feedback as necessary.
Stage 3: LearnMonitor Outcomes and Collect Feedback
The team should also devise mechanisms to collect feedback, experiential data, opportunities for improvement, and opportunities for learning from active programs, and these mechanisms should be included in research protocols if appropriate. Validated questionnaires such as the Post-Study System Usability Questionnaire [50] or the Technology Assessment Model [51] are widely used, and additional quantitative metrics such as device use or help desk engagement rates may also provide insights. Qualitative interviews with patients and health care professionals can identify specific opportunities to improve the implementation strategies, training materials and methods, technologies, or technology support systems.
Share and Implement Lessons Learned
The design and learning processes should not stop when the program is launched [30]. Quantitative, qualitative, and experiential data collected during all three stages of the framework should be used to continually refine the implementation strategy to ensure efficacy, efficiency, user engagement, ease of use, and clinical utility. In the case of a clinical study where continuous adjustments to the implementation strategy may jeopardize a program’s scientific goals, feasibility studies or clinical process evaluations may be used to test and refine the implementation strategy [4,20,52]. Sometimes, devices or technologies selected for an investigational system may not be practical for use in a scaled clinical practice. In this case, appropriate technologies should be selected or designed to fit the system requirements that were collected during investigational implementation. Both positive and negative findings should be shared to inform technology selection decisions in future programs.
ResultsRADAR-CNS Case Study
RADAR-CNS is a public-private research program leveraging RMT to develop new ways of assessing disease progression in depression, epilepsy, and MS [53]. The RADAR Device Selection framework was used to select devices for several RADAR-CNS studies; however, only its use in a study on MS disease progression is explored here. In this 2-year study, wearable devices and a custom application collect longitudinal health-related data from people with relapsing-remitting MS. The aim is to develop algorithms that can predict relapse and improve patient care. Details of the study’s full protocol are outside the scope of this publication, and only device selection procedures are described here.
RADAR-CNS: Prepare
A cross-functional team of clinicians, researchers, and technical experts was established, and RADAR-CNS’ patient advisory board [54] was also regularly consulted. We worked with people living with MS to understand their perspectives on research priorities, usability requirements, desired device features, and factors influencing sustained engagement with RMT. We conducted a systematic literature review to identify relevant discussion topics [22] and initiated a series of surveys and semistructured focus groups for people living with MS to identify factors affecting engagement with RMT [55]. Participants provided feedback on preferred device features and engagement schemes as well as perspectives on value and privacy. Much of this work has been published previously [55-57]. Participants emphasized the importance in accommodating MS symptoms, making the system easily usable, and enabling users to exert control within the RMT system [55].
We then explored areas of scientific research priority, including cognition, mood, physical activity, sleep, social interactions, speech, and stress. We identified variables that aligned with patient and scientific research priorities, discussed potential measurement schemes, and began to research technological options (eg, data streams, sensors, active tasks, analytical methods). We also began to discuss a variety of technical, user experience, regulatory, and other considerations relevant to the research program. These are described in Multimedia Appendix 1.
Milestone 1: Propose a Monitoring Plan
We prioritized the identified variables based on clinical utility, technological feasibility, alignment with patient priorities, and ethical considerations to select a final measurement scheme for the biosensors (Table 1). Additional clinical, traditional, and mobile data collection methods were also selected, but are outside the scope of this case study. Based on this scheme and patient insights, we defined a preliminary list of required and desired device selection criteria, their relative priorities, and opportunities for compromise. Briefly, the criteria described desired technical capabilities, data quality, user experience, regulatory status, privacy, required investment, and vendor characteristics. Opportunities for compromise included conditions under which multiple devices could be used, acceptable concessions described by patients, and acceptable trade-offs to meet the study budget (eg, willingness to develop bespoke software if device costs are reduced). A summary of these criteria and compromises is available in Multimedia Appendix 1.
Device-based remote measurement scheme for the RADAR-CNS multiple sclerosis study.
Factor
Measurement
Measurement Frequency
Gait
Measured via accelerometer and gyroscope during a 2-Minute Walk Test, tandem walk test, and normal daily activities
Clinical testsa, home testsb, free livingc
Balance
Measured via accelerometer placed on the chest during Romberg’s Test and normal daily activities
Clinical tests, home tests, free living
Fatigue
Measured via heart rate variability and accelerometer during a 2-Minute Walk Test and normal daily activities
Clinical tests, home tests, free living
Heart rate and heart rate variability
Measured via one-lead electrocardiogram placed on chest during tests and normal daily activities
Clinical tests, home tests, free living
Heart rate and heart rate variability
Measured via photoplethysmography
Dailyd
Sleep
Total sleep time and sleep patterns monitored via actigraphy or other mechanism
Daily
Daily Activity
Measured via actigraphy
Daily
aClinical tests: once every 3 months.
bHome tests: once every 3 months.
cFree living: one week every 3 months.
dDaily: daily over the course of the study.
RADAR-CNS: Select
We then identified relevant commercially-available consumer and research-grade devices. As no published database contained up-to-date information on available RMT, we conducted an online search and a literature search to identify devices that contained some or all of the sensors in the desired measurement scheme. This search yielded over 100 devices of various embodiments. Devices were systematically excluded through an iterative review process with clinical, analytical, and technical experts, during which potential technologies, priorities, and protocol adjustments were discussed. No single technology fulfilled all selection criteria; however, several devices that fulfilled most criteria were selected for further consideration either as stand-alone devices or for use in conjunction with other devices. These included the Fitbit Charge 2 (Fitbit, Inc., San Francisco, CA), the Withings Steel HR (Withings, Issy-les-Moulineaux, France), the Actigraph Link (ActiGraph LLC, Pensacola, FL), the Suunto Movesense sensor (Suunto Oy, Vantaa, Finland), the eMotion Faros 180 (Biomation, Ottawa, ON, Canada), and the MetaMotion R (MBIENTLAB Inc, San Francisco, CA).
Proof of Concept Testing
Questions regarding usability, data quality, and technical characteristics of the devices arose, prompting appropriate PoC testing of usability, technical features, and training procedures. This section describes two examples of these PoC tests and their impacts on technology selection.
Example: User Experience Proof of Concept
Sustained patient engagement with the devices was critical to the study’s success, because participants could be enrolled for up to 2 years. The patient advisory board participated in a workshop to provide feedback on candidate devices. Board members, including two members living with MS (authors JW and PB), interacted with each device and provided feedback on user-friendliness, technology preferences, potential impacts of MS symptoms on use, and suggestions for the implementation strategy. This feedback provided us with important context for prioritizing desired device characteristics. The board preferred adhesive patches over chest straps to affix chest-based devices and wrist-based wearables with a subtle or mainstream appearance. They also noted that any goals or feedback shown by the devices, such as daily activity counts, should be customizable. They voiced concern that displaying unrealistic goals could negatively impact participants’ motivation to engage with RMT or participate in the study, as people living with MS will almost certainly observe a decline in function over time.
Example: Technical Proof of Concept
Following a brainstorming session, the team decided to explore the option of sourcing sensors from an original equipment manufacturer. These devices would be less expensive and more customizable but required additional validation and configuration compared to other options. For commercial reasons, the identities of these devices are not shared. Data were collected from two devices to understand data structure, battery life, reliability of the Bluetooth connection, potential for data loss, data transfer requirements (eg, time, file size, memory availability), and device durability. The devices’ published specifications met the requirements; however, the testing demonstrated that neither device met study requirements. The first device’s data files were too large to sync more than a few hours of data over a Bluetooth connection, but the study required devices to sync data over Bluetooth outside the clinic. The second device did not meet battery life or data quality requirements in the desired configuration. We tested other candidate devices similarly to address the risks identified by the advisory board and the study teams.
In response to this PoC, we adjusted our technology landscape to include more expensive devices since the tested devices were the only two to meet original budget requirements. To accommodate this change, we also adjusted the implementation strategy to include logistics associated with device returns and reprovisions, thereby reducing the number of required devices and reducing the device cost per patient. This PoC did not yield positive results, but it allowed the team to make data-informed decisions on device candidates without compromising timelines or posing risks to the study.
Milestone 2: Select Technology and Implementation Strategy
Ultimately, we selected 2 devices to conduct all desired measurements. The eMotion Faros 180 was selected to monitor cardiac activity, gait, and balance during home-based active tasks and normal daily activities. The Fitbit Charge 2 was selected to monitor daily activity and sleep based on its superior user experience and battery life, as well as the precedence of Fitbit devices in MS programs [58-60], despite its inability to provide raw accelerometer data. Since no device containing an electrocardiogram, accelerometer, and gyroscope met the necessary criteria, data from the gyroscope sensor in participants’ cell phones were collected to identify turns during the 2-Minute Walk Test. A discussion guide used by the team to facilitate the final selection of the wrist-based device is included in Multimedia Appendix 2.
RADAR-CNS: Learn
The RADAR-CNS study is ongoing at the time of publication. Surveys and interviews with participants are being conducted periodically throughout the study and device use rates will be monitored as the study progresses. Feedback will also be collected from investigators who conducted the studies. Insights gained through these interactions will be published at the end of the study and will be used to identify improvements to the measurement scheme, device selection, and implementation strategy before the system is available for use in clinical practice.
Discussion
The RADAR-CNS Device Selection Framework provides methods to assess, prioritize, and adapt device selection criteria for health care programs according to stakeholder needs. The framework is presented linearly, but it is intended to be flexible so teams can move forward, backward, or repeat steps as needed to support device selection. In the RADAR-CNS study, we conducted several iterations of the Prepare and Select stages as our thinking evolved during the study design. These iterations enabled dialogue between the technical and clinical experts on the project, allowing us to establish common ground between stakeholders and ensure consensus on the final decision. We found that our success depended on the engagement of a multifunctional team during each stage of the framework, including investigators, IT specialists, data analysts, patients, health care professionals, and others. Each brought unique perspectives and needs to the process, and each ultimately made compromises to agree on a single technology and implementation strategy. To ensure alignment and mutual understanding between these stakeholders, it was important that members of the device selection team were skilled in “translating” clinical and technical requirements and their contexts for team members of diverse backgrounds.
Navigating complex stakeholder needs is one of the strengths of HCD, especially when program success is dependent on the willingness of people to continually engage with a technology. As its name suggests, HCD starts by asking designers to understand the people who will be using the technology [38,40,44]. It then enables designers to simultaneously explore program contexts and constraints, identifying connections and priorities between human and nonhuman factors [38,44]. In a systematic review of systematic reviews, Ross et al (2016) [30] found that early engagement with relevant stakeholders such as patients, clinicians, and others was important for successful mHealth implementation, and most frameworks for digital health care solution design echo that sentiment [33]. However, Altman et al (2018) [24] found that user engagement activities were frequently not conducted in such programs. Limited stakeholder centricity during program design and technology selection may ultimately threaten the program’s success. Poor user experiences caused by increased burdens [4,26], technical issues [22], lack of accommodations for health status [22], impersonal experiences [26], slowness [22,26], and poor or unclear interface designs [22] may cause patients to stop using the technology, or worse, drop out of the program. Altman et al [24] suggested that, by addressing user needs, HCD methods such as design thinking could increase uptake, adherence, and impact of health care programs that use RMT.
Here, we use HCD methods not to create new designs, but to identify which existing designs are best suited to a particular program. In the RADAR-CNS program, we used HCD methods to identify and prioritize a vast number of often conflicting needs and constraints, not only from patients but also from other “users” of the program: the clinicians caring for patients, the researchers studying diseases, and the technologists developing new monitoring tools. Many common HCD strategies such as empathizing with users, brainstorming, and iterative designing are present in this framework, making it compatible with other HCD approaches to program design or validation.
Though the RADAR Device Selection framework was implemented successfully in an observational research program, its validity in other settings, such as clinical trials of investigational therapies or interventional mHealth program design, must be established in future work. Examples of successful implementation of human-centered methods in health care and academic environments exist; however, their use is not yet routine. Such methods require a mindset shift, new skills, and adoption of additional study planning activities, with more time spent initially on stakeholder engagement [24].
Though selecting off-the-shelf devices for health care programs is often difficult, few best practices exist to guide program designers. To address this gap, we developed and successfully implemented the RADAR device selection framework, which incorporates HCD strategies into a three-stage approach for systematically identifying selection criteria, testing and selecting devices, and monitoring device-related outcomes. To improve RMT implementation in future programs, the methods used and lessons learned during device selection should be more routinely shared.
RADAR-CNS device selection considerations and selection criteria.
RADAR-CNS Multiple Sclerosis Study Device Selection Discussion Guide.
AbbreviationsCNS
Central Nervous System
HCD
human-centered design
IT
information technology
mHealth
mobile health
MS
multiple sclerosis
PoC
proof of concept
RADAR
Remote Assessment of Disease and Relapse
RMT
remote measurement technologies.
This paper was written in support of the RADAR-CNS program, a collaborative research effort focusing on the development of RMT for monitoring central nervous system disease progression. We acknowledge all partners in the RADAR-CNS consortium (www.radar-cns.org) for their input and support of this work. The RADAR-CNS project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking (www.imi.europa.eu) under grant agreement no 115902. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation program and European Federation of Pharmaceutical Industries and Associations (EFPIA; www.imi.europa.eu). This communication reflects the views of the RADAR-CNS consortium, and neither the Innovative Medicines Initiative nor the European Union and EFPIA are liable for any use that may be made of the information contained herein. This paper represents independent research part funded by the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. MH and TW are National Institute of Health Research senior investigators.
AP, JN, JF, VH, MD, and GT are employees of Merck Sharp & Dohme Corp, a subsidiary of Merck & Co, Inc, Kenilworth, NJ, USA and may hold stock or stock options in the company. VN and NVM are employed by Janssen Research & Development, LLC and may hold stock or stock options in the company. The remaining authors declare no conflict of interest.
NoahBKellerMSMosadeghiSSteinLJohlSDelshadSTashjianVCLewDKwanJTJusufagicASpiegelBMRImpact of remote patient monitoring on clinical outcomes: an updated meta-analysis of randomized controlled trialsNPJ Digit Med201812017210.1038/s41746-017-0002-4313043462PMC6550143DorseyERTopolEJState of TelehealthN Engl J Med2016071437521546110.1056/NEJMra160170527410924MarziniakMBrichettoGFeysPMeyding-LamadéUVernonKMeuthSGThe use of digital and remote communication technologies as a tool for multiple sclerosis management: narrative reviewJMIR Rehabil Assist Technol2018042451e510.2196/rehab.780529691208v5i1e5PMC5941090PolhemusAMKadhimHBarnesSZebrowskiSESimmondsAMasandSNBannerJDupontMAccelerating adoption of patient-facing technologies in clinical trials: a pharmaceutical industry perspective on opportunities and challengesTher Innov Regul Sci20190153182410.1177/216847901880156630373404DavisMMFreemanMKayeJVuckovicNBuckleyDIA systematic review of clinician and staff views on the acceptability of incorporating remote monitoring technology into primary careTelemed J E Health2014052054283810.1089/tmj.2013.016624731239PMC4011427PerryBHerringtonWGoldsackJCGrandinettiCAVasishtKPLandrayMJBatailleLDiCiccoRABradleyCNarayanAPapadopoulosEJShethNSkodacekKStemKStrongTVWaltonMKCorneliAUse of mobile devices to measure outcomes in clinical research, 2010-2016: a systematic literature reviewDigit Biomark201821113010.1159/00048634729938250dib-0002-0011PMC6008882VegesnaATranMAngelaccioMArconaSRemote patient monitoring via non-invasive digital technologies: a systematic reviewTelemed J E Health20170123131710.1089/tmj.2016.005127116181PMC5240011MarcolinoMSOliveiraJAQD'AgostinoMRibeiroALAlkmimMBMNovillo-OrtizDThe impact of mHealth interventions: systematic review of systematic reviewsJMIR Mhealth Uhealth2018011761e2310.2196/mhealth.887329343463v6i1e23PMC5792697LobeloFKelliHMTejedorSCPrattMMcConnellMVMartinSSWelkGJThe wild wild west: a framework to integrate mHealth software applications and wearables to support physical activity assessment, counseling and interventions for cardiovascular disease risk reductionProg Cardiovasc Dis20165865849410.1016/j.pcad.2016.02.00726923067S0033-0620(16)30015-9PMC5362162WilsonKBellCWilsonLWittemanHAgile research to complement agile development: a proposal for an mHealth research lifecycleNPJ Digit Med201814610.1038/s41746-018-0053-13130432653PMC6550198ChuAHYNgSHXPaknezhadMGauterinAKohDBrownMSMüller-RiemenschneiderFComparison of wrist-worn Fitbit Flex and waist-worn ActiGraph for measuring steps in free-living adultsPLoS One2017122e017253510.1371/journal.pone.017253528234953PONE-D-16-33057PMC5325470DooleyEEGolaszewskiNMBartholomewJBEstimating accuracy at exercise intensities: a comparative study of self-monitoring heart rate and physical activity wearable devicesJMIR Mhealth Uhealth2017031653e3410.2196/mhealth.704328302596v5i3e34PMC5374271RosenbergerMEBumanMPHaskellWLMcConnellMVCarstensenLLTwenty-four hours of sleep, sedentary behavior, and physical activity with nine wearable devicesMed Sci Sports Exerc2016034834576510.1249/MSS.000000000000077826484953PMC4760880FergusonTRowlandsAVOldsTMaherCThe validity of consumer-level, activity monitors in healthy adults worn in free-living conditions: a cross-sectional studyInt J Behav Nutr Phys Act201503271214210.1186/s12966-015-0201-92589016810.1186/s12966-015-0201-9PMC4416251KoPTKientzJAChoeEKKayMLandisCAWatsonNFConsumer sleep technologies: a review of the landscapeJ Clin Sleep Med20151215111214556110.5664/jcsm.528826156958jc-00498-14PMC4661339BrookeSMAnHKangSNobleJMBergKELeeJConcurrent validity of wearable activity trackers under free-living conditionsJ Strength Cond Res2017043141097110610.1519/JSC.000000000000157127465631WangRBlackburnGDesaiMPhelanDGillinovLHoughtalingPGillinovMAccuracy of wrist-worn heart rate monitorsJAMA Cardiol201701012110410610.1001/jamacardio.2016.3340277327032566167GillinovSEtiwyMWangRBlackburnGPhelanDGillinovAMHoughtalingPJavadikasgariHDesaiMYVariable accuracy of wearable heart rate monitors during aerobic exerciseMed Sci Sports Exerc2017084981697170310.1249/MSS.00000000000012842870915500005768-201708000-00022MurakamiHKawakamiRNakaeSNakataYIshikawa-TakataKTanakaSMiyachiMAccuracy of wearable devices for estimating total energy expenditure: comparison with metabolic chamber and doubly labeled water methodJAMA Intern Med201605011765702310.1001/jamainternmed.2016.0152269997582500062Clinical Trials Transformation Initiative20182019-05-30CTTI Recommendations: Advancing the Use of Mobile Technologies for Data Capture & Improved Clinical Trialshttps://www.ctti-clinicaltrials.org/files/mobile-devices-recommendations.pdfGagnonMNganguePPayne-GagnonJDesmartisMm-Health adoption by healthcare professionals: a systematic reviewJ Am Med Inform Assoc2016012312122010.1093/jamia/ocv05226078410ocv052SimblettSGreerBMatchamFCurtisHPolhemusAFerrãoJGamblePWykesTBarriers to and facilitators of engagement with remote measurement technology for managing health: systematic review and content analysis of findingsJ Med Internet Res20180712207e1048010.2196/1048030001997v20i7e10480PMC6062692ParkerSJJesselSRichardsonJEReidMCOlder adults are mobile too!Identifying the barriers and facilitators to older adults' use of mHealth for pain managementBMC Geriatr20130506134310.1186/1471-2318-13-43236479491471-2318-13-43PMC3673892AltmanMHuangTTBrelandJYDesign thinking in health carePrev Chronic Dis2018092715E11710.5888/pcd15.18012830264690E117PMC6178900DonnellySReginattoBKearnsOMc CarthyMByromBMuehlhausenWCaulfieldBThe burden of a remote trial in a nursing home setting: qualitative studyJ Med Internet Res20180619206e22010.2196/jmir.963829921563v20i6e220PMC6030571O'ConnorSHanlonPO'DonnellCAGarciaSGlanvilleJMairFSUnderstanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studiesBMC Med Inform Decis Mak2016091516112010.1186/s12911-016-0359-32763002010.1186/s12911-016-0359-3PMC5024516ByromBWatsonCDollHCoonsSJEremencoSBallingerRMc CarthyMCrescioniMO'DonohoePHowryCePRO ConsortiumSelection of and evidentiary considerations for wearable devices and their measurements for use in regulatory decision making: recommendations from the ePRO ConsortiumValue Health20180621663163910.1016/j.jval.2017.09.01229909867S1098-3015(17)33532-5Clinical Trials Transformation Initiative20182019-05-30Framework of Specifications to Consider During Mobile Technology Selectionhttps://www.ctti-clinicaltrials.org/sites/www.ctti-clinicaltrials.org/files/technology-selection-tool.pdfOstrovskyADeenNSimonAMateKInstitute for Healthcare Improvement20142019-05-30Cambridge, MAA framework for selecting digital health technology: IHI innovation reporthttp://www.ihi.org/resources/Pages/Publications/AFrameworkforSelectingDigitalHealthTechnology.aspxRossJStevensonFLauRMurrayEFactors that influence the implementation of e-health: a systematic review of systematic reviews (an update)Implement Sci2016102611114610.1186/s13012-016-0510-72778283210.1186/s13012-016-0510-7PMC5080780TransCelerate BioPharma, Inc20182019-05-30Patient technology implementation frameworkhttps://transceleratebiopharmainc.com/patient-technology-implementation-framework/MummahSARobinsonTNKingACGardnerCDSuttonSIDEAS (Integrate, Design, Assess, and Share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behaviorJ Med Internet Res201612161812e31710.2196/jmir.592727986647v18i12e317PMC5203679van Gemert-PijnenJENijlandNvan LimburgMOssebaardHCKeldersSMEysenbachGSeydelERA holistic framework to improve the uptake and impact of eHealth technologiesJ Med Internet Res20111205134e11110.2196/jmir.167222155738v13i4e111PMC3278097SchnallRRojasMBakkenSBrownWCarballo-DieguezACarryMGelaudeDMosleyJPTraversJA user-centered model for designing consumer mobile health (mHealth) applications (apps)J Biomed Inform201604602435110.1016/j.jbi.2016.02.00226903153S1532-0464(16)00024-1PMC4837063WilhideCCPeeplesMMAnthonyKRCEvidence-based mHealth chronic disease mobile app intervention design: development of a frameworkJMIR Res Protoc2016021651e2510.2196/resprot.483826883135v5i1e25PMC4773596SadeghSSKhakshour SaadatPSepehriMMAssadiVA framework for m-health service development and success evaluationInt J Med Inform20180411212313010.1016/j.ijmedinf.2018.01.00329500009S1386-5056(18)30003-0Eslami AndargoliAScheepersHRajendranDSohalAHealth information systems evaluation frameworks: A systematic reviewInt J Med Inform2017019719520910.1016/j.ijmedinf.2016.10.00827919378S1386-5056(16)30223-4NormanDThe Design Of Everyday Things2013New York, NYBasic BooksDesign Kit20192019-02-28IDEO.orgWhat is human-centered design?http://www.designkit.org/human-centered-designDesign Council News and Opinion20152019-02-27London, UKDesign CouncilWhat is the framework for innovation? Design Council's evolved Double Diamondhttps://www.designcouncil.org.uk/news-opinion/design-process-what-double-diamondRiesEThe Lean Startup: How Today's Entrepreneurs Use Continuous Innovation To Create Radically Successful Businesses2011New YorkCrown BusinessHevnerAA three cycle view of design science researchScand J Inf Syst20071927Van VelsenLWentzelJVan Gemert-PijnenGJEDesigning eHealth that matters via a multidisciplinary requirements development approachJMIR Res Protoc2013062421e2110.2196/resprot.254723796508v2i1e21PMC3815432Standford DSchool20182019-02-28Design thinking bootleghttps://dschool.stanford.edu/resources/design-thinking-bootlegWoodsLCummingsEDuffJWalkerKPartnering in Digital Health Design: Engaging the Multidisciplinary Team in a Needs AnalysisStud Health Technol Inform201825217618130040702WoodsLCummingsEDuffJWalkerKDesign thinking for mHealth application co-design to support heart failure self-managementStud Health Technol Inform20172419710228809190BatePRobertGExperience-based design: from redesigning the system around the patient to co-designing services with the patientQual Saf Health Care2006101553071010.1136/qshc.2005.0165271707486315/5/307PMC2565809U.S. Food and Drug Administration20182019-05-30Patient-focused drug development: collecting comprehensive and representative input, guidance for industry, food and drug administration staff, and other stakeholdershttps://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm610442.pdfJames Lind Alliance20182019-01-23The PSPshttp://www.jla.nihr.ac.uk/priority-setting-partnerships/LewisJRPsychometric evaluation of the PSSUQ using data from five years of usability studiesInternational Journal of Human-Computer Interaction200209143-446348810.1080/10447318.2002.9669130ChinWWJohnsonNSchwarzAA fast form approach to measuring technology acceptance and other constructsMIS Quarterly200832468710.2307/25148867MatchamFBarattieri di San PietroCBulgariVde GirolamoGDobsonRErikssonHFolarinAAHaroJMKerzMLamersFLiQManyakovNVMohrDCMyin-GermeysINarayanVBwjhPRanjanYRashidZRintalaASiddiSSimblettSKWykesTHotopfMRADAR-CNS consortiumRemote assessment of disease and relapse in major depressive disorder (RADAR-MDD): a multi-centre prospective cohort study protocolBMC Psychiatry201902181917210.1186/s12888-019-2049-z3077704110.1186/s12888-019-2049-zPMC6379954RADAR-CNS Consortium20192019-01-23RADAR-CNS: Remote Assessment of Disease and Relapsehttps://www.radar-cns.org/RADAR-CNS Consortium20192019-02-28Patient Advisory Board | RADAR-CNShttps://www.radar-cns.org/patient-engagement/patient-advisory-boardSimblettSKEvansJGreerBCurtisHMatchamFRadaelliMMuleroPArévaloMJPolhemusAFerraoJGamblePComiGWykesTRADAR-CNS consortiumEngaging across dimensions of diversity: A cross-national perspective on mHealth tools for managing relapsing remitting and progressive multiple sclerosisMult Scler Relat Disord2019073212313210.1016/j.msard.2019.04.02031125754S2211-0348(19)30180-4BrunoESimblettSLangABiondiAOdoiCSchulze-BonhageAWykesTRichardsonMPRADAR-CNS ConsortiumWearable technology in epilepsy: the views of patients, caregivers, and healthcare professionalsEpilepsy Behav2018088514114910.1016/j.yebeh.2018.05.04429940377S1525-5050(18)30356-1SimblettSMatchamFSiddiSBulgariVBarattieri di San PietroCHortas LópezJFerrãoJPolhemusAHaroJMde GirolamoGGamblePErikssonHHotopfMWykesTRADAR-CNS ConsortiumBarriers to and facilitators of engagement with mHealth technology for remote measurement and management of depression: qualitative analysisJMIR Mhealth Uhealth2019013071e1132510.2196/1132530698535v7i1e11325PMC6372936DasMahapatraPChiauzziEBhaleraoRRhodesJFree-living physical activity monitoring in adult US patients with multiple sclerosis using a consumer wearable deviceDigit Biomark201841321476310.1159/000488040BlockVJLizéeACrabtree-HartmanEBevanCJGravesJSBoveRGreenAJNourbakhshBTremblayMGourraudP-ANgMYPletcherMJOlginJEMarcusGMAllenDDCreeBACGelfandJMContinuous daily assessment of multiple sclerosis disability using remote step count monitoringJ Neurol201702264231632610.1007/s00415-016-8334-62789643310.1007/s00415-016-8334-6PMC5292081SasakiJESandroffBBammanMMotlRWMotion sensors in multiple sclerosis: Narrative review and update of applicationsExpert Rev Med Devices201711141189190010.1080/17434440.2017.138655028956457PMC6291837