The umbrella review protocol was developed based on the Cochrane Handbook for Systematic Reviews of Interventions [33] and other relevant methodology sources [34] and was registered with PROSPERO (CRD42022338785). StArt (State of the Art through Systematic Review) software [35] was used for the first- and second-level screening of result datasets and extracting relevant information.

Based on the objectives of the study, the following inclusion criteria were formulated: (1) articles published between January 1, 2015, and April 27, 2023 (the date range reflected the launch of Apple ResearchKit in 2015, which accelerated mHealth development and research [36]); (2) containing data on human participants; (3) with the “unit of searching” [33] being “systematic reviews” [37,38] in order to reduce the effect of cumulative bias that may arise when including nonsystematic reviews; (4) examining usability assessment instruments of mobile apps for health professionals and for health care consumers; and (5) published in the English language to enable all contributing authors to perform screening, extraction, and synthesis of the search results. No post hoc modifications were made to the inclusion criteria. Systematic reviews of usability assessment instruments of other (mobile) solutions such as wearables, sensors, virtual reality, blockchain, Internet of Things, simulated data, or solutions for health care professionals only were excluded.

The following databases were searched with a combination of the search terms mobile application*, mobile app, usab*, usab* criteria, usab* evaluat*, systematic review, mhealth, mobile health, and physical exercise: PubMed, Cochrane, IEEE Xplore, Epistemonikos, Web of Science, and CINAHL Complete, combined using Boolean operators OR and AND and customized for each database in accordance with their filtering specifications. The result sets were imported into StArt [35]. The full search syntax for each database are presented in Table S1 in Multimedia Appendix 1.

A preliminary search of existing systematic reviews was conducted before finalizing the search terms in order to scope the extent and type of existing evidence [33]. The subsequent final search terms produced a result set that was more refined in focus and feasible in terms of the size of the expected result set. Following the removal of duplicates, 2-level screening was performed: title and abstract screening was performed by the primary author (SH), and a randomly selected subset of articles (118/1479, approximately 8%) was screened by a second author (VS; κ=0.87). Second-level, full-text screening was performed by the primary author (SH) using StArt for data extraction from the final result set. A data extraction form including basic reference details, as well as information such as population of interest and interventions studied, was discussed and agreed on by 3 authors (SH, GA, NS) before data extraction (see review protocol PROSPERO CRD42022338785 for more detail).

Quality assessment was completed using AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews 2; Institute for Clinical Evaluative Sciences) [39] by the primary author (SH) and a second author (VS) separately (κ=0.823). Any disagreement was discussed and resolved via consensus. In line with recommendations by Shea et al [39], a discussion to determine AMSTAR 2 critical domains for this umbrella review occurred among 2 authors (SH, NS). Criteria 2, 4, and 7 were retained on the premise of constituting critical criteria as defined by the original publication [39]. The original critical criteria 9, 11, 13, and 15 were classified as noncritical for the purpose of this umbrella review due to pertaining to meta-analytic steps that none of the included systematic reviews performed. Instead, the following criteria were classified as critical: criterion 5 due to the variety of study designs and target user groups and/or clinical contexts included within the systematic reviews; and criterion 16 due to the context of mHealth usability, where the borders between academic enquiry and commercialization are more blurred and funding could constitute a significant source of bias and/or conflict of interest. A summary rating was produced according to recommendations by Shea et al [39].

Finally, to gauge potential skewing of the data caused by significant overlap of primary studies contained within the systematic reviews included in this umbrella review [40], overlap assessment was achieved via citation matrix [41,42] for the systematic reviews including the System Usability Scale (SUS) as an exemplar. The SUS was chosen because it is one of the most well-known instruments [43] and preliminary searches of the literature demonstrated its frequency of use and reference.

The exponential growth of research evidence related to the effectiveness of mobile solutions for rehabilitation [85-88] and the proliferation of technological solutions that afford new modes of treatment delivery [89,90] underscore the critical need for high-quality mHealth usability evaluation. Usability attributes such as efficiency, learnability, and memorability [21] are particularly important to consider for mHealth users who may face challenges due to neurological compromise [91], age-related issues [23], or limited technology experience [13]. This umbrella review aimed to summarize usability assessment instruments for mHealth researchers, clinicians, and consumers to guide the development, assessment, and selection of high-quality mHealth tools.

The review identified, first, significant diversity and common use of custom-made instruments when usability assessment instruments were employed to evaluate mHealth tools for rehabilitation. Second, there was a notable lack of theoretical grounding for selection of the assessment of usability. Third, a scarcity of psychometric data for widely used instruments for mHealth usability assessment was evident in the systematic reviews included.

Regarding the first critical point, a wide range of different instruments for the assessment of usability was evident across the systematic reviews included. This range included adaptations of preexisting usability assessment instruments for the context of mobile apps [59,66] as well as assessment instruments, such as the Mobile App Rating Scale (MARS) [72], specifically designed for usability assessment of mHealth tools. In addition, both completely custom-made instruments and hybrids [92] of preexisting instruments with custom elements were prevalent in the mHealth usability literature.

Although the use of hybrid assessment instruments and adaptations of preexisting assessment instruments may increase flexibility and thereby possibly improve the experience for respondents, the fact that most studies are limited in sample size prevents validation of hybrid and adapted instruments [51]. Alternative approaches to increasing flexibility and improving respondent experience while ensuring psychometric integrity are needed instead. A good example of this may be seen in the creation of a hybrid version of the SUS with the inclusion of pictorial elements, which increased respondent motivation [92]. Importantly, acceptable validity, consistency, and sensitivity were also evidenced, allowing future users of the hybrid measure to place greater trust in the quality of the data.

Second, and similar to what has been found to be the case for individual-level studies assessing the usability of specific mHealth tools [47], this review revealed that some systematic reviews examining the broader literature related to usability assessment lacked connection to theoretical models of usability. This observation resonates with previous criticisms of the quality of reviews of health-related mobile apps [8] as well as research exploring technology adoption in fields beyond mHealth [93]. The latter exposed a reliance on a wide array of theoretical models of technology adoption in the literature and in some cases several within one review. To address this, it has been suggested that generic models for different service categories (eg, information and transaction) be developed [93]. A theoretically grounded, generic guide for mHealth usability assessment could similarly promote broader adoption and enhance comparison of usability across studies and use cases.

Third, systematic reviews included in our overview also reported significant limitations regarding the psychometric properties of preexisting instruments. For example, the MARS tool, which has been put forward as an instrument for standardized use in mHealth usability assessment [51], lacks structural validity. Moreover, other constructs such as internal consistency and criterion validity have been documented as significant areas of future work for measuring the implementation of interventions [53], with usability assessment playing a significant role.

Although consistent with previous research, this umbrella review did not specifically search for psychometric evaluations of usability assessment instruments; instead, it relied on summaries of psychometric evaluations presented as part of the included systematic reviews. As a result, it is likely that psychometric evaluation of other instruments is available. For example, psychometric evaluation of the popular USE Questionnaire [58] is available and, consistent with our observation, has been shown to be affected by a lack of reliability and validity [94]. Furthermore, outside of the academic literature, there is a still greater portion of mHealth solutions on the market that likely will not have undergone empirical evaluation of usability.

Although some of the acceptable psychometric information was referenced for the SUS [95], both the IBM Usability Satisfaction Questionnaire and the USE Questionnaire appear to lack reliability assessment. Reliability, or the freedom of measurement error [96], may be regarded as crucial with regard to any metrics that are gathered after, rather than during, a user’s interaction with an application. The inability to separate true change in users’ estimate of the usability of mHealth tools from random variation, or measurement error, originating from recall bias [9,97,98], for example, means that mHealth tool iterations [99] are unable to be evaluated appropriately.

Moreover, the widespread use of custom-made and hybrid assessment instruments leads to the loss of the original instrument’s integrity and compromises its already-documented psychometric strengths [100]. Consequently, establishing the validity of results from individual usability investigations becomes challenging, and comparison across studies is difficult. Hence, there is an urgent need to assess the accuracy and appropriateness [101] of individual usability assessment instruments to capitalize on the promise of mHealth tools in rehabilitation [5,6].

Another important psychometric aspect of usability assessment instruments that the systematic reviews included in this umbrella review highlight as missing from the published literature is responsiveness. mHealth development usually involves iterative design and testing cycles [30,99] with associated formative and summative usability evaluation [45]. Across the life of mHealth development, iterative cycles are likely to span different stages of development and be undertaken in different clinical contexts [54,102]. Integrating usability assessment into this process requires instruments that are generic enough to capture user responses to a wide variety of mHealth strategies but also fine-grained enough to possess sufficient responsiveness [96].

Finally, with regard to the argument of lacking psychometric assessment, none of the preexisting mHealth usability assessment instruments referenced as part of the literature included in this umbrella review appear to have been informed by a breadth of cultural perspective or undergone cross-cultural validity testing. Given the global potential of mHealth to address inequities in access to and outcomes from rehabilitation [5,6], it is particularly important to establish cross-cultural validity of the usability assessment instruments employed in mHealth development. In addition, with the pervasiveness of technology, there is a certain element of unpredictability of the context in which mHealth tools will be trialed and used “in the wild” [103,104]. For that reason, an alternative argument could be made for innovative, culturally responsive methodology for mHealth tool design including usability testing [105]. A key difference in such attempts is user participation at multiple stages of development and responsiveness to expanding the stages of development as guided by stakeholders. This process likely includes constant negotiation and may be resource heavy but is arguably needed if the aim is to create mHealth solutions impacting indigenous outcomes, for example [74,106].

Considering the identified issues, including lack of theoretical grounding, common use of custom-made assessment instruments, and the scarcity of psychometric data for widely used mHealth usability assessment instruments, multimethod usability assessment appears paramount. This is consistent with recommendations made by a number of research groups [13,44,102,107] and reinforces the argument often advanced in favor of Ecological Momentary Assessment approaches, which are recognized for their advantage over retrospective assessment [97]. It is therefore proposed that standards be developed that specify the time points in the mHealth life cycle at which usability assessment is completed, with an emphasis on what methods to use. Moreover, these standards should mandate that individual assessment instruments are grounded in a theoretical framework and possess a minimum threshold for psychometric properties [53,108].

The establishment of a universal usability scoring system or algorithm would further facilitate the integration of these assessments into an overall framework [109]. It has been observed that, at present, less than half of existing evaluation frameworks include such a scoring system, but that such systems could support funding decisions [29] and advance the vision of prescribable mHealth apps [10]. Although technological advancement often outpaces academic enquiry necessitating new approaches to mHealth evaluation frameworks [110], usability factors are enduring [16] and investing resources into establishing standards will therefore be valuable.

In the context of an area of practice where the lines between commercial and academic work are blurred and usability assessment constitutes a common practice in the global commercial environment [111], this umbrella review is limited to only including English language systematic reviews published within the academic literature indexed in the databases included. Furthermore, the quality of the included systematic reviews was found to be limited, and the fit of the AMSTAR 2 tool with methodological papers is not perfect. However, AMSTAR 2 could be argued to be more detailed than instruments developed for umbrella reviews specifically [31], and, in line with the AMSTAR 2 recommendations [39], the authors modified the list of critical criteria to reflect the specific aim of the overview. Finally, with regard to the review’s methodology, 2 limitations are of note. First, although the search syntax for this umbrella review included the keyword “physical exercise,” for pragmatic reasons, no validation step was included to confirm that all mHealth tools examined as part of the primary studies included within the systematic reviews included a physical exercise component. Regardless, the observations presented here are valid for mHealth tools for rehabilitation overall and provide valuable guidance to developers, researchers, and clinicians. Second, for practical reasons, data selection could only be performed by the primary author (SH) with a subset of articles being screened by a second author (VS). However, agreement on study selection was high (>80%), supporting the quality of the review.

There is considerable variety in approaches to and instruments for the assessment of usability in mHealth for rehabilitation, many of which lack theoretical foundation. Clinicians are therefore advised to critically evaluate mHealth literature and solutions, paying particular attention to the population in which usability testing was performed and the specific usability assessment instruments were employed. Future research efforts should be focused on producing high-quality systematic reviews and psychometric evaluations of usability assessment instruments. A collaborative effort between researchers, designers, and developers is essential to establish mHealth tool development standards. These standards should emphasize the incorporation of usability assessment instruments underpinned by a robust theoretical base. This umbrella review represents a valuable reference tool in this endeavor. Inclusion of multimethod usability assessment within the wider mHealth development cycle could also be part of these standards, which will ensure that we can capitalize on the widely heralded promise of mHealth to promote access to and outcomes from rehabilitation.