Mobile technology has become a nearly ubiquitous technology in the United States, where almost two-thirds of the population owns a mobile phone, but also worldwide with more than half of the world population owning a mobile phone [1]. Mobile technology has the advantage of accessibility anywhere Internet access is available, relative affordability, and has been promoted specifically as a solution to reach stigmatized and disenfranchised populations [2,3]. Mobile technology can be particularly useful in the delivery of health interventions because this technology can allow us to deliver interventions to scale, cover broad geographic areas, and deliver technologies in highly tailored ways based on the preferences or characteristics of users [4,5].

Concurrent with the proliferation of mobile technology has been the vast increase in the number of mobile health (mHealth) apps available to consumers. For example, in 2013, a report by the IMS Institute for Healthcare Informatics estimated more than 40,000 health-related iPhone apps were available for consumer use [6]. By 2015, this number had more than doubled to 90,088 apps available on the iOS. In total, IMS estimated that 165,000 health apps were available for use both among iOS and Android platforms [6]. The mHealth apps use mobile devices (eg, mobile phones or tablets) for the deployment of health interventions and vary widely in the types of technology used (eg, text message reminders, app-based alerts and activities, and Web-based educational modules) [7].

The dramatic growth of mHealth technology is not surprising given that mobile technologies are a novel method for the delivery of cost-effective, timely, and relevant health promotion and management information [8]. Mobile delivery specifically has a number of key advantages over traditional face-to-face delivery models of care, including consumer control, decreased time burden, reduction of monetary and time costs associated with travel to a provider, and the ability to monitor and assess the use of digital analytics [8]. Additionally, mHealth technology presents an opportunity for consumers to self-monitor their health status and allows health care providers to organically reach persons who may be disengaged from the health care system [9]. Finally, mHealth technology allows for the dissemination of information quickly and broadly [10].

Use of mobile technology as a platform for behavioral interventions adjuncts to health care delivery and monitoring of health status has widely proliferated in the past 5 years. Despite this growth, few mHealth apps or interventions have undergone systematic and rigorous usability evaluation prior to their dissemination [11,12]. Usability evaluations remain challenging because of the cost and time necessary for completion of these assessments [13].

Another barrier to usability assessments is that a dearth of validated instruments for assessing mHealth technology persists [14-16]. Implications of insufficient usability testing in mHealth include the assertion that attrition is common within randomized controlled trials using technology due to usability errors [11,12,17]. However, without a standardized measure of usability, it is very difficult to assess, report on, and compare usability, as well as link it to attrition from the trial. At the same time, a standardized usability instrument would improve methodological consistency, making it possible to begin comparing findings across mHealth app evaluations [18].

In our earlier work, we presented preliminary validation of the Health Information Technology Usability Evaluation Model (Health-ITUEM), a theoretical framework to guide usability evaluations for assessing mHealth technologies [19]. The model was developed in response to the current gaps in the extant usability literature [20]. The Health-ITUEM is an integrated model of multiple usability theories based on the concepts of usability from the Technology Acceptance Model [21] and the International Organization for Standardization (ISO) standard 9241-11 [22]. The Health-ITUEM has been widely used as a framework for understanding the use of mobile technology studies since its validation [23,24], but there remains a lack of validated instruments for usability assessments of mHealth technology.

A need for a validated usability instrument for mHealth technologies persists. In response to this need, we sought to validate the use of the Health Information Technology Usability Evaluation Scale (Health-ITUES) in a sample of users of a mHealth app developed for community-dwelling adults living with HIV. The Health-ITUES, derived from the Health-ITUEM, is a validated instrument that explicitly considers task by addressing various levels of expectation of support for the task by the health information technology (IT). The Health-ITUES also has the added benefit of being customizable, which can address the study needs and concepts measured without item addition, deletion, or modification. This is an important benefit of this instrument because it allows for harmonization of findings across studies. The factorial validity and internal consistency of the Health-ITUES was demonstrated through an exploratory factor analysis [25]. The development of the Health-ITUES was further advanced through a confirmatory factor analysis and structural equation modeling to demonstrate its construct validity and predictive validity [26]. Early development and valuation of the Health-ITUES was conducted using a Web-based communication system that supported nurse staffing and scheduling. The Health-ITUES is not widely used and it has not been validated in a sample of patients or with the use of mobile technology. These two distinctions make the findings from this work highly relevant, timely, and provide an important contribution to the literature.

Study participants were recruited from June 2016 to February 2017 through in-person and email recruitment. Recruitment sites included HIV clinics and community-based organizations. Eligibility criteria included: (1) diagnosed with HIV, (2) older than 18 years of age, (3) able to communicate in English, (4) experienced at least two of 13 HIV-related symptoms in the past week, (5) met the cognitive state minimum score (ie, 24 of 30) measured by the Mini-Mental State Examination (MMSE) [27], and (6) own a mobile phone or tablet. All participants completed written informed consent prior to the start of study activities. Following enrollment, study participants were given access to the mVIP app. A total of 20 participants enrolled as part of usability testing of the app and 80 participants enrolled in the 12-week feasibility study of the app. Data reported in this paper were collected at the baseline visit for each participant after he or she had used the app for the first time.

Participants in both groups were given access to the app on a mobile phone. Following log-in and use of the mobile app, participants completed a number of surveys, which are detailed subsequently. The surveys collected demographic information, the Health-ITUES [26], and the Post-Study System Usability Questionnaire (PSSUQ) [28]. Participants were given US $40 as a token of appreciation for their time for participating in the usability testing. Participants in the 12-week feasibility trial were given US $30 at baseline and US $40 at follow-up as a token of appreciation for their time.

A total of 92 persons completed the Health-ITUES survey. Twenty participants completed the survey as part of the end-user usability testing of the beta version of the app and 72 persons completed the survey on enrollment into the 12-week trial. As noted in Table 1, the majority of participants were African American/black and had an annual median household income of less than US $20,000. Nearly all our participants used their mobile phones several times per day. The mean age of the participants was 50.0 (SD 10.5) years with a range of 23.0 to 72.0 years. There was no significant correlation between demographic characteristics, such as income and education, and each of the Health-ITUES subscales.

The range, mean, median, and standard deviation for each of the subscales’ scores at baseline are reported in Table 2. Completion rates were identical for all subscales.

Variability was evaluated for each of the subscales. The full range of responses was not observed for any of the Health-ITUES subscales. Slightly more than a half (59%) scored the maximum score for the quality of life scale, a third (36%) scored the maximum score for the perceived usefulness scale, slightly more than a half (60%) scored the maximum score for the perceived ease of use scale, and more than a third (42%) scored the maximum score for the user control scale. No users scored the lowest possible score. Overall, the users rated the usability of mVIP app as being high. The full range of usability scores was not demonstrated because no users rated the app as the lowest possible score.

By examining the boxplots and the raw data, extremely low scores were treated as outliers and were removed from the reliability and validity analysis. There is evidence that all the subscales were negatively skewed, indicating more favorable usability scores.

This paper presents the validation of an instrument for measuring usability of mHealth technology. The validation study reported in this paper was conducted to explore the psychometric properties of the Health-ITUES, a customizable usability evaluation instrument that includes subscales of impact, perceived usefulness, ease of use, and user control in a sample of community-dwelling adults living with HIV who were users of a mobile app for symptom self-management. The Health-ITUES varies from most traditional measurement scales in that it is designed to support customization at the item level to match the specific task/expectation and health IT system while retaining comparability at the construct level. The results of this study support the construct and criterion validity and reliability of the Health-ITUES for usability assessments of mobile technologies.

In this study, the reliabilities of the four Health-ITUES subscales and overall score were examined in terms of internal consistency. The internal consistency reliability of each of the subscales were high but less than the Cronbach alpha=.95 threshold suggesting that there is no indication of redundancy in scale items. Moreover, a comparison of the results from the Health-ITUES to the PSSUQ had moderate to strong correlations (r=.46-.70). Information quality, a subscale on the PSSUQ, had the lowest correlation (r=.46, P<.001), which is to be expected because the constructs are substantively different. The overall scale scores for the PSSUQ and the Health-ITUES had the highest correlation (r=.70, P<.001). This would be expected because the two usability scales have some overlap in their constructs. At the same time, the overall correlations were only moderately strong [42], suggesting that these two instruments are not redundant and demonstrating good criterion validity.

A strength of this validation is our study sample, which further supports the utility of this instrument for usability assessments for consumer health informatics. Our study sample is comprised of nearly all racial and ethnic minority persons and those persons of the lowest socioeconomic groups in the United States. Nonetheless, our study participants were owners and frequent users of mobile phone devices, demonstrating the widespread use of mobile technology and the relevance of mHealth technology across end users. In contrast to the earlier development and validations of the Health-ITUES in nurses using a Web-based system [25,26], this paper presents a validation of this instrument for mHealth technology and with patients [43].

Construct validity was assessed by measuring differences in mean scale scores at baseline between control group and intervention group participants. Statistically significant differences between groups were found in the expected direction for all subscales except impact. These trends were expected because at baseline we would not expect to find a significant difference in impact as a result of the study intervention. In contrast, a statistically significant difference in perceived usefulness, ease of use, and user control was demonstrated, with intervention group participants finding the mobile app to be more usable than control group participants did.

The construct validity of the measure was also supported by the degree to which interscale correlations corresponded to what was expected. The higher correlations among the impact and perceived usefulness subscales than between the other subscale items supports the convergent and discriminant validity of this instrument because participants who perceive the technology to be useful are more likely to show an improvement in their impact as a response.

There are a number of limitations of this validation study. First, most of our study sample were mobile phone users and so we were unable to validate the use of this instrument as a usability assessment tool in non-mobile phone users. This limitation is mitigated as the number of mobile phone users continues to grow across geographic regions and socioeconomic groups. A second limitation is that these findings may not be generalizable beyond our study population. Further research is most definitely needed to further validate this instrument in other study populations. Given the dearth of current instruments for usability assessments, we believe that the findings from this study provide an important contribution to the literature and an opportunity for future study of this instrument with additional study populations.

This study provides preliminary evidence to support the validity and reliability of the Health-ITUES for usability assessments of mobile technology. In light of our findings, the authors recommend use of the Health-ITUES as a measurement tool for assessing the usability of mobile technologies. Validation of the Health-ITUES is an important step in ensuring that usability is attended to and maintained in mHealth technology interventions. This work is particularly important given the proliferation of mobile technologies and the push for consumers to take control of their own health and become greater users of technology for the acquisition and delivery of health information. Further, these findings will hopefully stimulate additional research and practice to ensure the usability of mobile technologies and particularly for those tools that are developed for consumers’ use.