Adapting and Validating Tools to Assess the Usability and Acceptability of mHealth Tools Among Community Health Workers in Rural Settings: Development and Usability Study

Introduction

Mobile health (mHealth) apps have emerged as important tools to support health care workers in delivering high-quality care, particularly in low- and middle-income countries. Over the past 2 decades, an increasing number of mHealth solutions have been developed to assist community health workers (CHWs) in performing critical tasks across a range of health interventions [1,2]. CHWs, who often have limited formal training, nonetheless play a vital role in screening, diagnosing, and managing health conditions at the community level [3]. mHealth apps targeting CHWs are typically designed to support specific workflows or interventions. However, to ensure that these tools are adopted and used effectively, it is essential to evaluate their usability and acceptability among intended users. Robust assessment can guide the refinement of interface design and core functionalities, ultimately improving user experience and the quality of care delivered. Despite the increasing deployment of CHW-focused mHealth tools, few published studies have quantitatively assessed their usability or acceptability in sub-Saharan Africa (SSA) [4-6], and to our knowledge, there are no validated quantitative tools that have been developed for this purpose.

There are several tools that assess the usability of digital interventions or mHealth apps among target users outside of CHWs. Examples of usability assessment questionnaires geared toward patients include the Mobile App Rating Scale, a tool to measure the quality of apps commonly used in the health domain [7]; the System Usability Scale, developed to assess the usability of health care innovations [8]; and the Health Information Technology Usability Evaluation Scale, a tool used to assess the usability of mHealth technologies in groups of adults living with chronic diseases [9]. Tools that assess usability among clinical providers are the Telehealth Usability Tool [10] and the mHealth App Usability Questionnaire (MAUQ) [11]. The Telehealth Usability Tool was designed to evaluate the interface quality, reliability, and satisfaction of computer-based telehealth videoconferencing software, whereas the MAUQ was developed to evaluate mHealth apps.

Evaluation of user acceptability is also a critical aspect in the development of mHealth apps. The Practitioner Opinion (Acceptability) Scale (POAS) is an example of a tool designed to evaluate the acceptability of decision aids during the development process and early evaluation stages [12]. Other examples of acceptability tools to evaluate mHealth technology include the Acceptability E-scale, [13], the technology acceptance model [14], and the User Engagement Scale [15]. However, none of these were developed to assess acceptability among CHWs.

In this paper, we describe our process of adapting existing tools to develop and validate a quantitative tool to assess CHWs’ acceptability of an mHealth app and its usability. This validated tool was specifically developed to assess the usability and acceptability of a new mHealth app to support CHW-led comprehensive home-based follow-up care among women who had delivered via cesarean section in rural Rwanda.

Methods

Study Design

Between October 2022 and March 2023, we adapted and validated a tool to assess the usability and acceptability of a novel mHealth screening app in a sample of target end users (CHWs). The mHealth app supports comprehensive home-based follow-up care among women who have delivered via cesarean section. The usability and acceptability assessment tool we developed is referred to as the Community Health Worker mHealth Usability and Acceptability Assessment Tool (CHW-MUAAT).

Study Setting

This study was implemented in the Kirehe district located in Eastern Province, rural Rwanda. The Kirehe district is represented by Kirehe District Hospital, with 19 health centers and 2448 CHWs. This hospital operates under the Rwanda Ministry of Health and is supported by Inshuti Mu Buzima, a Rwandan sister organization of the US-based nongovernmental organization Partners In Health since 2005. This study was conducted as part of the study titled “mHealth - community health worker tool for comprehensive postcesarean follow-up in rural Rwanda,” which was being conducted in the same area.

Selection of the Existing Tools and Initial Questions

After reviewing multiple existing tools to assess usability and acceptability of mHealth or digital health interventions, our team decided to move forward with 2 existing tools due to the relevant information they provided to our study. To assess usability among CHWs, we adapted the MAUQ for stand-alone mHealth apps, which was developed and validated in English by Zhou et al [11]. The original MAUQ comes in 4 versions depending on the type of app (interactive or stand-alone) and the target users (patients or health care providers). Given that our mHealth app is stand-alone and geared toward CHWs, we adapted the MAUQ for stand-alone mHealth apps used by health care providers. This version of the MAUQ includes 14 questions. For acceptability, we selected the questions most relevant to our study from the POAS, validated by O’Connor and Cranney [12]; we started with 4 questions that were deemed relevant out of the 15 from this tool. In addition, we developed and added 7 usability questions; these questions were drafted by the core study team members to reflect core functions of the mHealth screening app under development. These functions were not captured in the MAUQ or POAS questions and underwent the validation steps described in subsequent sections. To assess the reading level of our assessment tool, we intentionally designed all items to meet a below 8th‑grade reading level, following widely accepted best practices in health communication. This decision was guided by recommendations from the National Institutes of Health (NIH), which emphasize using plain language, short and clear sentences, and familiar vocabulary in survey instruments to promote comprehension and reduce respondent burden. These NIH guidelines support creating materials that are accessible across diverse literacy levels and help minimize misunderstanding or response error (NIH Plain Language Guidance [16]).

Translation and Adaptation

After ensuring its comprehensibility with a reading level equivalent to sixth grade, the CHW mHealth usability and acceptability assessment tool underwent a cross-cultural adaptation process, which consisted of translation into Kinyarwanda, back translation, and harmonization, as described by Beaton et al [17]. A study team member fluent in Kinyarwanda and English translated the adapted questionnaire from English into Kinyarwanda. After translation, an independent translator with skills in the health domain and who did not have access to the original version of the assessment questions translated the questions back into English. The study team and translator met to review and complete a harmonization process to help identify any ambiguity, errors, or confusion in the translated version and ensure that the tool was appropriate for the cultural context.

Validity and Reliability Testing

Validity testing comprised 2 assessments: content validity testing and face validity testing, whereas reliability was evaluated through internal consistency analysis.

Content Validity

Content validity assessed the relevancy and clarity of the tool among content experts. Eight content experts, including app developers, study team members, and clinicians, were conveniently asked to rate the relevancy and clarity of items from the CHW-MUAAT on a scale from 1 to 4 (1=“not relevant” or “not clear”; 4=“very relevant” or “very clear”). Content validity was established using the content validity index (CVI). The CVI was calculated using the number of experts who agreed that the item was relevant or clear (scores of 3 or 4) divided by the total number of experts. The scores for each individual item are referred to as the “item CVI.” On the basis of recommended CVI thresholds from the literature [18], items with an item CVI of >0.78 were deemed content valid and remained on the survey. Those with an item CVI score of ≤0.78 were removed.

Face Validity

Face validity was established by assessing the clarity and comprehensibility of the translated tool among target end users. In total, 10 target users (CHWs) from the Kirehe district were selected at random and asked to rate the clarity and comprehensibility of the translated items on the assessment tool on a scale from 1 to 4 (1=“not at all clear” or “not at all understandable”; 4=“very clear” or “very understandable”). The face validity index (FVI) was calculated by dividing the number of CHWs who agreed that the item was understandable (score of 3 or 4) by the total number of CHWs (n=10). On the basis of recommendations from the literature [19], an FVI above 0.74 was considered excellent, an FVI of 0.60 to 0.74 was considered good, and an FVI of 0.54 to 0.59 was considered fair. Items with an FVI of ≥0.60 were considered acceptable and kept.

Reliability Testing

After the content and face validity testing, the adapted tool was used to assess the usability and acceptability of the mHealth app among 30 CHWs. The 30 CHWs received explanations on the use of the mHealth CHW app and were asked to respond to all questions on the app during 3 sample patient vignette role-plays led by a study team facilitator. After the sample role-play, the CHWs were asked to answer questions from the CHW-MUAAT. The full results of this assessment are presented elsewhere. For this study, we used the data from reliability testing to assess the internal consistency of the tool. The internal consistency of the tool was evaluated by calculating the Cronbach α coefficient [20]. A higher α value suggests greater internal consistency reliability, with an α value greater than 0.70 deemed to indicate good reliability [21,22].

Ethical Considerations

This tool adaptation and validation study falls under a larger study titled “mHealth - community health worker tool for comprehensive postcesarean follow-up in rural Rwanda,” which received human participant ethical review approval from the Rwanda National Ethics Committee (109/RNEC/2022) and the Institutional Review Board of the Harvard Faculty of Medicine (22-1025). For the reliability testing, all CHWs provided informed consent before participating in the mHealth usability and acceptability assessments and were able to opt out of participation in the study activities. Data used in measuring validity and reliability were deidentified. Participants were compensated with 6000 Rwandan francs (approximately US $4.15) for their time and travel related to the study.

Results

The first adapted tool included 14 questions from the MAUQ, 4 POAS questions, and 7 questions added by the study team, yielding a total of 25 questions. All questions were modified for a sixth-grade reading level and underwent cross-cultural translation. The final questions for this first adaptation are presented in Table 1.

A total of 10 content experts participated in the content validity survey. In total, 88% (22/25) of the items received an item CVI score of >0.78 on both the clarity and relevance scales (Table 2). Three items had an item CVI lower than this threshold and were removed. The 3 items that were removed from the first adapted set of questions were “The app does a good job of letting me know my progress,” “The navigation is consistent between screens,” and “I can use all the functions (such as entering information, responding to reminders, and viewing information) in the app.”

All the remaining 22 items received individual FVI scores of >0.60 for clarity and relevance and, thus, were retained in the full final tool (Table 2). The FVI score for clarity across all questions was 0.991 and was the same for relevance. The full final tool is available in Table 3. The Cronbach α value was 0.86 for the 12 MAUQ usability questions, 0.73 for the 4 POAS acceptability questions, and 0.87 for the questions added by the study team, indicating the high internal consistency reliability of these questions.

Discussion

Principal Results

In this study, we adapted 2 existing tools, the MAUQ and POAS, to evaluate CHWs’ perceptions of the usability and acceptability of a novel mHealth app for comprehensive home-based follow-up care among women who have delivered via cesarean section in Rwanda. Our final tool included 22 items in total: 12 (55%) out of the 15 original items from the MAUQ tool, all 4 (18%) of the items that we pulled from the POAS, and 6 (27%) new items added by our study team. The final items had high content validity among content experts and high face validity and internal reliability among CHWs.

While there is an increasing number of mHealth apps designed to support CHWs across SSA, very few are operating at a large scale and beyond isolated research settings [2,23,24]. One of the key barriers to scale-up is the limited understanding of whether these tools are perceived as usable and acceptable by the CHWs expected to adopt them. Without sufficient usability and acceptability, even well-designed tools are unlikely to integrate effectively into CHW workflows. Although a few studies in SSA have examined these dimensions among CHWs [2,4-6], most have not used adapted or validated tools, limiting the reliability and comparability of their findings. A recent review found that 70% of researcher-developed usability questionnaires for mHealth apps were not validated—and those validated were not tailored to the community health context [25]. Validated assessment frameworks are critical to draw robust conclusions about user experience and determining the potential for mHealth tools to be well integrated into routine practice.

This quantitative CHW-MUAAT scale provides a standardized, scalable way to summarize usability and acceptability constructs, which facilitates comparison of potential digital tools in one environment or of the functionality of a single tool in different contexts [26]. However, complementary qualitative data can offer deeper insights into the contextual factors that shape CHW perceptions of usability or acceptability. For example, a mixed methods study identified that CHWs in Uganda had high rates of digital health acceptance but low digital tool use due to low smartphone ownership [27]; similarly, a study of YendaNafe, a mobile app to support CHW activities in Malawi, found an overall positive acceptance of the app but that there were several facilitators of and barriers to actual app use [28]. For this study, our qualitative results indicated general acceptability and emphasized that strong CHW-patient relationships and trust were needed for successful implementation (full results available elsewhere [29]).

To meaningfully inform the integration of an mHealth app into CHWs’ workflows, usability and acceptability assessments must be tailored to the local context and to the needs and capacities of end users. In our study, this required careful linguistic and cognitive adaptation of existing measurement tools. Given that most CHWs in Rwanda have completed only primary school education, our team revised the wording of several items to ensure that they were easily understandable. For example, the original item “There is a high probability that using this strategy may cause or result in more benefit than harm” was simplified to “I think using this mHealth app will be more helpful than hurtful” [30]. Additionally, all items were translated into Kinyarwanda, the primary language spoken by CHWs, with attention to clarity and cultural appropriateness.

Beyond linguistic adaptation, we also modified the content of existing tools to better capture usability and acceptability as related to the specific task—home-based follow-up care after cesarean delivery. This included rewording items from the MAUQ and POAS tools and adding step-specific questions relevant to CHWs’ real-world experience. For example, questions about ease of logging into the app were added given that many CHWs have limited experience with smartphones. This consideration is important in Rwanda, where, as of 2024, although 80% of households own a mobile phone, only 34% own a smartphone [31]. By demonstrating a structured approach to adapting and contextualizing usability and acceptability assessment tools, this study offers both a practical instrument and a replicable methodology for future CHW-focused mHealth evaluations in similar settings across the region.

This instrument can be adapted for other teams interested in assessing the usability and acceptability of a particular CHW mHealth tool with appropriate adaptation and validation. First, they may need to identify usability questions tailored to community health interventions, similarly to how our team added 7 usability questions specific to our postcesarean monitoring app. These questions will need to undergo reliability and validity assessments. The full tool should then be piloted in their settings, whereas smaller sample sizes may be warranted given the baseline work presented in this paper.

Limitations

We acknowledge that the adaptation and validation of the tool has some limitations that should be considered. First, this study had a small sample size of content experts and CHWs. Because of the overall high scores and consistent conclusions, we are confident in the validity of the final tool developed, but future validation studies will likely need larger sample sizes if there is more variability in participant responses. Second, this tool focuses on a quantitative assessment that cannot pick up nuanced responses. Our final evaluation includes the quantitative responses collected using this tool during the usability and acceptability testing of the app. However, we do believe that these standardized assessments can help those evaluating mHealth apps identify areas for improvement and test against prespecified targets to determine whether the mHealth app is ready for broader use. Finally, this tool was designed for a specific intervention—a mobile phone app developed for CHWs to use during the home visit follow-up of women who delivered via cesarean section—and was only validated in the Kinyarwanda language. Other research groups should proceed cautiously and likely conduct their own validation studies when applying this tool to a different intervention or in a different language.

Conclusions

Our team is using the final tool as a core component of evaluating an mHealth app used by CHWs to support women after cesarean delivery. The processes and tool described in this paper serve as a model for others conducting CHW usability and acceptability assessments of mHealth apps in SSA. Such assessments will be critical for moving toward better integration of innovative solutions into CHW activities.