Audio recordings from the interviews were transcribed verbatim and handwritten notes were compiled without participant identifiers. Anonymized transcripts were analyzed using a deductive coding process [36]. One researcher (AS) conducted the initial coding, categorizing participant feedback based on the role pMARS played in assessing the app. Preliminary codes and emerging themes were discussed by the researcher and a collaborator until consensus was reached. The final coding scheme was subsequently reviewed and validated by a second researcher (AADNSS). Findings from this phase were used to refine item wording and enhance the content validity of the pMARS tool.

This research project received an ethics review exemption from the Projects Committee of the Department of Laboratory Medicine at Ng Teng Fong General Hospital, Singapore. The exemption was granted in accordance with the department’s policy for minimal-risk research involving health care staff and was reviewed and approved by the head of the department. All participants were provided with an information sheet describing the study objectives, the voluntary nature of participation, and the commitment to data confidentiality. Informed consent was obtained before participation in each study phase. Participants who completed the qualitative interviews (phase 1) or paper-based questionnaires (phase 2) provided written consent, while those who completed the online survey via REDCap on the hospital intranet platform provided electronic consent.

Basic demographic data, including age, biological sex, and professional role, were collected. No personally identifiable data were collected. All files were stored on password-protected institutional drives accessible only to study investigators. In phase 2, participants were offered a Singapore $10 (S$1=US $0.78 as of June 17, 2026) voucher for a local coffee chain as compensation for their time spent completing the questionnaire. No other payments or incentives were provided.

The first version of the questionnaire consisted of a demographic data section and 5 domains. Certain question stems were reworded and/or split into 2 separate stems to ensure that there were no double-barreled questions, for example, item 1 of section A of MARS reads, “Is the app fun/entertaining to use? Does it use any strategies to increase engagement through entertainment?” This was broken into two separate question stems: (1) “Is the app fun/entertaining to use?” and (2) “Does the app use strategies to increase engagement through entertainment (e.g., strategies such as interactivity/gamification)?” This version of the questionnaire consisted of question stems presented in question form and shortened but descriptive answer stems presented as a Likert scale. Moreover, almost all the question stems had examples or descriptive statements to further elaborate or clarify the ideas.

Insights from the qualitative assessment, categorized into 4 key areas, guided the multidisciplinary panel in developing the second version of pMARS at the end of phase 1 (Table 2).

Findings from the qualitative assessment of the questionnaire (ie, interviews with HCPs) shed light on several areas of the questionnaire for focused improvement: (1) shorten the questionnaire by reducing the length of the answers, (2) further simplify the language and minimize jargon, and (3) introduce an additional option for those who do not know the answer to a specific question. It is noteworthy that no new items were proposed by the HCPs. Based on the findings, we changed the question stems from question forms to statements, for example, (1) “arrangement and size of buttons, icons, and menus on the screen is appropriate” and (2) “it uses strategies to increase engagement through entertainment (e.g., strategies such as interactivity/gamification).” Moreover, we limited examples and/or descriptive statements in question stems to a minimum. Simultaneously, we changed the answer stems to a simple 5-point Likert scale without embedded descriptive statements: strongly agree, agree, neutral, disagree, and strongly disagree. Finally, we added a new answer stem: “don’t know.”

In addition, we redesigned the questionnaire by changing its orientation from portrait to landscape. The reasons for this change were to limit the length of the questionnaire to 3 pages despite including additional questions and to provide an easy-to-complete format. The final version of the questionnaire at the end of phase 1 consisted of 26 questions spread across 5 domains: engagement (6 items), functionality (5 items), aesthetics (4 items), information (7 items), and subjective quality (4 items). The length of the questionnaire was deemed appropriate by the committee (Multimedia Appendix 2).

We distributed 197 printed questionnaires among HCPs in person, and 189 participated, resulting in a 95.9% response rate. Additionally, 29 HCPs accessed the survey through the hospital intranet, bringing the total number of responses to 218. A further 15 REDCap surveys were incomplete, resulting in a REDCap complete response rate of 65.9%. We were unable to calculate an overall response rate for REDCap as we could not track the reach through the hospital intranet. Among the in-person and intranet-administered questionnaires returned, the proportion of nonresponses per question item ranged from 0% to 13.3%, with the item “Would you pay for this app?” showing the highest nonresponse rate (n=29). Most items (29/30; 96.7%) had less than 5.1% missing responses. Detailed item-level missing data, including counts of “don’t know” responses, are provided in Multimedia Appendix 3.

The Cronbach α (ranging from 0.855 to 0.931) showed that the internal consistency reliability of pMARS was good to excellent for all the domains (Table 3). SEM analysis demonstrated that functionality (P<.001) had the strongest influence on end-user willingness to use or recommend and purchase the MHA, after adjusting for age, sex, and the HCPs’ role in the hospital (Figure 3).

For domains with Cronbach α values >0.90, we conducted a manual item review to assess potential redundancy. We examined item wording and conceptual distinctness within each domain. The review confirmed that, although some items were related, they measured complementary aspects of the construct and contributed meaningfully to the overall scale. All such items were therefore retained in the current version of pMARS.

IRT (Table 4) identified that for the engagement domain, items 4 (customization) and 5 (interactivity) had a weak impact on latent constructs, whereas items 1 and 2 (both entertainment) had a higher impact on the domain. Similarly, items 9 (ease of use) and 11 (gestural design) had a weak impact on functionality, item 13 (arrangement and size of content) had a strong impact on aesthetics, and items 18 (quantity) and 19 (quality) had a strong impact on information. The engagement domain demonstrated the lowest reliability, and IRT highlighted weakly impacting items within the domain, enabling the researchers to shorten the questionnaire if required.

Our study shows that the newly developed pMARS tool has strong internal consistency reliability (Cronbach ɑ) across all domains, indicating its potential utility in assessing the quality of MHAs designed for HCPs from their perspective. Although most items performed well, item 22 of the information domain (“the app has been trialled/tested; verified by evidence”) demonstrated weak correlations (Table 3). The item was retained in the final version of pMARS as it reflects a critical aspect of evidence-based practice in clinical settings and warrants further examination in future validation studies. Furthermore, SEM generated valuable insights into interrelationships between the latent constructs. Not all items were equally good indicators for the dimensions. The strongest influence was observed between subjective quality and functionality. Similarly, items 1 to 3 of the engagement domain, items 16 to 22 of the information domain, and items 23 and 26 of the subjective quality domain demonstrated relatively strong influence.

On the other hand, IRT revealed weakly impacting items such as items 4 and 5 of the engagement domain; items 9 and 11 of the functionality domain; and items 16, 17, and 20 of the information domain. These items provide an opportunity to further shorten the questionnaire. However, it should be evaluated within the theoretical factor model of the scale, and therefore, this is an area for future research. Overall, our findings suggest that pMARS is an MHA quality assessment tool of good metric quality considering the reliability and construct validity results.

In the busy health care settings of Singapore, engaging HCPs in discussions unrelated to direct patient care can be challenging. With its brevity and standardized format, pMARS enables rapid assessment of MHA quality from the perspective of HCPs as clinical end users. Unlike MARS, which was designed for expert evaluators to classify and assess general MHA quality, and uMARS, which targets general end users, pMARS fills a critical gap by capturing HCP-specific clinical usability and professional decision-making considerations that are not specifically addressed by existing tools [10,21]. The use of simplified language in pMARS ensures accessibility for HCPs without requiring specific training, including those who may be unfamiliar with MHA-related terminology. Consequently, pMARS provides a practical, contextually relevant, and adaptable tool for service providers and HCPs to evaluate the quality of MHAs within real-world clinical settings. Notably, pMARS is designed as a flexible framework that can be iteratively refined to incorporate emerging clinical-, workflow-, and patient safety–related quality considerations as digital health technologies evolve.

The direct use of MARS in developing pMARS, as described by Baptista et al [27], may eliminate certain aspects of MHAs that are important to end users. To overcome this limitation, we introduced a qualitative approach using interviews with HCPs; however, the inputs generated by this procedure were limited to a few aspects of the questionnaire, as discussed in the Results section. The qualitative component introduced changes to question stems by rewording, shortening, and simplifying them without adding or removing any items from the questionnaire. According to our observations during the interviews, the interview answers were brief and the discussions were limited, possibly as a result of time pressures.

Legislative frameworks such as the Health Insurance Portability and Accountability Act (HIPAA) in the United States and the Personal Data Protection Act (PDPA) in Singapore address crucial concerns for HCPs regarding MHAs, including data privacy, security, and patient confidentiality [42,43]. These regulations set important standards for protecting patient information and ensuring the reliability of MHAs. However, the adoption of such regulations varies globally, with many countries still developing or implementing their standards. As regulatory frameworks continue to evolve, the availability of a robust tool to evaluate the quality of MHAs from the perspective of HCPs remains essential to ensure these apps meet the necessary standards. Furthermore, health care organizations can use pMARS to evaluate and select suitable MHAs that comply with professional standards.

As mHealth technology continues to evolve, several areas for potential improvement in future versions of the pMARS tool have emerged. First, interoperability between MHAs, health systems, and electronic health records is crucial for HCPs and health care organizations when integrating these technologies [44]. Second, evaluating the effectiveness and security of communications among HCPs, including end-to-end encryption, is important for ensuring coordinated care and protecting sensitive information [45]. Third, understanding potential patient safety concerns associated with MHAs is vital for maintaining high standards of care. Lastly, language barriers among HCPs should be addressed to enhance the app’s quality, ethical considerations, and utility. Future versions of pMARS should take these evolving quality requirements into account. Additionally, with the growing integration of AI and large language models into mHealth apps, future versions of pMARS should also account for quality requirements specific to AI and AI-generated content [46,47].

Our study has several methodological and participant-related limitations. There were 2 methodological limitations. First, the psychometric evaluation of pMARS was based on data from only 1 MHA. This focus may affect certain aspects of statistical analysis, particularly if the MHA had quality deficiencies in specific domains. To address this limitation, future validation studies should involve a broader range of MHAs across different medical fields. Second, the IRT depends on the accuracy of the underlying model. The utility of the IRT is contingent upon how well the model reflects the data, making further validation studies essential to evaluate the appropriateness of pMARS items.

Participant-related limitations include the potential influence of the questionnaire’s ease of administration on construct validity. The simplicity and repetitive nature of the pMARS may lead to patterned responses, highlighting the need for negatively worded questions in future studies to counteract response biases. Second, the expert panel in phase 1, comprising a physician, public health expert, and human-computer interactions specialist, did not fully represent the demographics of the study participants or potential users, potentially limiting the generalizability of the pMARS tool. Further studies are necessary to assess and validate the pMARS tool across multiple MHAs. Finally, the disproportionate number of female participants should be noted as a limitation, as it may reduce the generalizability of findings to male HCPs. However, to some degree, this distribution reflects the gender composition of the hospital workforce rather than sampling bias.

To our knowledge, this study is the first to develop a tool to evaluate MHAs designed for HCPs from their perspective. Additionally, it is the first study to apply IRT to assess and identify the impact of each question on the latent constructs within MARS and its derivatives. Our findings demonstrate that pMARS is a robust and metrically sound tool for evaluating MHAs intended for HCPs. It has the potential to enable HCPs to assess and recommend high-quality MHAs. We recommend that future research on pMARS incorporate IRT to refine the questionnaire by identifying and removing less-effective items, thereby enhancing its utility and precision in assessing MHA quality.