Letter to the Editor
We read with interest the article, “Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS)," by Dr Stoyanov et al . The authors report on the development and validation of the user version of the Mobile Application Rating Scale (uMARS). If applied appropriately, this scale has the potential to improve the quality and standardization of reporting, and assist in the progress of the scientific evidence base in mHealth research. The uMARS is based on the original ‘expert’ Mobile Application Rating scale (MARS), which consists of items identified from a literature search of web and mobile application (app) quality rating criteria, and that were tested for reliability using 60 wellbeing apps from the iTunes store [ ]. The authors report on how the MARS was adapted into the uMARS for lay users by simplifying items and removing those that require professional or content expertise. Both scales are similar in structure, and include objective quality subscales: engagement, functionality, aesthetics and information quality, as well as a subjective quality subscale.
A reliable measure of app quality for end-users is urgently needed as health apps continue to proliferate without rigorous evaluation . Although there have been notable attempts to measure the quality of mHealth apps, there is as yet no widely accepted standardized method for end-users [ - ]. This limits the ability to identify and scale-up successful apps, thereby limiting the population impact of mHealth [ , ].
While there is no doubt that the development of the MARS and uMARS represent significant progress towards greater consistency and transparency in mHealth, we would argue that it would be further strengthened by both a clear conceptual definition of app quality as well as a theoretical framework for testing app quality. To that end, we propose considering how the perception of app quality differs between experts (health professionals, researchers, app developers) and lay/end-users. For example, a health professional assesses the quality of a health app in order to identify apps to recommend to their patients. On the other hand, an end-user will assess the quality of a health app with the intent to use it. Indeed, evidence suggests that health professionals consider clinical effectiveness of mHealth apps to be most important, whereas consumers are looking for a seamless user experience, reduced data entry burden, and integration with their health care experience [, ]. An important implication of these differences is that the simple adaptation of ‘expert’ items of the MARS may not accurately and adequately reflect the ‘end-user’ perspective on app quality. More research is needed to understand how expert and end-user perspectives on app quality differ in terms of their expectations, needs, preferences and attitudes.
Secondly, the authors have used current literature to inform the development of the MARS and the uMARS. However, because the field is in its infancy, the development of an mHealth quality scale cannot rely solely on this inductive approach as there is much we do not know about how mHealth apps are accessed and evaluated by users. We suggest that the authors examine the substantial body of theoretical research on usability and engagement with technology to improve the comprehensiveness of the uMARS.
We also recommend more research on how the perception of quality varies with respect to these technologies, especially from the user perspective. For example, there are clear distinctions between the functionality, programmability, interactivity and features of internet-based interventions and mobile apps. While web-based interventions imply that a user accesses information via the Internet from any connected device, mHealth involves a complex relationship between a user and their portable, personal mobile device . Additionally, mobile apps offer greater functionality, personalisation and real-time interactivity. Thus, although there are many similarities between web and mobile app interventions, more research is needed to inform how effectively items describing the quality of a web based intervention translate into items measuring mobile app quality.
Finally, the MARS and uMARS are relatively new advances in mHealth, and to date the MARS has been applied to the review of a handful of apps, including mindfulness apps , heart failure symptom monitoring apps, self-care management apps [ ], and weight management apps [ ]. Although there is an urgent need for consistent and transparent evaluation and reporting of app quality, especially from the user perspective, much more research is needed to validate the uMARS before it can be widely adopted and included into standardised mHealth reporting guidelines such as mERA and CONSORT-EHEALTH [ , ]. Further reducing the length and complexity of the response options of the uMARS will make it easier for researchers and others to apply the scale in their research.
Shaira Baptista is supported by a National Health and Medical Research Council post-graduate scholarship (ID GNT1133316) and a PhD top- up scholarship from Diabetes Australia.
This work is supported by a grant from the National Health and Medical Research Council (ID 1057411).
Dr Adrienne O’Neil is supported by a Future Leader Fellowship (ID 101160) from the Heart Foundation, Australia.
Conflicts of Interest
- Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016 Jun 10;4(2):e72 [FREE Full text] [CrossRef] [Medline]
- Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps. JMIR mHealth and uHealth. JMIR Publications Inc; 2015:A.
- O'Neil A, Cocker F, Rarau P, Baptista S, Cassimatis M, Taylor B. Using digital interventions to improve the cardio-metabolic health of populations: a meta review of reporting quality (Accepted). Journal of the American Medical Informatics Association 2016:a.
- Powell AC, Torous J, Chan S, Raynor GS, Shwarts E, Shanahan M. Interrater Reliability of mHealth App Rating Measures: Analysis of Top Depression and Smoking Cessation Apps. JMIR mHealth and uHealth. JMIR Publications Inc; 2016:A.
- BinDhim NF, Hawkey A, Trevena L. A systematic review of quality assessment methods for smartphone health apps. Telemedicine and e-Health. Mary Ann Liebert, Inc. 140 Huguenot Street, 3rd Floor New Rochelle, NY 10801 USA; 2015;21(2):104.
- Yasini M, Marchand G. Mobile Health Applications, in the Absence of an Authentic Regulation, Does the Usability Score Correlate with a Better Medical Reliability? In: MEDINFO. 2015 Dec Presented at: Proceedings of the 15th World Congress on Health and Biomedical Informatics; 2015 p. E.
- Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling up mHealth: where is the evidence? PLoS Med 2013;10(2):e1001382 [FREE Full text] [CrossRef] [Medline]
- Hingle M, Patrick H. There Are Thousands of Apps for That: Navigating Mobile Technology for Nutrition Education and Behavior. Journal of nutrition education and behavior. Elsevier; 2016:213.
- Krebs P, Duncan DT. Health app use among US mobile phone owners: a national survey. JMIR mHealth and uHealth. JMIR Publications Inc; 2015:a.
- Davis TL, DiClemente R, Prietula M. Taking mHealth Forwardxamining the Core Characteristics. JMIR mHealth and uHealth. JMIR Publications Inc., Toronto, Canada 2016;4(3):E.
- Mani M, Kavanagh DJ, Hides L, Stoyanov SR. Review and evaluation of mindfulness-based iPhone apps. JMIR mHealth and uHealth. JMIR Publications Inc 2015;3(3).
- Creber RMM, Maurer MS, Reading M, Hiraldo G, Hickey KT, Iribarren S. Review and Analysis of Existing Mobile Phone Apps to Support Heart Failure Symptom Monitoring and Self-Care Management Using the Mobile Application Rating Scale (MARS). JMIR mHealth and uHealth. JMIR Publications Inc., Toronto, Canada 2016;4(2):e74.
- Bardus M, van BSB, Smith JR, Abraham C. A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management. International Journal of Behavioral Nutrition and Physical Activity. BioMed Central 2016;13(1):1.
- Eysenbach G, CONSORT- E. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011 Dec 31;13(4):e126 [FREE Full text] [CrossRef] [Medline]
- Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. bmj. British Medical Journal Publishing Group; 2016.
Edited by G Eysenbach; submitted 27.07.16; peer-reviewed by S Beidas, A Paglialonga, M Hart, WC Su, TR Soron; comments to author 12.09.16; revised version received 05.12.16; accepted 28.01.17; published 09.06.17Copyright
©Shaira Baptista, Brian Oldenburg, Adrienne O'Neil. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 09.06.2017.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.