This paper is in the following e-collection/theme issue:

Quality Evaluation and Descriptive Analysis/Reviews of Multiple Existing Mobile Apps (362) mHealth for Wellness, Behavior Change and Prevention (2681) Mobile Health (mhealth) (2679) Mobile Health in Psychiatry (392)

Published on in Vol 10, No 10 (2022): October

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/41282, first published .
Mental Health Mobile Apps in the French App Store: Assessment Study of Functionality and Quality

Mental Health Mobile Apps in the French App Store: Assessment Study of Functionality and Quality

Mental Health Mobile Apps in the French App Store: Assessment Study of Functionality and Quality

Authors of this article:

Florence Carrouel1 Author Orcid Image ;   Benjamin du Sartz de Vigneulles1 Author Orcid Image ;   Denis Bourgeois1, 2 Author Orcid Image ;   Bernard Kabuth3, 4 Author Orcid Image ;   Nicolas Baltenneck5 Author Orcid Image ;   Fanny Nusbaum1 Author Orcid Image ;   Valérie Burge3, 4 Author Orcid Image ;   Sylvain Roy1 Author Orcid Image ;   Sophie Buchheit3, 4 Author Orcid Image ;   Marie-Line Carrion-Martinaud6 Author Orcid Image ;   Catherine Massoubre7 Author Orcid Image ;   Laurie Fraticelli1 Author Orcid Image ;   Claude Dussart1, 2 Author Orcid Image
Article Authors Cited by (5) Tweetations (4) Metrics

    Original Paper

    1Health Systemic Process, Research Unit UR4129, University Claude Bernard Lyon 1, University of Lyon, Lyon, France

    2Hospices Civils de Lyon, Lyon, France

    3Prisme Team, Interpsy Laboratory, Research Unit EA4432, University of Lorraine, Nancy, France

    4Nancy Psychotherapeutic Center, Laxou, France

    5Development, Individual, Process, Disability, University Lyon 2, Lyon, France

    6The Constellation, Health and Wellness Center, Rive-de-Gier, France

    7Department of Psychiatry, Research Unit EA7423, Saint-Etienne University Hospital Center of Saint Etienne, University Jean Monnet, Saint Etienne, France

    *these authors contributed equally

    Corresponding Author:

    Florence Carrouel, PhD

    Health Systemic Process, Research Unit UR4129

    University Claude Bernard Lyon 1

    University of Lyon

    7 rue Guillaume Paradin

    Lyon, 69372

    France

    Phone: 33 0478785745

    Email: florence.carrouel@univ-lyon1.fr


    Background: Approximately 800 million people, representing 11% of the world’s population, are affected by mental health problems. The COVID-19 pandemic exacerbated problems and triggered a decline in well-being, with drastic increase in the incidence of conditions such as anxiety, depression, and stress. Approximately 20,000 mental health apps are listed in mobile app stores. However, no significant evaluation of mental health apps in French, spoken by approximately 300 million people, has been identified in the literature yet.

    Objective: This study aims to review the mental health mobile apps currently available on the French Apple App Store and Google Play Store and to evaluate their quality using Mobile App Rating Scale–French (MARS-F).

    Methods: Screening of mental health apps was conducted from June 10, 2022, to June 17, 2022, on the French Apple App Store and Google Play Store. A shortlist of 12 apps was identified using the criteria of selection and assessed using MARS-F by 9 mental health professionals. Intraclass correlation was used to evaluate interrater agreement. Mean (SD) scores and their distributions for each section and item were calculated.

    Results: The highest scores for MARS-F quality were obtained by Soutien psy avec Mon Sherpa (mean 3.85, SD 0.48), Evoluno (mean 3.54, SD 0.72), and Teale (mean 3.53, SD 0.87). Mean engagement scores (section A) ranged from 2.33 (SD 0.69) for Reflexe reussite to 3.80 (SD 0.61) for Soutien psy avec Mon Sherpa. Mean aesthetics scores (section C) ranged from 2.52 (SD 0.62) for Mental Booster to 3.89 (SD 0.69) for Soutien psy avec Mon Sherpa. Mean information scores (section D) ranged from 2.00 (SD 0.75) for Mental Booster to 3.46 (SD 0.77) for Soutien psy avec Mon Sherpa. Mean Mobile App Rating Scale subjective quality (section E) score varied from 1.22 (SD 0.26) for VOS – journal de l’humeur to 2.69 (SD 0.84) for Soutien psy avec Mon Sherpa. Mean app specificity (section F) score varied from 1.56 (SD 0.97) for Mental Booster to 3.31 (SD 1.22) for Evoluno. For all the mental health apps studied, except Soutien psy avec Mon Sherpa (11/12, 92%), the subjective quality score was always lower than the app specificity score, which was always lower than the MARS-F quality score, and that was lower than the rating score from the iPhone Operating System or Android app stores.

    Conclusions: Mental health professionals assessed that, despite the lack of scientific evidence, the mental health mobile apps available on the French Apple App Store and Google Play Store were of good quality. However, they are reluctant to use them in their professional practice. Additional investigations are needed to assess their compliance with recommendations and their long-term impact on users.

    JMIR Mhealth Uhealth 2022;10(10):e41282

    doi:10.2196/41282

    Keywords

    mobile apps; behavior change; mental; prevention; mobile health; mHealth; lifestyle; French; well-being 


    Approximately 800 million people, representing 11% of the world’s population, are affected by mental health problems [1]. In addition, the COVID-19 pandemic exacerbated problems and triggered a decline in well-being, with a drastic increase in the incidence of conditions such as anxiety, depression, and stress [2]. However, within the World Health Organization’s classification of mental, behavioral, and neurodevelopmental disorders (International Classification of Diseases–11), a broad spectrum of mental conditions exists, varying from mild, one-time disorders to severe, chronic, and disabling disorders [3]. Similarly, the mental well-being concept encompasses more than the absence of mental health disorders and symptoms and may include psychological parameters such as subjective autonomy, well-being, personal fulfillment, and positive relationships [4]. There is growing recognition that mental health and mental well-being are distinct entities with specific determinants [5]. Preserving positive identity, maintaining good self-esteem, being able to control and adapt to one’s own life, and preventing social isolation and solitude are all positive aspects of well-being that can contribute to optimizing autonomy [6,7].

    Mental disorders are a real public health problem [8]. Each year, approximately 20% of adults are affected by anxiety disorders [9]. Depression affects >300 million people worldwide [10,11]. In France, the mental health of French adults deteriorated between 2015 and 2020, partly because of the COVID-19 crisis, as indicated by a longitudinal study of adults [12]. French adults were more affected by depressive or anxiety symptoms. In addition, in recent years, the number of workers reporting complaints of severe stress has increased. Thus, stress is one of the most common occupational health problems. In Europe, the prevalence of men and women reporting work-related stress always or most of the time is 26% and 27%, respectively [13]. Stress at an early stage of working life can contribute to burnout, depression, and unfavorable work outcomes in later life, depending on the life course perspective [14,15]. Various psychosocial interventions have been suggested to overcome stress, such as relaxation, mindfulness practices, or social engagement, ideally performed in groups [16,17].

    New technologies have the potential to overcome these challenges by providing large-scale health literacy programs, low-threshold approaches [18], or education for the population and health care workers [19]. In this sense, disruptive technologies provide a fantastic opportunity for enhancing mental health approaches [20]. Beyond supporting people with mental disorders, apps can also be used to improve their overall well-being by encouraging behavior changes, including the practice of meditation and mindfulness. Mobile apps related to mental health are known as mental health apps and vary in type and number [21]. In 2022, global spending on mental health mobile apps is estimated, not including China, to be approximately US $500 million [22]. Currently, there are reportedly up to 20,000 mental health apps [22].

    Growing digital device use for mental health support also suggests mental health apps as a possible pertinent aspect for a proactive mental health and wellness model in the coming years [23]. Apps that capture the user’s moods or emotional conditions and then deliver information have significant and positive support, as do apps offering regular, short meditation sessions [24]. Mental health apps promote positive mental health and well-being, involving a decrease in symptoms of mental health conditions such as anxiety, stress, and depression. Mental health apps have the potential to be effective in improving the symptoms of certain mental disorders [25] and to improve well-being [26] and life satisfaction, with a more effective emotional management [21,24]. During the first 10 months of 2020, consumers spent a record of US $1.1 billion on wellness apps worldwide [22].

    Nevertheless, no exploratory studies, which identify the development and application trends of evidence-based apps on topics such as outcomes of mental health or well-being apps, are currently available [25]. Further comprehensive validation and evaluation of these apps from a clinical perspective is needed [27].

    The study aimed to review the mental health mobile apps currently available on the French Apple App Store and Google Play Store and to evaluate their quality using Mobile App Rating Scale–French (MARS-F).


    Selection of the French Mobile Health Apps

    Mental health–related apps were screened in the French App Store (iPhone Operating System [iOS]) and in the French Google Play Store (Android) from June 10, 2022, to June 17, 2022, by 2 academic researchers. The following search terms were used: “bien être mental” (mental well-being), “santé mentale” (mental health), and “bien être” (well-being). Each search term was introduced separately in the Apple App Store and Google Play Store because no truncation or use of logic operators (AND, OR, and NOT) was possible.

    Each researcher eliminated duplicate apps by cross-checking the names of apps and developers. Then, both researchers checked that they had the same list of apps and downloaded them. They checked the following inclusion criteria: (1) mainly in the French language, (2) mental well-being as subject matter, (3) targeting adult users, and (4) self-personalized programs. The exclusion criteria were apps (1) focusing on content unrelated to mental health services, such as mental training, yoga, physical activity, and nutrition; (2) targeting people with specific disorders such as suicidal tendencies, eating disorders, or addiction; and (3) providing a single function only.

    Selection of Mental Health Professionals

    The inclusion criteria for raters were the following: (1) mental health professional and (2) practicing at a hospital or performing private clinical activity in France. The exclusion criteria were (1) not having a mobile phone; (2) not being able to download apps from the Apple or Google stores; (3) never having used a mobile app; and (4) having hearing, visual, or motor disabilities.

    Selection of a Standardized Rating Scale for Mobile Apps

    MARS-F [28] was used in this study. The first part of this scale, named App classification, includes the main characteristics of the app, such as the name, version, developer, focus or target, theoretical background or strategies, age group, and so on. This part was reviewed by the 2 academic researchers. The Mobile App Rating Scale (MARS) scale is composed of a main part (23 items organized into 5 sections, named A, B, C, D, and E) and an additional part (section F with 6 items).

    Section A (engagement section; 5 items) determines whether the app is interesting, fun, customizable, and interactive (sends alerts, feedback, reminders, and messages and allows sharing). Section B (functionality section; 4 items) analyzes the app operation, ease of learning, flow logic, navigation, and gestural design of the app. Section C (aesthetics section; 3 items) focuses on the graphic design of the app, color palette, overall visual appeal, and stylistic consistency. Section D (information quality; 7 items) assesses whether the app contains high-quality information (feedback, text, references, and measurements) from a credible source. Section E (subjective section; 4 items) determines the interest of the user in the app. Section F (mobile app specificities; 6 items) analyzes the point of view of mental health professionals regarding the effect of selected apps on knowledge, possible changes in user attitudes and intentions to change, and probability of changing the identified targeted behaviors. In our study, the targeted health behavior was mental well-being.

    Each item was rated on a 5-point Likert scale (1=strongly disagree to 5=strongly agree). The score for each section was obtained by calculating the mean score of the items. The mean of section scores (sections A, B, C, and D) corresponds to the overall quality MARS score. The mean score of section E corresponds to the subjective quality score, and the mean score of section F evaluates the specificities of the app. The scores range from a minimum of 1 (poor quality) to a maximum of 5 (high quality).

    Methodology of Evaluation

    Training of the Raters

    The raters were 9 mental health professionals (Multimedia Appendix 1). Before rating the mental well-being apps, the raters had to train in the use of MARS-F. For this, they viewed a training video in French (available on request to the corresponding author), which was developed for MARS-F [28] and adapted from the English training video of Stoyanov et al [29]. In this video, each item and answer is explained based on examples. At the end of the video, a training exercise with an app not included in the sample of the study was proposed. The raters downloaded the app, tested it for at least 10 minutes, completed MARS-F, and then compared their results with those in the video. When the individual score for an item differed by >2 points, the raters discussed until a consensus was reached to ensure that they had the same understanding.

    Evaluation of the Selected Apps by the Raters

    The mental health apps were evaluated by the 9 mental health professionals during the month of July 2022. They downloaded all the included apps, used each app for at least 10 minutes, and immediately evaluated the app using a web-based MARS-F questionnaire.

    Statistical Analysis

    To assess interrater reliability, the intraclass correlations (ICCs; 2-way random, average measures and absolute concordance) were calculated [30,31]. For each item, each section, and MARS-F quality score (sections A, B, C, and D), the 95% CIs were calculated. On the basis of the 95% CI of the ICC estimate, values <0.5, between 0.5 and 0.75, between 0.75 and 0.9, and >0.90 are indicative of poor, moderate, good, and excellent reliability, respectively [31]. Mean values and SDs were calculated for each item and each mental health app section (presented as mean [SD]). Owing to missing values, item 19 was excluded from all analyses, and the mean for section D was adjusted accordingly.

    To assess the differences between the quality of the apps, by item and by section, box plots were generated. The points that appear on either side of the boxes are calculated using 1.5 times the IQR (the distance between the first and third quartiles) to highlight the extreme values.

    To provide an overview of the average scores for each item (row) and each app (column), a heat map was constructed. The color gradients represent whether the score is low (near 1 [yellow]) or high (near 5 [green]).

    To assess the correlation between average quality and subjective item 23 (“What is your overall star rating of the app?”), the Pearson coefficient (r) was calculated. To provide a complete overview of the popularity of each mobile app and the number of reviewers, the number of stars awarded by users in the iOS and Android stores was reported.

    Statistical analyses were performed using R, with the dplyr, psych, and ggplot2 packages from the R Project for Statistical Computing (version 4.1.1; R Foundation for Statistical Computing).


    Selection of Mental Health Mobile Apps

    The use of keywords allowed the identification of 35.51% (408/1149) of apps from Apple App Store and 64.49% (741/1149) of apps from Google Play Store (Figure 1). Once the 2 lists were cross-checked according to the name of the app and developer, 59.87% (688/1149) of the apps were common to both systems. After a thorough review of the remaining apps’ download page and application of the inclusion and exclusion criteria, 1.04% (12/1149) of the apps were finally selected.

    ‎
    Figure 1. Flowchart of mental health mobile apps selection. iOS: iPhone Operating System.
    View this figure

    Characteristics of Mental Health Mobile Apps

    Descriptive and technical information regarding the mobile mental health apps is provided in Tables S1-S6 in Multimedia Appendix 2. None of these apps had the same developer. An app had a different name depending on the store (VOS – journal de l’humeur in the Apple App Store and VOS – journal intime in the Google Play Store), whereas all the other apps had the same name. Of the 12 apps, 1 (8%) was fully free of charge, 8 (67%) were free with in-app purchases, and 3 (25%) were accessible exclusively with a company code. The number of downloads depends on the app. According to Google Play Store (as the data were not available for the Apple store), Sanvello was the most downloaded app, followed by VOS – journal de l’humeur.

    Table 1 presents the characteristics of the mental health mobile apps. All the apps (12/12, 100%) targeted increasing happiness or well-being; mindfulness, meditation, or relaxation; negative emotion reduction; anxiety or stress; anger; behavior change; and goal setting, whereas only 17% (2/12) of the apps targeted alcoholism or substance use, 17% (2/12) focused on relationships, and another 17% (2/12) aimed at physical health. The theoretical background and strategies of all apps were the following: assessment; information or education; monitoring or tracking; goal setting; advice, tips, strategies, or skills training; cognitive behavioral therapy (CBT)—behavioral (positive events); CBT—cognitive (thought challenging); acceptance commitment therapy; and mindfulness or meditation. Most mental health apps (11/12, 92%) also used monitoring or tracking as strategies. The apps were designed for young adults (12/12, 100%) and adults (12/12, 100%) but not for adolescents and children (aged <12 years). All the mental health apps (12/12, 100%) allowed password protection, required log-in, sent reminders, and needed web access to function, and 67% (10/15) of the apps required web access to function. Of the 12 apps, 6 (50%) allowed sharing (Facebook, Twitter, etc) and 5 (42%) had an app community.

    Table 1. Characteristics of the mental health mobile apps (N=12).
    CharacteristicsApps, n (%)
    Focus—what the app targetsa

    		Increase happiness or well-being12 (100)

    		Mindfulness, meditation, or relaxation12 (100)

    		Reduce negative emotions12 (100)

    		Anxiety or stress12 (100)

    		Anger12 (100)

    		Behavior change12 (100)

    		Alcohol or substance use2 (17)

    		Goal setting12 (100)

    		Relationships2 (17)

    		Physical health2 (17)
    Theoretical background or strategiesa

    		Assessment12 (100)

    		Information or education12 (100)

    		Monitoring or tracking11 (92)

    		Goal setting12 (100)

    		Advice, tips, strategies, or skills training12 (100)

    		CBTb—behavioral (positive events)12 (100)

    		CBT—cognitive (thought challenging)12 (100)

    		Acceptance commitment therapy12 (100)

    		Mindfulness or meditation12 (100)

    		Relaxation12 (100)
    Age groupa

    		Children (<12 years)0 (0)

    		Adolescents (13-17 years)0 (0)

    		Young adults (18-25 years)12 (100)

    		Adults (>25 years)12 (100)
    Technical aspects of the appa

    		Allows sharing (Facebook, Twitter, etc)6 (50)

    		Has an app community5 (42)

    		Allows password protection12 (100)

    		Requires log-in12 (100)

    		Sends reminders12 (100)

    		Needs web access to function12 (100)

    aParticipants could choose several answers.

    bCBT: cognitive behavioral therapy.

    Reliability of the Evaluation

    The ICC, calculated based on the 12 apps, was considered as good, with 0.79 (95% CI 0.73-0.83) for the A, B, C, and D sections. The ICC per section was good for section A (0.68, 95% CI 0.54-0.79), section B (0.62, 95% CI 0.44-0.76), section C (0.63, 95% CI 0.42-0.78), and section D (0.81, 95% CI 0.73-0.87).

    Assessing the Quality of Content of Mental Health Mobile Apps

    MARS-F quality score for each section (sections A, B, C, and D) and each app are presented in Figure 2 and Tables S1 and S2 in Multimedia Appendix 3. Mean engagement scores (section A) ranged from 2.33 (SD 0.69) for Reflexe reussite to 3.80 (SD 0.61) for Soutien psy avec Mon Sherpa. Mean functionality scores (section B) ranged from 2.89 (SD 0.63) for VOS – journal de l’humeur to 4.47 (SD 0.53) for Soutien psy avec Mon Sherpa. Mean esthetics scores (section C) ranged from 2.52 (SD 0.62) for Mental Booster to 3.89 (SD 0.69) for Soutien psy avec Mon Sherpa. Mean information scores (section D) ranged from 2 (SD 0.75) for Mental Booster to 3.46 (SD 0.77) for Soutien psy avec Mon Sherpa. For all mental health apps, except Sanvello (11/12, 92%), the mean score for functionality was consistently higher than that for the other sections.

    MARS-F quality score (sections A, B, C, and D) and MARS-F subjective quality scores (section E) for each app are presented in Figure 3 and Table S1 in Multimedia Appendix 3. The best MARS-F quality scores were obtained by Soutien psy avec Mon Sherpa (mean 3.85, SD 0.48), Evoluno (mean 3.54, SD 0.72), and Teale (mean 3.53, SD 0.47), whereas the worst quality scores were obtained by Reflexe reussite (mean 2.59, SD 0.61), VOS – journal de l’humeur (mean 2.55, SD 0.71), and Mental Booster (mean 2.49, SD 0.61). Mean MARS-F subjective quality score varied from 1.22 (SD 0.26) for VOS – journal de l’humeur to 2.69 (SD 0.84) for Soutien psy avec Mon Sherpa. The best subjective quality scores were obtained by Soutien psy avec Mon Sherpa (mean 2.69, SD 0.84), Teale (mean 2.53, SD 0.91), and Evoluno (mean 2.42, SD 1), whereas the worst quality scores were obtained by Mental Booster (mean 1.25, SD 0.33), Reflexe reussite (mean 1.25, SD 0.33), and VOS – journal de l’humeur (mean 1.22, SD 0.26).

    ‎
    Figure 2. Qualitative evaluation of mental health mobile apps—section A: engagement, section B: functionality, section C: esthetics, and section D: information.
    View this figure
    ‎
    Figure 3. Qualitative (sections A, B, C, and D) and subjective qualitative (section E) evaluation of mental health mobile apps.
    View this figure

    Assessing the Content Specificity of Mental Health Mobile Apps

    The evaluation of the specificity (section F) of the mental health apps is summarized in Figure 4 and Table S1 in Multimedia Appendix 3. This score (mean) ranged from 1.56 (SD 0.97) for Mental Booster to 3.31 (SD 1.22) for Evoluno. The best specificity scores were achieved by Soutien psy avec Mon Sherpa (mean 2.59, SD 0.92), Teale (mean 3.20, SD 0.90), and Evoluno (mean 3.31, SD 1.22), whereas the worst quality scores were obtained by Mental Booster (mean 1.56, SD 0.97), Reflexe reussite (mean 1.57, SD 0.76), and VOS – journal de l’humeur (mean 1.65, SD 1.02).

    ‎
    Figure 4. App-specific scores of mental health mobile apps (section F).
    View this figure

    Evaluation of the Strengths and Weaknesses of Each App

    The app-specific score (section F) was always lower than the subjective quality score (section E), which was always lower than the MARS-F quality score (sections A, B, C, and D), except for Soutien psy avec Mon Sherpa. This score was lower than the rating from Apple App Store or Google Play Store, when the apps are rated in the stores (Multimedia Appendix 2).

    The graphical comparison of the average scores for each item and each app (Table S2 in Multimedia Appendix 3) is shown in the heat map (Figure 5). The weakness of the quality score (sections A, B, C, and D) was mainly owing to sections A and D, except for Coach et Moi, Evoluno, Livewell - Your health partner, and Soutien psy avec Mon Sherpa. More particularly, in section D, the worst scores were observed for the credibility of the app (item 18), except for Coach et Moi, Soutien psy avec Mon Sherpa, and Teale, with mean 2.78 (SD 1.30), mean 2.78 (SD 0.83), and mean 2.56 (SD 0.73), respectively. The subjective quality (section E) was affected by the weak score regarding the price of the app (item 22). For the specificity of the apps (section F), no real weakness was observed because all the apps had similar item scores. It should be noted that Evoluno and Soutien psy avec Mon Sherpa obtained the highest scores.

    ‎
    Figure 5. Heat map of the average scores for each item and each app. The colors are related to the scores and range from yellow (1=worst score) to green (5=best score).
    View this figure

    Correlation Between MARS-F Score and Star Rating

    The correlation between the MARS-F quality score (sections A, B, C, and D) and MARS-F overall star rating (item 23: “What is your overall star rating of the app?”) was considered as good (r=0.86; P<.001). The MARS-F quality score was higher than the MARS-F overall star rating for all the mental health apps tested (12/12, 100%). Correlation analysis between the MARS-F quality score of the apps and their respective star rating in the mobile app stores was limited by the availability of star rating in the stores and discrepancies among the number of raters. The store ratings were high compared with the MARS-F quality score. These user ratings fluctuate from 2.7 (VOS – journal de l’humeur) to 4.8 (Alan Mind: thérapie bien-être) for the iOS store and from 3.2 (Coach et Moi) to 4.6 (Envol - Pleine santé) for the Android store (Table 2).

    Table 2. MARS-Fa overall star rating, overall quality MARS-F score, star rating in the Apple App Store, and star rating in the Google Play Store.
    AppsMARS-F overall star rating (item 23), mean (SD)MARS-F quality score (sections A, B, C, and D), mean (SD)Apple App Store, star rating; number of ratersGoogle Play Store, star rating; number of raters
    Alan Mind: thérapie bien-être2.44 (0.53)3.22 (0.56)4.8; 32174.2; 125
    Coach et Moi2.89 (1.27)3.47 (0.55)4.6; 283.2; 32
    Envol - Pleine santé2.67 (1.41)3.23 (0.97)4.8; 304.6; 156
    Evoluno3.00 (1.22)3.54 (0.72)N/AbN/A
    goalmap: objectifs bien etre1.78 (0.44)2.97 (0.49)4.2; 2693.6; 2000
    Livewell - Your health partner2.44 (1.13)3.28 (0.73)N/AN/A
    Mental Booster1.67 (0.71)2.49 (0.61)3.1; 11N/A
    Reflexe reussite1.67 (0.71)2.59 (0.61)4.4; 104; 28
    Sanvello2.22 (1.09)3.18 (0.54)4.3; 3394.2; 23,000
    Soutien psy avec Mon Sherpa3.67 (1.00)3.85 (0.48)4.6; 14094.3; 1000
    Teale3.22 (1.09)3.53 (0.47)4.2; 174.3; 14
    VOS – journal de l’humeur1.78 (0.83)2.55 (0.71)2.7; 144.4; 16,000

    aMARS-F: Mobile App Rating Scale–French.

    bN/A: not applicable.


    Principal Findings

    Mental health apps aim to improve mental health and wellness, whether in terms of curing a mental illness or implementing habits to improve emotional health [32]. A public survey showed that 76% of the respondents would like to have access to free apps to self-manage and self-monitor their mental health [33]. Thus, mental health apps can play a critical role in mental health care [34]; making mental health care accessible; and reducing barriers to help-seeking, as few people with mood or anxiety problems seek professional help. In addition, geographic, financial, or social constraints may make it difficult to access support [21].

    Mental health apps cover all stages of clinical care delivery, including prevention, diagnosis, primary treatment [35], and so on. Our study focused on mental health well-being apps that were consistent with health promotion. Thus, we included apps that enabled users’ mental health self-management and that did or did not offer appointments with mental health professionals. We excluded apps whose only purpose was to schedule appointments with mental health professionals, because their focus was more treatment-oriented. Screening of mental health apps available in the French iOS and Android stores led to the inclusion of 12 apps. In Singapore, screening of iTunes and Google Play Store led to inclusion of 44 mental health apps [36]. Compared with our study, they used more keywords, and several of them were related to COVID-19. In China, the screening permitted to select 40 apps, but the authors screened the apps in Apple App Store (for iOS apps), Tencent My App, and Huawei App Gallery (for Android apps), and they selected apps in the Chinese language [37]. In our 2 previous studies using the same methodology but focusing on the risk factors of noncommunicable diseases, 15 apps related to nutrition [38] and 9 apps related to oral hygiene [39] were included.

    All the included apps aimed to increase happiness, well-being, mindfulness, meditation, and relaxation and reduce negative emotions, anxiety, stress, and anger. Some apps additionally targeted alcohol or substance use (Livewell - Your health partner and Teale), relationships (Livewell - Your health partner and VOS – journal de l’humeur), or physical activity (goalmap: objectifs bien etre and Livewell - Your health partner). These mental health apps mainly used CBT, acceptance commitment therapy, and mindfulness or mediation to reach their goals. The use of CBT in mental health apps was one of the recommendations of Bakker et al [21]. CBT is a form of collaborative, individualized psychological treatment that is recognized as the most sustained approach for generating behavioral, cognitive, and emotional adaptation to a wide range of common psychological problems [40]. Several studies have concluded that CBT is an effective treatment for a wide range of psychological disorders, which can be administered via a mobile device and still retain its therapeutic validity [21]. Moreover, CBT can also prevent psychological problems from becoming worse [41-43].

    The MARS-F quality score was >2.5, for all the mental health apps tested, except for Mental Booster (11/12, 92%; mean 2.49, SD 0.61). Mean score ranged from 2.49 (SD 0.61) to 3.85 (SD 0.48). Similar scores were observed for apps related to nutrition available in France (mean 3.34, SD 0.39 to mean 3.84, SD 0.32) [38] and oral hygiene (mean 1.80, SD 0.79 to mean 3.4, SD 0.97) [39]. Soutien psy avec Mon Sherpa, which obtained the highest score (mean 3.85, SD 0.48), had the particularity of offering real-time interactions with an artificial intelligence–enabled chatbot to collect and record daily mood. Thus, the user can develop a relationship with the chatbot, similar to the one observed between a therapist and their patient, which involves collaborative empiricism [44]. The 2 apps (Evoluno: mean 3.54, SD 0.72 and Teale: mean 3.53, SD 0.47) that received the highest MARS-F scores had a different concept. They provided some information to the user (resilience, stress, anxiety, negative thoughts, etc) in the form of a podcast (Evoluno) or videos (Teale). Evoluno proposed to evaluate the user (sleep quality, risk of burnout, depression, etc) and suggested breathing exercises. Teale proposed to set goals for the user (self-control, self-esteem, self-fulfillment, etc) with various activities (meditation, breathing, writing, etc) to reach them. Moreover, these 2 apps permitted to contact mental health professionals.

    The functionality (section B) was the strength for all the apps tested, except for Sanvello (11/12, 92%). Our previous studies demonstrated that it was also the strength of French nutrition–related and oral hygiene–related apps [38,39]. Information quality (section D) was the weakness of all the apps, except 4 apps (8/12, 67%; Alan Mind: thérapie bien-être, Coach et Moi, goalmap: objectifs bien etre, and Livewell - Your health partner). The weak point of Coach et Moi was the aesthetics (section C), which was not observed for any of the French apps related to nutrition or oral hygiene included in our previous studies [38,39]. The worst score in section D was observed for the credibility of the app because the mental health professionals identified the source of information, but they evaluated that its validity or reliability was questionable (eg, commercial enterprise with vested interest). In addition, as the level of scientific evidence is difficult to assess, mental health professionals chose “N/A The application has not been tested” in most cases to answer item 19, which therefore could not be included in the statistical analysis. To our knowledge, of the 12 apps, only 1 (8%) app included in this study (Sanvello) was indexed in PubMed [45-48]. Myers et al [45] evaluated apps for depression self-management, Mehdi et al [46] evaluated apps for Tinnitus, and Lau et al [47] evaluated apps for mental health. In these 3 studies, the observed MARS scores for Sanvello were 4.6, 4.6, and 4.28, respectively, which were higher than those observed in our study (mean 3.18, SD 0.54). This difference could be explained by the fact that, in these studies, the number of raters was 2 [45,47] or 4 [46] at most.

    The MARS-F quality score (sections A, B, C, and D) was higher than the specificity of the app (section F), which was higher than the subjective quality (section E), except for Soutien psy avec Mon Sherpa. For this app, the specificity of app (section F) obtained the low score. The subjective quality score (section E) was >2.5 only for Soutien psy avec Mon Sherpa (mean 2.69, SD 0.84) and Teale (mean 2.53, SD 0.91). For all the apps tested (12/12, 100%), the main problem was the cost, because mental health professionals declared that they will not pay for this app. The specificity of the apps (section F) assessed the perceived impact of the app on the user’s knowledge, attitudes, intent to change, and likelihood of actual change regarding mental health. Only 50% (6/12) of the mental health apps included obtained a score >2.5 for this section. Evoluno (mean 3.31, SD 1.22), Teale (mean 3.20, SD 0.90), and Coach et Moi (mean 2.89, SD 0.95) obtained the best scores, followed by Livewell - Your health partner (mean 2.78, SD 1.12), Alan Mind: thérapie bien-être (mean 2.67, SD 0.72), and Soutien psy avec Mon Sherpa (mean 2.59, SD 0.92). These results can be explained because professionals preferred apps containing information that promotes mental health and well-being than a chatbot.

    In contrast to previous studies on French nutrition–related [38] and oral hygiene–related [39] apps, the star rating score from the iOS and Android stores was always lower than the subjective quality score (section E). This difference may be explained by the fact that, when the mental health apps were evaluated in the stores, they were evaluated by a very small number of evaluators, which is in contrast to the nutrition and oral hygiene apps, which were evaluated by a large number of evaluators. The difference between the star rating score and subjective quality score was owing to the fact that the evaluations in the mobile app stores are made by all the users, whereas the evaluation in this study was made by mental health professionals using the MARS-F. Thus, it could be interesting to evaluate the same apps using the user version of the MARS (uMARS) that is designed for users to compare the results.

    Limitations

    This study has several limitations. First, only mental health apps available on both French Apple App Store and Google Play Store were included, even though several other stores could be investigated (Huawei store, BlackBerry World, Windows Phone Store, or Samsung store). Second, regarding the protection of personal data, no criteria have been defined, except for password protection to log in to the apps. This criterion is not sufficient to prove that the user’s personal data are protected.

    Third, the commercial use of mobile mental health apps was not considered as a key criterion for assessing the acceptability of the app to users when many mobile apps are developed by anonymous companies but funded by large groups such as pharmaceutical companies. Fourth, MARS was selected for this investigation because it is the most commonly used scale in scientific literature for the assessment of mental health apps [38,39,47,49-59], and its use has been recommended by the French High Authority of Health [60]. Nevertheless, other scales, such as ENLIGHT, which also assesses mobile health apps, could have been used [61]. Correlation and good parallel validity between the MARS and ENLIGHT have been shown [62]. Fifth, our study is only applicable to the French-speaking community, where these mobile apps are available. Sixth, the evaluation was conducted by mental health professionals, whereas the mobile health apps were developed for the general population. Therefore, it will be interesting to have these apps evaluated by users, using the uMARS scale [63] that was developed for them, and then compare the results.

    Perspectives

    The COVID-19 pandemic has changed the use of digital technology in the field of mental health and, more particularly, by mental health professionals. The pandemic has made it possible to revisit the ways in which the most fragile people are cared for with connected tools and to move toward an evolution of practices [64]. Mental health apps can improve accessibility to mental health services and be beneficial for mental health [32]. They have positive effects on depressive symptoms, anxiety, and other mental health problems [65] and help overcome some of the barriers that make it difficult to access mental health services (stigma, cost, capacity, and geography) [66]. Thus, this study can help mental health professionals and mobile app users to choose the best mental health apps available in France, in terms of quality.

    Health care professionals generally have a different perspective than users. Thus, it may be interesting to have the mental health apps studied in this study evaluated by users via the uMARS scale and then compared.

    This study qualitatively assessed mental health–related apps but did not focus on content analysis of the included apps. A review of the literature specific to apps supporting mental health and wellness concluded that apps for bipolar disorder were cost-effective and convenient but that most of them did not provide information on all the core principles of psychoeducation and did not adhere to best practice guidelines [67]. Similarly, Larsen et al [68] showed that many suicide prevention apps were not evidence-based and determined that some apps may be more harmful than helpful. They found that some apps included potentially extremely harmful content describing or supporting access to lethal means, encouraging people to end their lives, and presenting suicide in a trendy manner [66]. Moreover, as many apps are developed for commercial purposes, ethical questions arise regarding the use of personal data. Therefore, additional studies analyzing the content of the apps are needed.

    Furthermore, the implementation of randomized clinical trials or longitudinal studies including the 12 mental health apps selected in this study should allow for the analysis of the change in mental health status after the use of the apps.

    Conclusions

    Mental health professionals assessed that, despite the lack of scientific evidence, the mental health mobile apps available on the French Apple App Store and Google Play Store were of good quality. However, they are reluctant to use them in their professional practice. Additional investigations are needed to assess their compliance with recommendations and their long-term impact on users.

    Acknowledgments

    The authors would like to thank the mental health professionals for their availability and engagement. The authors would also like to thank Union Prévention Santé pour la Fonction publique, a French player in preventive health care, as manager of the public service’s compulsory health insurance scheme, because the subject of this study appeared during a reflection on the mental health support for civil servants. A recent ministerial circular (NOR: TFPF2133602C; February 23, 2022) calls for the development of a culture of prevention of situations of psychological distress and promotion of mental health, particularly in professional environments and within public jobs.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Characteristics of the raters and the hardware and software used.

    DOCX File , 18 KB
    ‎
    Multimedia Appendix 2

    Descriptive and technical information about the mental health mobile apps.

    DOCX File , 40 KB
    ‎
    Multimedia Appendix 3

    Mobile App Rating Scale scoring.

    DOCX File , 47 KB
    1. Dattani S, Ritchie H, Roser M. Mental health. Our World in Data. 2018 Apr.   URL: https://ourworldindata.org/mental-health [accessed 2022-07-15]
    2. Nochaiwong S, Ruengorn C, Thavorn K, Hutton B, Awiphan R, Phosuya C, et al. Global prevalence of mental health issues among the general population during the coronavirus disease-2019 pandemic: a systematic review and meta-analysis. Sci Rep 2021 May 13;11(1):10173 [FREE Full text] [CrossRef] [Medline]
    3. CIM-11 pour les statistiques de mortalité et de morbidité (CIM-11 SMM). ICD.   URL: https://icd.who.int/browse11/l-m/fr [accessed 2022-07-15]
    4. Yeo ZZ, Suárez L. Validation of the mental health continuum-short form: the bifactor model of emotional, social, and psychological well-being. PLoS One 2022;17(5):e0268232 [FREE Full text] [CrossRef] [Medline]
    5. McAneney H, Tully MA, Hunter RF, Kouvonen A, Veal P, Stevenson M, et al. Individual factors and perceived community characteristics in relation to mental health and mental well-being. BMC Public Health 2015 Dec 12;15:1237 [FREE Full text] [CrossRef] [Medline]
    6. Moksnes UK, Bjørnsen HN, Ringdal R, Eilertsen MB, Espnes GA. Association between loneliness, self-esteem and outcome of life satisfaction in Norwegian adolescents aged 15-21. Scand J Public Health 2022 Mar 31:14034948221081287 [FREE Full text] [CrossRef] [Medline]
    7. Santini ZI, Pisinger VS, Nielsen L, Madsen KR, Nelausen MK, Koyanagi A, et al. Social disconnectedness, loneliness, and mental health among adolescents in danish high schools: a nationwide cross-sectional study. Front Behav Neurosci 2021;15:632906 [FREE Full text] [CrossRef] [Medline]
    8. Reijneveld S. Mental health as a public health issue. Eur J Public Health 2015 Apr 1;15(2):111. [CrossRef]
    9. Munir S, Takov V. Generalized anxiety disorder. In: StatPearls. Treasure Island (FL): StatPearls Publishing; 2022.
    10. Depression and other common mental disorders: global health estimates. World Health Organization. 2017.   URL: https://apps.who.int/iris/bitstream/handle/10665/254610/WHO-MSD-MER-2017.2-eng.pdf [accessed 2022-07-15]
    11. GBD 2015 DiseaseInjury IncidencePrevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 310 diseases and injuries, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015. Lancet 2016 Oct 08;388(10053):1545-1602 [FREE Full text] [CrossRef] [Medline]
    12. Ramiz L, Contrand B, Rojas Castro MY, Dupuy M, Lu L, Sztal-Kutas C, et al. A longitudinal study of mental health before and during COVID-19 lockdown in the French population. Global Health 2021 Mar 22;17(1):29 [FREE Full text] [CrossRef] [Medline]
    13. Eurofound, European Agency for Safety and Health at Work, Psychosocial risks in Europe : prevalence and strategies for prevention : a joint report from the European Foundation for the Improvement of Living and Working Conditions and the European Agency for Safety and Health at Work. Publications Office of the European Union. 2014.   URL: https:/​/op.​europa.eu/​en/​publication-detail/​-/​publication/​0567add6-db0e-4065-b13d-7e2599090851/​language-en [accessed 2022-07-15]
    14. Melchior M, Caspi A, Milne BJ, Danese A, Poulton R, Moffitt TE. Work stress precipitates depression and anxiety in young, working women and men. Psychol Med 2007 Aug;37(8):1119-1129 [FREE Full text] [CrossRef] [Medline]
    15. Veldman K, Reijneveld SA, Verhulst FC, Ortiz JA, Bültmann U. A life course perspective on mental health problems, employment, and work outcomes. Scand J Work Environ Health 2017 Jul 01;43(4):316-325 [FREE Full text] [CrossRef] [Medline]
    16. Mashaphu S, Talatala M, Seape S, Eriksson L, Chiliza B. Mental health, culture and resilience-approaching the COVID-19 pandemic from a South African perspective. Front Psychiatry 2021;12:611108 [FREE Full text] [CrossRef] [Medline]
    17. Maric V, Mishra J, Ramanathan DS. Using mind-body medicine to reduce the long-term health impacts of COVID-specific chronic stress. Front Psychiatry 2021;12:585952 [FREE Full text] [CrossRef] [Medline]
    18. Kola L, Kohrt BA, Hanlon C, Naslund JA, Sikander S, Balaji M, et al. COVID-19 mental health impact and responses in low-income and middle-income countries: reimagining global mental health. Lancet Psychiatry 2021 Feb 24;8(6):535-550. [CrossRef] [Medline]
    19. Esponda GM, Hartman S, Qureshi O, Sadler E, Cohen A, Kakuma R. Barriers and facilitators of mental health programmes in primary care in low-income and middle-income countries. Lancet Psychiatry 2020 Jan;7(1):78-92. [CrossRef] [Medline]
    20. Global governance toolkit for digital mental health. World Economic Forum. 2021 Apr 6.   URL: https://www.weforum.org/whitepapers/global-governance-toolkit-for-digital-mental-health/ [accessed 2022-07-15]
    21. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Ment Health 2016 Mar 01;3(1):e7 [FREE Full text] [CrossRef] [Medline]
    22. Auxier B, Westcott K, Bucaille A. Mental health goes mobile: The mental health app market will keep on growing. Deloitte Insights. 2021 Dec 1.   URL: https:/​/www2.​deloitte.com/​xe/​en/​insights/​industry/​technology/​technology-media-and-telecom-​predictions/​2022/​mental-health-app-market.​html [accessed 2022-07-15]
    23. Cangiano F, Parker S. Proactivity for mental health and well-being. In: The Wiley Blackwell Handbook of the Psychology of Occupational Safety and Workplace Health. Hoboken, New Jersey, United States: Wiley Blackwell; 2016.
    24. Eisenstadt M, Liverpool S, Infanti E, Ciuvat RM, Carlsson C. Mobile apps that promote emotion regulation, positive mental health, and well-being in the general population: systematic review and meta-analysis. JMIR Ment Health 2021 Nov 08;8(11):e31170 [FREE Full text] [CrossRef] [Medline]
    25. Hwang WJ, Ha JS, Kim MJ. Research trends on mobile mental health application for general population: a scoping review. Int J Environ Res Public Health 2021 Mar 02;18(5):2459 [FREE Full text] [CrossRef] [Medline]
    26. Gál É, Ștefan S, Cristea IA. The efficacy of mindfulness meditation apps in enhancing users' well-being and mental health related outcomes: a meta-analysis of randomized controlled trials. J Affect Disord 2021 Jan 15;279:131-142. [CrossRef] [Medline]
    27. Eis S, Solà-Morales O, Duarte-Díaz A, Vidal-Alaball J, Perestelo-Pérez L, Robles N, et al. Mobile applications in mood disorders and mental health: systematic search in apple app store and google play store and review of the literature. Int J Environ Res Public Health 2022 Feb 15;19(4):2186 [FREE Full text] [CrossRef] [Medline]
    28. Saliasi I, Martinon P, Darlington E, Smentek C, Tardivo D, Bourgeois D, et al. Promoting health via mHealth applications using a French version of the mobile app rating scale: adaptation and validation study. JMIR Mhealth Uhealth 2021 Aug 31;9(8):e30480 [FREE Full text] [CrossRef] [Medline]
    29. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
    30. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979 Mar;86(2):420-428. [CrossRef] [Medline]
    31. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropractic Med 2016 Jun;15(2):155-163 [FREE Full text] [CrossRef] [Medline]
    32. Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013 Nov 15;15(11):e247 [FREE Full text] [CrossRef] [Medline]
    33. Proudfoot J, Parker G, Hadzi Pavlovic D, Manicavasagar V, Adler E, Whitton A. Community attitudes to the appropriation of mobile phones for monitoring and managing depression, anxiety, and stress. J Med Internet Res 2010 Dec 19;12(5):e64 [FREE Full text] [CrossRef] [Medline]
    34. Simon GE, Ludman EJ. It's time for disruptive innovation in psychotherapy. Lancet 2009 Aug 22;374(9690):594-595. [CrossRef] [Medline]
    35. Chandrashekar P. Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. Mhealth 2018;4:6 [FREE Full text] [CrossRef] [Medline]
    36. Li LS, Wong LL, Yap KY. Quality evaluation of stress, anxiety and depression apps for COVID-19. J Affect Disord Rep 2021 Dec;6:100255 [FREE Full text] [CrossRef] [Medline]
    37. Wu X, Xu L, Li P, Tang T, Huang C. Multipurpose mobile apps for mental health in Chinese app stores: content analysis and quality evaluation. JMIR Mhealth Uhealth 2022 Jan 04;10(1):e34054 [FREE Full text] [CrossRef] [Medline]
    38. Martinon P, Saliasi I, Bourgeois D, Smentek C, Dussart C, Fraticelli L, et al. Nutrition-related mobile apps in the French app stores: assessment of functionality and quality. JMIR Mhealth Uhealth 2022 Mar 14;10(3):e35879 [FREE Full text] [CrossRef] [Medline]
    39. Carrouel F, Bourgeois D, Clément C, Tardivo D, Martinon P, Guiral S, et al. Oral-hygiene-related mobile apps in the French app stores: assessment of functionality and quality. Int J Environ Res Public Health 2022 Jun 14;19(12):7293 [FREE Full text] [CrossRef] [Medline]
    40. Payne L, Ellard K, Fairholme C, Barlow D. Emotional disorders: a unified transdiagnostic protocol. In: Clinical Handbook of Psychological Disorders, Fourth Edition: A Step-by-Step Treatment Manual. New York, United States: The Guilford Press; 2008.
    41. Bakker G. Practical CBT Using Functional Analysis and Standardised Homework in Everyday Therapy. Queensland, Australia: Australian Academic Press; 2008.
    42. Moorey S. The six cycles maintenance model: growing a "vicious flower" for depression. Behav Cogn Psychother 2010 Mar;38(2):173-184. [CrossRef] [Medline]
    43. Gyani A, Shafran R, Layard R, Clark DM. Enhancing recovery rates: lessons from year one of IAPT. Behav Res Ther 2013 Sep;51(9):597-606 [FREE Full text] [CrossRef] [Medline]
    44. Person J. The Case Formulation Approach to Cognitive-Behavior Therapy. New York: The Guilford Press; 2008.
    45. Myers A, Chesebrough L, Hu R, Turchioe MR, Pathak J, Creber RM. Evaluating commercially available mobile apps for depression self-management. AMIA Annu Symp Proc 2020;2020:906-914 [FREE Full text] [Medline]
    46. Mehdi M, Stach M, Riha C, Neff P, Dode A, Pryss R, et al. Smartphone and mobile health apps for tinnitus: systematic identification, analysis, and assessment. JMIR Mhealth Uhealth 2020 Aug 18;8(8):e21767 [FREE Full text] [CrossRef] [Medline]
    47. Lau N, O'Daffer A, Yi-Frazier JP, Rosenberg AR. Popular evidence-based commercial mental health apps: analysis of engagement, functionality, aesthetics, and information quality. JMIR Mhealth Uhealth 2021 Jul 14;9(7):e29689 [FREE Full text] [CrossRef] [Medline]
    48. Leong QY, Sridhar S, Blasiak A, Tadeo X, Yeo G, Remus A, et al. Characteristics of mobile health platforms for depression and anxiety: content analysis through a systematic review of the literature and systematic search of two app stores. J Med Internet Res 2022 Feb 04;24(2):e27388 [FREE Full text] [CrossRef] [Medline]
    49. Talwar D, Yeh Y, Chen W, Chen L. Characteristics and quality of genetics and genomics mobile apps: a systematic review. Eur J Hum Genet 2019 Jun;27(6):833-840 [FREE Full text] [CrossRef] [Medline]
    50. Chen R, Santo K, Wong G, Sohn W, Spallek H, Chow C, et al. Mobile apps for dental caries prevention: systematic search and quality evaluation. JMIR Mhealth Uhealth 2021 Jan 13;9(1):e19958 [FREE Full text] [CrossRef] [Medline]
    51. Ciapponi A, Donato M, Gülmezoglu AM, Alconada T, Bardach A. Mobile apps for detecting falsified and substandard drugs: a systematic review. PLoS One 2021;16(2):e0246061 [FREE Full text] [CrossRef] [Medline]
    52. Mandracchia F, Llauradó E, Tarro L, Valls RM, Solà R. Mobile phone apps for food allergies or intolerances in app stores: systematic search and quality assessment using the mobile app rating scale (MARS). JMIR Mhealth Uhealth 2020 Sep 16;8(9):e18339 [FREE Full text] [CrossRef] [Medline]
    53. Romero RL, Kates F, Hart M, Ojeda A, Meirom I, Hardy S. Quality of deaf and hard-of-hearing mobile apps: evaluation using the mobile app rating scale (MARS) with additional criteria from a content expert. JMIR Mhealth Uhealth 2019 Oct 30;7(10):e14198 [FREE Full text] [CrossRef] [Medline]
    54. Escriche-Escuder A, De-Torres I, Roldán-Jiménez C, Martín-Martín J, Muro-Culebras A, González-Sánchez M, et al. Assessment of the quality of mobile applications (Apps) for management of low back pain using the mobile app rating scale (MARS). Int J Environ Res Public Health 2020 Dec 09;17(24):9209 [FREE Full text] [CrossRef] [Medline]
    55. Salehinejad S, Niakan Kalhori SR, Hajesmaeel Gohari S, Bahaadinbeigy K, Fatehi F. A review and content analysis of national apps for COVID-19 management using Mobile Application Rating Scale (MARS). Inform Health Soc Care 2021 Mar 02;46(1):42-55. [CrossRef] [Medline]
    56. Kalhori SR, Hemmat M, Noori T, Heydarian S, Katigari MR. Quality evaluation of english mobile applications for gestational diabetes: app review using mobile application rating scale (MARS). Curr Diabetes Rev 2021;17(2):161-168. [CrossRef] [Medline]
    57. Davalbhakta S, Advani S, Kumar S, Agarwal V, Bhoyar S, Fedirko E, et al. A systematic review of the smartphone applications available for coronavirus disease 2019 (COVID19) and their assessment using the mobile app rating scale (MARS). medRxiv 2020 Jul 04 (forthcoming) [FREE Full text] [CrossRef] [Medline]
    58. Siddiqui NR, Hodges SJ, Sharif MO. Orthodontic apps: an assessment of quality (using the Mobile App Rating Scale (MARS)) and behaviour change techniques (BCTs). Prog Orthod 2021 Sep 13;22(1):25 [FREE Full text] [CrossRef] [Medline]
    59. Martín-Martín J, Muro-Culebras A, Roldán-Jiménez C, Escriche-Escuder A, De-Torres I, González-Sánchez M, et al. Evaluation of android and apple store depression applications based on mobile application rating scale. Int J Environ Res Public Health 2021 Nov 27;18(23):12505 [FREE Full text] [CrossRef] [Medline]
    60. Évaluation des Applications dans le champ de la santé mobile (mHealth) - État des lieux et critères de qualité du contenu médical pour le référencement des services numériques dans l’espace numérique de santé et le bouquet de services des professionnels. Haute Autorité de la Santé. 2021 Jun 24.   URL: https://www.has-sante.fr/upload/docs/application/pdf/2021- [accessed 2022-09-22]
    61. Baumel A, Faber K, Mathur N, Kane JM, Muench F. Enlight: a comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. J Med Internet Res 2017 Mar 21;19(3):e82 [FREE Full text] [CrossRef] [Medline]
    62. Naydenova G. An overview of existing methods for evaluating mobile applications to change user behavior. Int Scientific J Scientific Technical Union Mechanical Eng 2022;7(1):21-25.
    63. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016 Jun 10;4(2):e72 [FREE Full text] [CrossRef] [Medline]
    64. Kane H, Gourret Baumgart J, El-Hage W, Deloyer J, Maes C, Lebas M, et al. Opportunities and challenges for professionals in psychiatry and mental health care using digital technologies during the COVID-19 pandemic: systematic review. JMIR Hum Factors 2022 Feb 04;9(1):e30359 [FREE Full text] [CrossRef] [Medline]
    65. Firth J, Torous J, Nicholas J, Carney R, Pratap A, Rosenbaum S, et al. The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry 2017 Oct;16(3):287-298 [FREE Full text] [CrossRef] [Medline]
    66. Zelmer J, van Hoof K, Notarianni M, van Mierlo T, Schellenberg M, Tannenbaum C. An assessment framework for e-mental health apps in Canada: results of a modified Delphi process. JMIR Mhealth Uhealth 2018 Jul 09;6(7):e10016 [FREE Full text] [CrossRef] [Medline]
    67. Nicholas J, Larsen ME, Proudfoot J, Christensen H. Mobile apps for bipolar disorder: a systematic review of features and content quality. J Med Internet Res 2015 Aug 17;17(8):e198 [FREE Full text] [CrossRef] [Medline]
    68. Larsen ME, Nicholas J, Christensen H. A systematic assessment of smartphone tools for suicide prevention. PLoS One 2016;11(4):e0152285 [FREE Full text] [CrossRef] [Medline]

    ‎
    CBT: cognitive behavioral therapy
    ICC: intraclass correlation
    iOS: iPhone Operating System
    MARS: Mobile App Rating Scale
    MARS-F: Mobile App Rating Scale–French
    uMARS: user version of the Mobile App Rating Scale

    Edited by L Buis; submitted 21.07.22; peer-reviewed by F Denis, A Finny, D Faye, J Grosjean; comments to author 02.09.22; revised version received 03.09.22; accepted 09.09.22; published 12.10.22

    Copyright

    ©Florence Carrouel, Benjamin du Sartz de Vigneulles, Denis Bourgeois, Bernard Kabuth, Nicolas Baltenneck, Fanny Nusbaum, Valérie Burge, Sylvain Roy, Sophie Buchheit, Marie-Line Carrion-Martinaud, Catherine Massoubre, Laurie Fraticelli, Claude Dussart. Originally published in JMIR mHealth and uHealth (https://mhealth.jmir.org), 12.10.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license information must be included.