Published on 18.08.20 in Vol 8, No 8 (2020): August
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/21767, first published Jun 24, 2020.
Smartphone and Mobile Health Apps for Tinnitus: Systematic Identification, Analysis, and Assessment
Background: Modern smartphones contain sophisticated high-end hardware features, offering high computational capabilities at extremely manageable costs and have undoubtedly become an integral part in users' daily life. Additionally, smartphones offer a well-established ecosystem that is easily discoverable and accessible via the marketplaces of differing mobile platforms, thus encouraging the development of many smartphone apps. Such apps are not exclusively used for entertainment purposes but are also commonplace in health care and medical use. A variety of those health and medical apps exist within the context of tinnitus, a phantom sound perception in the absence of any physical external source.
Objective: In this paper, we shed light on existing smartphone apps addressing tinnitus by providing an up-to-date overview.
Methods: Based on PRISMA guidelines, we systematically searched and identified existing smartphone apps on the most prominent app markets, namely Google Play Store and Apple App Store. In addition, we applied the Mobile App Rating Scale (MARS) to evaluate and assess the apps in terms of their general quality and in-depth user experience.
Results: Our systematic search and screening of smartphone apps yielded a total of 34 apps (34 Android apps, 26 iOS apps). The mean MARS scores (out of 5) ranged between 2.65-4.60. The Tinnitus Peace smartphone app had the lowest score (mean 2.65, SD 0.20), and Sanvello—Stress and Anxiety Help had the highest MARS score (mean 4.60, SD 0.10). The interrater agreement was substantial (Fleiss κ=0.74), the internal consistency was excellent (Cronbach α=.95), and the interrater reliability was found to be both high and excellent—Guttman λ6=0.94 and intraclass correlation, ICC(2,k) 0.94 (95% CI 0.91-0.97), respectively.
Conclusions: This work demonstrated that there exists a plethora of smartphone apps for tinnitus. All of the apps received MARS scores higher than 2, suggesting that they all have some technical functional value. However, nearly all identified apps were lacking in terms of scientific evidence, suggesting the need for stringent clinical validation of smartphone apps in future. To the best of our knowledge, this work is the first to systematically identify and evaluate smartphone apps within the context of tinnitus.
JMIR Mhealth Uhealth 2020;8(8):e21767
Tinnitus is a condition mainly associated with the perception of a continuous ringing noise in the ears in the absence of any external sound source. The direct causative factors of the perception of subjective tinnitus are manifold and are not fully understood. However, tinnitus is often associated with underlying damage in the inner ear, such as the loss of cochlear hair cells. Worldwide, roughly 15% of the population suffers from tinnitus; among them 2% experience a substantial decrease in quality of life due to the phantom percept . At present, tinnitus is considered to be a condition that involves changes at different levels of the auditory pathway, the auditory cortex, and nonauditory areas such as the limbic system. These changes may additionally be influenced by psychosocial stress (eg, negative thoughts, stress at home, increased workload, etc), which affects not only the emotional status of the patient but also the auditory system [ , ]. Consequently, people with tinnitus often report that their perception of it varies, including its loudness or related distress [ ]. This moment-to-moment variability can be captured by utilizing ecological momentary assessments [ ]. Herein, we consider smartphone-based solutions and apps that can be employed to better understand tinnitus, or to offer assistance to patients with tinnitus in managing tinnitus-related distress.
Interestingly in recent years, smartphones, smartphone apps, and auxiliary health devices such as heart monitors and smart wristbands have gained significant popularity by helping patients to monitor and treat their health problems [- ]. Specifically, smartphones provide an app ecosystem that can easily be extended with new apps. For instance, on one hand, smartphone-based app solutions can be applied to monitor the ecological or environmental surroundings of patients to better understand health phenomena [ , ]. On the other hand, these solutions can also easily be designed or tailored to assist patients in managing or mitigating the symptoms of their health problems [ ]. Within the scope of this paper, such solutions can be directly applied within the context of tinnitus and other closely related health complaints, such as stress [ - ], Ménière disease [ , ], hearing loss [ - ], vertigo [ - ], and dementia [ , ]. Consequently, developers frequently push new apps to the markets to capitalize on the growing interest in health care–related smartphone apps in both academia and industry. Thus, timely assessment and evaluation of these smartphone apps are critically important, particularly due to the sensitive nature of their target domain.
Several mobile health (mHealth) app assessment tools and models are available [- ], yet they either are focused on a particular health domain or are too time consuming and complex to employ in research. Furthermore, it is often hard to establish an objective score based on these rating and assessment tools. Nonetheless, among them the Mobile Application Rating Scale (MARS) [ ] is a reliable and valid instrument for the quality assessment of smartphone-based medical apps or mHealth apps. It offers a straightforward multidimensional rating process to objectively assess health care apps without requiring excessive training for the rater. In addition, MARS has been diversely and widely used in evaluating smartphone-based health apps, for instance, pain management apps [ ], weight management apps [ ], diabetes management apps [ ], and rheumatology apps [ , ]. Additionally, MARS can be employed for rating smartphone apps for complex psychological or physiological conditions such as depression [ ], hypertension [ ], or tinnitus [ ]. In the field of tinnitus, Sereda and colleagues [ ] gathered tinnitus management apps based on patient opinion via a web-based survey. Features of the patients’ most cited apps were then analyzed with MARS. We again apply MARS to evaluate tinnitus management apps, yet in this paper, we emphasized systematic search and exploration of the most prominent commercial mobile app platforms to identify the relevant tinnitus apps. Furthermore, we aimed to identify and evaluate recent apps (those newly added since 2019 or not previously identified) in comparison to those previously reviewed [ ].
Our work offers 3 major contributions. First, based on the PRISMA guidelines , we systematically searched, screened, and identified smartphone apps aimed at assisting patients with tinnitus. Second, with respect to the objective quality of the smartphone apps and the user experience ratings, all identified apps were critically evaluated and assessed based on MARS. As an added step, we compared our MARS ratings to ratings from other sources and computed interrating agreements. Last, we gathered information for quality ratings of the health apps from various established information platforms, as well as the star ratings from the Google Play Store and the Apple App Store.
Finding Relevant Apps
In order to generate an exhaustive overview of relevant tinnitus apps, we employed PRISMA guidelines for a systematic search, screening, and identification of the apps.
We performed an open keyword search () on 2 of the most prominent app markets, namely Google Play and Apple App stores to cover both major mobile platforms (ie, Android and iOS, respectively). Due to the device-specific limitations of apps from different app stores, we did not include app stores such as Amazon App Store, Sony Apps, Samsung Galaxy Apps, Huawei App Store, and LG SmartWorld in our app search workflow. Furthermore, third-party app providers such as Aptoide and F-Droid were not taken into account as they are not considered to be reliable sources because of security issues and their reliance on rooted devices. Rooting is the process of acquiring full system access or administrative control of mobile devices. This process is highly discouraged by device manufacturers and app developers as it may introduce security vulnerabilities [ ].
The overall workflow to systematically identify relevant apps was based on PRISMA guidelines and is illustrated in. The search yielded a total of 675 apps from both app markets (Google Play Store: 334; Apple App Store: 341 apps); 311 apps were identified after removing duplicates. These were screened based on the title and description resulting in 29 apps that satisfied the required criteria.
Using the same keywords (with the keyword app appended), Google searches were performed to find any missing or additional apps (in May 2019 and in December 2019). The Google search yielded multiple webpages and forum posts. The contents of these webpages and forum posts were manually investigated to identify additional potentially relevant apps. We also performed searches on 3 independent third-party mHealth app libraries: (1) the government-funded National Health Service Apps Library , (2) the privately funded AppScript Library [ ], and (3) the privately funded MyHealthApps Cochrane Library [ ]. These third-party mHealth app libraries are web portals that curate smartphone apps [ ]. This ancillary search of smartphone apps on webpages, tinnitus forums, and third-party mHealth app libraries resulted in the identification of 5 additional apps; therefore, a total of 34 apps were identified for assessment and evaluation. All were available for Android, whereas only 26 apps could also be used on the iOS platform.
Apps Assessments and Evaluations
The smartphone apps that were identified in the app-store search based on PRISMA guidelines were evaluated using MARS scoring guidelines. To rate the smartphone apps, 4 raters (2 raters from a tinnitus domain, and 2 raters from a mobile app–development domain with background in tinnitus research) were recruited. According to the recommendations of the MARS developers, the raters were instructed to watch a video presentation  to familiarize themselves with the scoring process. In addition and to further facilitate the familiarization, the raters were requested to read the MARS scoring paper [ ]. Next, all raters rated a sample app for training purposes, and the results were discussed briefly to ensure that all raters had the necessary understanding of the MARS scoring process as well as the individual items of the MARS scale. Finally, all of the raters were assigned all of the identified apps for rating purposes. Note that the quality rating of the MARS is based on a scale ranging from 1 to 5 points: 1=inadequate, 2=poor, 3=acceptable, 4=good, and 5=excellent. MARS further includes 19 items that are divided into 4 subscales, namely, engagement, functionality, aesthetics, and information quality. Additionally, MARS includes a fifth category, namely subjective, which is not included in the calculation of the final MARS score.
In order to evaluate MARS scores from the 4 raters, we calculated the interrater agreement based on Fleiss κ , the internal consistency was based on Cronbach α [ ], and the interrater reliability was based on Guttman λ6 [ ] as well as intraclass correlation—ICC(2,k) with 95% CI [ ].
Tinnitus Relief Using Smartphones
A comprehensive list of apps that assist patients (ie, for tinnitus-related relief) and that were identified through PRISMA is shown in, with their respective properties. Among the app properties, the downloads property provides insight into the apps’ usage, however, in the case of the iOS platform, the number of users is not publicly provided by the app store; rating provides a general understanding of the quality of the app based on the user opinion and is according to the app store’s rating system; update reports the last recorded update for the corresponding store; and for pricing, the app price is given, or if the app was free, further information is given.
A categorical distribution of the smartphone apps that were identified is depicted in. Two main categories were identified based on the app descriptions: tinnitus therapy (24 apps) and cognitive behavior therapy (10 apps). In the tinnitus-therapy category, 18 apps had the main focus of providing of sound therapy, including sounds for tinnitus masking (7 apps), sound habituation (4 apps), neuromodulation (4 apps), and distraction (3 apps). The remaining 6 apps of the tinnitus-therapy category were almost evenly distributed among zen therapy (1 app), notch therapy (1 app), game-based therapy (1 app), individual therapy (1 app), and tinnitus management (2 apps). Cognitive behavior therapy for tinnitus made up the other main category (self-help: 6 app; chatbots: 3 app; acceptance and commitment therapy: 1 app).
|Appa||Description||Platform||Downloads||Rating (out of 5)||Update||Pricing, € (US $)b|
|Beltone Tinnitus Calmerc||Combination of relaxation exercise and sound therapy to avoid tinnitus||Android||>1000||4.7||September 2019||In-app purchases|
|iOS||—||5.0||September 2019||In-app purchases|
|CBTd Companion||Employs visual tools to learn and practice CBT techniques||Android||>50,000||4.6||February 2019||In-app purchases|
|Diapason for tinnitusc||Game-based digital therapeutic providing app for tinnitus relief||Android||>5000||3.1||May 2019||In-app purchases|
|iOS||—||—||May 2019||In-app purchases|
|H & T Sound Therapy||Noise player (pink noise, white noise or brown noise) for masking tinnitus||Android||>10,000||4.3||October 2019||Free|
|Kalmeda mynoisec||Offers medically based individual tinnitus therapy||Android||>1000||3.0||July 2019||In-app purchases|
|iOS||—||3.6||July 2019||In-app purchases|
|MindShift CBTc||CBT tools to manage and control anxiety||Android||>100,000||3.9||October 2019||Free|
|Moodfit—Stress & Anxiety||Stress and anxiety management and tracking, and offers CBT exercises||Android||>5000||4.4||August 2019||Free|
|myNoisec||Controlling tinnitus via combination of different sounds and noises||Android||>100,000||4.4||March 2018||In-app purchases|
|iOS||—||4.6||April 2019||In-app purchases|
|Quirk CBT||Self-help CBT companion based on 3-column technique||Android||>10,000||3.6||July 2019||In-app purchases|
|iOS||—||4.7||September 2019||In-app purchases|
|Relax Melodiesc||Sleep assisting app that combines sounds and melodies||Android||>10,000,000||4.6||May 2019||Ad-supported, in-app purchases|
|iOS||—||4.8||May 2019||In-app purchases|
|Relax Noise 3c||Masking tinnitus by using red, white, or pink noise||Android||>100,000||4.2||March 2015||Free|
|ReSound Reliefc||Avoiding tinnitus using combination of sound therapy and relaxation exercise||Android||>100,000||4.5||February 2019||In-app purchases|
|iOS||—||4.7||January 2019||In-app purchases|
|Sanvello—Stress & Anxiety Help||Audio and video CBT exercises, anxiety tracking and management||Android||>1,000,000||4.6||February 2019||In-app purchases|
|iOS||—||4.8||Nov 2019||In-app purchases|
|SimplyNoisec||Controlling and managing stress and tinnitus using white and brown noise||Android||>50,000||3.7||June 2012||Free|
|iOS||4.4||May 2018||In-app purchases|
|Stress & Anxiety Companion||CBT based visual exercises to manage stress and anxiety||Android||>10,000||4.2||July 2019||In-app purchases|
|iOS||—||4.6||June 2019||In-app purchases|
|Starkey Relaxc||Tinnitus masking, self-management, and education app||Android||>10,000||4.3||October 2017||Free|
|StopTinnitusc||Masking tinnitus using customized tones||Android||>100||2.7||January 2015||7.95 (9.38)|
|iOS||—||1.3||January 2015||8.03 (9.48)|
|Tinnitracksc||Controlling and managing tinnitus by filtering out music for sound therapy||Android||>10,000||3.8||April 2019||In-app purchases|
|iOS||—||3.6||February 2019||In-app purchases|
|Tinnitus Balance Appc||Controlling annoying tinnitus using customized sounds or music||Android||>50,000||3.7||March 2016||Free|
|Tinnitus Helpc||Tinnitus masking using natural sounds or music||Android||>500||3.0||November 2015||9.90 (11.68)|
|iOS||—||4.4||January 2019||17.99 (21.23)|
|Tinnitus Notch||Provided custom tailored notch therapy for tinnitus relief||Android||>1000||2.7||September 2016||In-app purchases|
|Tinnitus Peace||Offers melodies to match the frequency of tinnitus to reduce its effects||Android||>5000||3.8||November 2015||Free|
|Tinnitus Relief||Tinnitus masking using headphones||Android||>10,000||4.2||December 2019||In-app purchases|
|Tinnitus Sound Therapy||Sound/acoustic therapy for masking tinnitus||Android||>10,000||3.9||June 2019||Free|
|Tinnitus Therapy (Lite)c||Avoiding tinnitus with sound masking and therapy||Android||>500||3.6||February 2019||6.49 (7.66)|
|iOS||—||5.0||March 2019||5.36 (6.32)|
|Tonal Tinnitus Therapyc||Helps to mitigate symptoms of tonal tinnitus based on acoustic neuromodulation||Android||>10,000||4.0||July 2018||In-app purchases|
|Track Your Tinnitusc||Managing tinnitus by tracking tinnitus patterns in daily activity||Android||>1000||2.1||October 2018||Free|
|What\'s Up? A Mental Health App||Offers CBT and ACTe methods to manage stress, anxiety as well as depression||Android||>50,000||4.4||June 2019||In-app purchases|
|iOS||—||4.6||December 2016||In-app purchases|
|Whistc||Controlling tinnitus using sounds with adjusted volume, pitch, etc||Android||>1000||4.2||March 2017||2.18 (2.57)|
|iOS||—||3.7||January 2019||1.78 (2.10)|
|White Noise (Lite)c||Masking and controlling tinnitus using environmental sounds||Android||>5000||4.6||September 2018||3.19 (3.76)|
|iOS||—||4.8||April 2019||2.67 (3.15)|
|Widex Zenc||Avoiding tinnitus using relaxing zen sounds and exercises to manage tinnitus||Android||>10,000||3.8||May 2017||Free|
|Woebot—Your Self-Care Expertc||A chatbot for guided-CBT to manage stress and anxiety||Android||>100,000||4.8||November 2019||Free|
|Wysa: Mental Health Therapyc||A chatbot offering CBT and DBTf techniques||Android||>1,000,000||4.7||November 2019||In-app purchases|
|iOS||—||4.7||December 2019||In-app purchases|
|Youper—Emotional Healthc||A chatbot based on CBT and ACT techniques monitoring and tracking mood changes||Android||>1,000,000||4.7||December 2019||In-app purchases|
|iOS||—||4.9||December 2019||In-app purchases|
aRetrieved December 15, 2019.
bAn exchange rate of €1 to US $1.18 is applicable.
cApps reported in literature.
dCBT: cognitive behavior therapy.
eACT: acceptance and commitment therapy.
fDBT: dialectical behavior therapy.
Evaluation of Tinnitus Relief Apps
The ratings of each individual rater and the mean of all 4 raters are depicted in the; it can be seen that evaluations are rather consistent between the 4 raters. Note, that MARS ratings range from 1 (inadequate) to 5 (excellent); however, none of the apps in our evaluation process scored less than 2. To ensure consistency between raters and internal consistency, as well as reliability, we performed statistical psychometric analyses ( ).
In addition to the objective MARS scores calculated using the arithmetic mean of 4 categories (engagement, functionality, aesthetics, and information quality), MARS guidelines also allow subjective scoring of the smartphone apps, reflecting individual rater opinion. In, the results of the subjective criteria of the MARS questionnaire are shown.
|Rating category||Fleiss κ||Cronbach α||Guttman λ6||ICC(2,k)a (95% CI)|
|Engagement||0.62 (substantial)||.93 (excellent)||0.93 (high)||0.92 (0.88-0.95) (excellent)|
|Functionality||0.52 (moderate)||.90 (excellent)||0.89 (high)||0.89 (0.82-0.93) (good)|
|Aesthetics||0.49 (moderate)||.96 (excellent)||0.97 (high)||0.94 (0.88-0.97) (good)|
|Information||0.78 (substantial)||.96 (excellent)||0.96 (high)||0.96 (0.94-0.98) (excellent)|
|MARSb score||0.74 (substantial)||.95 (excellent)||0.94 (high)||0.94 (0.91-0.97) (excellent)|
|Subjective score||0.45 (moderate)||.96 (excellent)||0.96 (high)||0.95 (0.92-0.97) (excellent)|
aICC: intraclass correlation.
bMARS: Mobile Application Rating Scale.
Additionally, we gathered information on the quality of the apps from various repositories. We used existing ratings both from established information platforms for health app quality ratings as well as from the Google Play Store and the iOS App Store (star ratings). The first information platform that we chose was PsyberGuide , which is a nonprofit website that is funded by One Mind and operated by Northwestern University. PsyberGuide’s app reviews consist of a credibility score that represents the research support, a user experience score that is based on MARS, and a transparency score that represents the app developer’s privacy information transparency. The second information platform that we chose was ORCHA [ ], an organization that offers evaluations of health apps and advice for governments, health, and social care organizations. ORCHA app reviews consist of a score that is a calculated mean of the 3 domains: data privacy, clinical assurance, and user experience plus a level that classifies the app in 1 of 5 levels based on their focus and functionality. This quality information is given in .
Since the objective MARS scores calculated in this paper and PsyberGuide and ORCHA ratings (apart from PsyberGuide’s user experience) are incomparable, a separate chart () depicts a comparison of MARS scores from our study with those from 2 papers from literature [ , ] and with PsyberGuide’s user experience score. The MARS scores in [ ] are for general tinnitus apps such as Beltone Tinnitus Calmer, Relax Noise 3, ReSound Relief, myNoise, Tinnitus Therapy (Lite), and White Noise (Lite), while MARS scores in [ ] are given for mindfulness and cognitive behavior therapy apps such as Relax Melodies and MindShift CBT.
|Credibility||User experience||Transparency||Score, %||Level||Android||iOS|
|Sanvello—Stress & Anxiety Help||4.29||4.79||Acceptable||N/Aa||N/A||4.6||4.8|
|Stress & Anxiety Companion||1.80||N/A||N/A||N/A||N/A||4.2||4.6|
|What’s Up? A Mental Health App||1.43||3.83||Unacceptable||51||3||4.4||4.6|
|Woebot—Your Self-Care Expert||4.64||4.38||Acceptable||N/A||N/A||4.8||4.7|
|Wysa: Mental Health Therapy||3.86||3.86||Acceptable||N/A||N/A||4.7||4.7|
|Tinnitus Therapy (Lite)||N/A||N/A||N/A||68||3||3.6||5.0|
|White Noise (Lite)||N/A||N/A||N/A||59||3||4.6||4.8|
|Beltone Tinnitus Calmer||N/A||N/A||N/A||54||3||4.7||5.0|
aN/A: not available.
A significant portion of tinnitus scientific literature [- ] reports on different smartphone and mobile crowdsensing apps to support clinicians in better understanding tinnitus, ranging from data collection to mitigating tinnitus symptoms via therapeutic interventions. These apps are specifically designed to assist patients, clinicians, and researchers alike. From the perspective of patients, these apps aim to provide the necessary means to mask, control, mitigate, or manage tinnitus symptoms. For example, the TrackYourTinnitus smartphone app systematically records data about fluctuations of tinnitus symptoms over time from patients, thus providing information about patient’s tinnitus variability [ ]. Similarly, Henry et al [ ] delve into the development of an app, based on progressive tinnitus management, to support patients in learning and using coping skills for tinnitus. From the perspective of clinicians and researchers, these apps support a better understanding of tinnitus, particularly in identifying symptom severity and tinnitus characteristics in different patients. For instance, the data collected from the TrackYourTinnitus app can be used to associate tinnitus with daily routines or activities [ ] or to shape recruiting strategies for tinnitus-related studies [ ]. Similarly, TrackYourTinnitus app was also used to better understand tinnitus variability and tinnitus loudness and stress associations [ ].
Generally, health care apps have been exhaustively discussed and reviewed in the literature ; however, the number of papers that focus on reviewing and evaluating tinnitus-related smartphone apps is underwhelming. Nevertheless, among pre-existing literature, some papers have discussed the role of tinnitus smartphone apps in clinical scenarios. For instance, internet- and smartphone-based solutions for treatment and management of tinnitus have been reviewed in [ ], similarly, the review by Kalle et al [ ] discusses internet- or smartphone-delivered cognitive behavior therapy with particular focus on self-help for tinnitus. Both of these papers demonstrate the role of several approaches and technologies involved in advancing tinnitus clinical practice but focus less on current and available apps for patients. Furthermore, the review by Lui et al [ ] addresses efficacy or effectiveness of therapeutic solutions provided by mental health apps and hearing health care apps have been discussed in [ ]. Comparatively, both [ , ] list a limited number of apps and many are no longer commercially available on the app stores. Specifically, in terms of the assessment of tinnitus-related smartphone apps using MARS, the reviews by Sereda et al [ ], which were further extended and repeated by Smith et al [ ], are the only closely related works in relation to this work. Both of these reviews [ , ] list tinnitus management apps based on patient opinions, gathered via a web-based survey. The added value of our review was primarily the exploration of smartphone app markets to reveal relevant apps as opposed to using a survey. Additionally, our proposed work also compares the star ratings and MARS scores with quality information gathered by third-party app assessment platforms.
A noticeable limitation of our work was the restricted search of relevant smartphone apps to only 2 app stores. Although the restriction was justified in the paper, it might be possible that there would be benefit in exploring other app stores, such as Amazon and Samsung app stores. Another possible limitation lay in the inclusion criteria for the apps. To include an app, we inspected the app description and a few top-rated comments from the users. Despite being effective and straight-forward, this approach is subjective and highly relies on the knowledge of the inspector about the domain. This limitation can be overcome or can be further improved by gathering additional opinions from domain experts.
In future research, the study will be extended and developed in 2 directions. First, additional app evaluation and assessment instruments (currently under development or newly developed) will be used to repeat the study. For example, recently, the THESIS app evaluation instrument was presented . Therefore, we intend to extend our current work by evaluating the identified apps using THESIS and comparing the results with those of MARS. This will include updates on the available and relevant apps in the app stores. Second, although the already developed instruments systematically and objectively measure the quality of mHealth apps from a user-experience perspective, the lack of instruments to clinically validate smartphone apps is undeniable. Therefore, we intend to further invest our efforts in this research direction. Additionally, as a consequence of this study, we learned that there exist 7 evidence-based tinnitus apps; therefore, we are currently working on a review paper detailing the scientific evidence of these 7 apps.
The aim of this study was to systematically identify smartphone apps within the context of tinnitus. The identification process yielded a total of 34 commercially available tinnitus smartphone apps, which were divided in 2 categories: tinnitus therapy (24 apps) and cognitive behavior therapy (10 apps). In an added step, we evaluated the identified apps using MARS. From MARS objective scores (), first, we can see that all 34 identified apps have MARS scores higher than 2, indicating that most of these apps provide some level of user experience and that they all have some functional value. Furthermore, the MARS rating process discovered that only 7 apps—Tinnitus Therapy (Lite), ReSound Relief, SimplyNoise, Audio Notch, Wysa, Woebot, and MindShift—out of 34 apps scored in the evidence-based subitem of the information category, suggesting a lack of clinical validation for most of the apps. Furthermore, the mean MARS scores ( ) of all 4 raters ranged from 2.65-4.60, with Tinnitus Peace having the lowest mean score (mean 2.65, SD 0.20) and Sanvello—Stress and Anxiety Help having the highest mean score (mean 4.60, SD 0.10). On the individual-rater level, Tinnitus Notch had the lowest (score: 2.39), while, Sanvello—Stress and Anxiety Help had the highest (score: 4.69). Furthermore, some apps were rated better in comparison to the others. For instance, Beltone Tinnitus Calmer, Resound Relief, Sanvello—Stress and Anxiety Help, as well as Woebot and Youper received very good ratings, whereas Tinnitus Help, Tinnitus Notch, and Tinnitus Peace were the worst of all. The mean subjective scores presented in ranged from 1.44-4.69, with Relax Noise 3 having the lowest scores (mean 1.44) and Beltone Tinnitus Calmer, Woebot, and Youper having the highest scores (mean 4.69). From , naturally, the range of the subjective scores was wider than those of the objective scores; however, it is notable that the mean subjective results were more or less in line with the mean objective results.
Values from the 4 psychometric measures () confirm the reliability and validity of our MARS rating procedures, especially for measures testing internal consistency and reliability (>0.89). The interrater agreement, as demonstrated by Fleiss κ, was merely moderate for subjective, functionality, and aesthetics scores. This is noteworthy and may be related to individual differences in the raters with regards to their backgrounds. In any case, from , , and , it is evident that our rating procedures generally produced valid, reliable, and thus viable results.
Interestingly, the expert ratings from both information platforms (PsyberGuide and ORCHA) and the user ratings from the app stores varied, sometimes considerably. For example, What’s Up? A Mental Health App received high ratings from its users in the app stores (Google Play Store: 4.4; Apple App Store: 4.6), whereas both expert ratings were considerably lower (PsyberGuide: credibility: 1.43, transparency: unacceptable, user experience: 3.38; ORCHA: 51%). Another app whose app store rating differed significantly from its ORCHA rating is Beltone Tinnitus Calmer (Google Play Store: 4.7; Apple App Store: 5.0; ORCHA: 54%). The ORCHA score was moderate as a result of a moderate rating for data privacy and clinical assurance; the apps hadn’t been rated by PsyberGuide. These examples illustrate that an independent rating with validated instruments is crucial for the informed selection of health apps.
From, it can be seen that the MARS scores in our work were more in line with those of PsyberGuide’s user experience score. Similarly, the differences between MARS scores from our paper and those from literature were evidently higher for all apps except myNoice, Relax Melodies, and White Noise (Lite). These differences were as a result of version changes in the apps. In our case, similar to PsyberGuide’s case, the MARS scores present the contemporary assessment of the most recent version of the apps, thus validating the need of an up-to-date MARS assessment of tinnitus apps.
All 34 identified apps obtained a MARS objective score higher than 2 (ranging between 2.65-4.60), indicating that they provide some level of user experience and at least some technical functional value for the user. Furthermore, in addition to presenting the objective MARS scores, the subjective MARS scores (ranging between the values of 1.44-4.69) were also discussed. The 4 psychometric measurements—Fleiss κ, Cronbach α, Guttman λ6, and ICC(2,k)—confirmed and depicted positive interrater agreement, internal consistency as well as reliability between the raters. The only exception was noticed in Fleiss κ with moderate values for subjective score, functionality, and aesthetics. The quality information of identified apps from PsyberGuide and ORCHA as well as the star ratings from the Google Play Store and the Apple App Store were compared. The quality information comparison exhibited incongruity. Finally, the comparison between MARS scores from this work and MARS scores of smartphone apps reported in previously published papers depicted a high coherence. Through these steps, we were able to comprehensively capture the wide array of heterogeneous apps for tinnitus and present an up-to-date assessment of identified apps.
This work highlighted the impact of smartphone apps, specifically within the context of tinnitus research. As a consequence, we demonstrated that there exists a plethora of smartphone apps utilized in supporting and controlling tinnitus symptoms, understanding tinnitus, and monitoring patients with tinnitus. Among the 34 identified apps, only 7 apps were evidence-based suggesting that the majority were in need of more stringent clinical validation.
This publication has emanated from research supported by funding from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement number 722064 (European School for Interdisciplinary Tinnitus Research).
MM undertook the database searches, cataloguing, conception, drafting, and revision of the paper. MS and AD helped with writing the Methods section, providing the app store scraper and app information database, and revising the manuscript. CR helped with Results section. PN and RP helped with writing Discussion section. CR, PN, WS, and MR helped by critically revising the manuscript. FH contributed by helping with writing the Introduction section, critical revisions, and final approval of the manuscript, as well as supervision of the entire work. All authors approved the submitted manuscript.
Conflicts of Interest
- Baguley D, McFerran D, Hall D. Tinnitus. Lancet 2013 Nov 09;382(9904):1600-1607 [FREE Full text] [CrossRef] [Medline]
- Mazurek B, Szczepek AJ, Hebert S. Stress and tinnitus. HNO 2015 Apr;63(4):258-265. [CrossRef] [Medline]
- Jastreboff P, Jastreboff M. Tinnitus Retraining Therapy (TRT) as a method for treatment of tinnitus and hyperacusis patients. J Am Acad Audiol 2000 Mar;11(3):162-177. [Medline]
- Probst T, Pryss R, Langguth B, Schlee W. Emotion dynamics and tinnitus: Daily life data from the. Sci Rep 2016 Aug 04;6:31166 [FREE Full text] [CrossRef] [Medline]
- Schlee W, Pryss RC, Probst T, Schobel J, Bachmeier A, Reichert M, et al. Measuring the Moment-to-Moment Variability of Tinnitus: The TrackYourTinnitus Smart Phone App. Front Aging Neurosci 2016;8:294 [FREE Full text] [CrossRef] [Medline]
- Ventola CL. Mobile devices and apps for health care professionals: uses and benefits. P T 2014 May;39(5):356-364 [FREE Full text] [Medline]
- Naslund JA, Aschbrenner KA, Barre LK, Bartels SJ. Feasibility of popular m-health technologies for activity tracking among individuals with serious mental illness. Telemed J E Health 2015 Mar;21(3):213-216. [CrossRef] [Medline]
- Nelson EC, Verhagen T, Noordzij ML. Health empowerment through activity trackers: An empirical smart wristband study. Computers in Human Behavior 2016 Sep;62:364-374. [CrossRef]
- Mehdi M. Smart mobile crowdsensing for tinnitus research: student research abstract. In: SAC '19: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing. In: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing. ACM; 2019 Presented at: SAC '19: The 34th ACM/SIGAPP Symposium on Applied Computing; April, 2019; Limassol, Cyprus p. 1220. [CrossRef]
- Mehdi M, Schwager D, Pryss R, Schlee W, Reichert M, Hauck F. Towards Automated Smart Mobile Crowdsensing for Tinnitus Research. In: Proceedings of the IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS).: IEEE; 2019 Jun Presented at: IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS); June, 2019; Cordoba, Spain p. 32. [CrossRef]
- Mehdi M, Mühlmeier G, Agrawal K, Pryss R, Reichert M, Hauck FJ. Referenceable mobile crowdsensing architecture: A healthcare use case. Procedia Computer Science 2018;134:445-451. [CrossRef]
- Kuhn E, Greene C, Hoffman J, Nguyen T, Wald L, Schmidt J, et al. Preliminary evaluation of PTSD Coach, a smartphone app for post-traumatic stress symptoms. Mil Med 2014 Jan;179(1):12-18. [CrossRef] [Medline]
- Kuhn E, Kanuri N, Hoffman JE, Garvert DW, Ruzek JI, Taylor CB. A randomized controlled trial of a smartphone app for posttraumatic stress disorder symptoms. J Consult Clin Psychol 2017 Mar;85(3):267-273. [CrossRef] [Medline]
- Economides M, Martman J, Bell MJ, Sanderson B. Improvements in Stress, Affect, and Irritability Following Brief Use of a Mindfulness-based Smartphone App: A Randomized Controlled Trial. Mindfulness (N Y) 2018;9(5):1584-1593 [FREE Full text] [CrossRef] [Medline]
- Van AM, Turna J, Khalesi Z, Pullia K, Patterson B. There is an app for that! The current state of mobile applications (apps) for DSM-5 obsessive-compulsive disorder, posttraumatic stress disorder, anxiety and mood disorders. Depress Anxiety 2017 Jun;34(6):526-539. [CrossRef] [Medline]
- Schmidt W, Sarran C, Ronan N, Barrett G, Whinney DJ, Fleming LE, et al. The Weather and Ménière's Disease: A Longitudinal Analysis in the UK. Otol Neurotol 2017 Feb;38(2):225-233 [FREE Full text] [CrossRef] [Medline]
- Delbaere K, Valenzuela T, Woodbury A, Davies T, Yeong J, Steffens D, et al. Evaluating the effectiveness of a home-based exercise programme delivered through a tablet computer for preventing falls in older community-dwelling people over 2 years: study protocol for the Standing Tall randomised controlled trial. BMJ Open 2015 Oct 22;5(10):e009173 [FREE Full text] [CrossRef] [Medline]
- Bright T, Pallawela D. Validated Smartphone-Based Apps for Ear and Hearing Assessments: A Review. JMIR Rehabil Assist Technol 2016 Dec 23;3(2):e13 [FREE Full text] [CrossRef] [Medline]
- Masalski M, Grysiński T, Kręcicki T. Hearing Tests Based on Biologically Calibrated Mobile Devices: Comparison With Pure-Tone Audiometry. JMIR Mhealth Uhealth 2018 Jan 10;6(1):e10 [FREE Full text] [CrossRef] [Medline]
- Kardous CA, Shaw PB. Evaluation of smartphone sound measurement applications. J Acoust Soc Am 2014 Apr;135(4):EL186-EL192 [FREE Full text] [CrossRef] [Medline]
- Dlugaiczyk J, Thiemer M, Neubert C, Schorn BA, Schick B. The aVOR App Increases Medical Students' Competence in Treating Benign Paroxysmal Positional Vertigo (BPPV). Otol Neurotol 2018 Jun;39(5):e401-e406. [CrossRef] [Medline]
- Bromwich MA, Parnes LS. The DizzyFix: initial results of a new dynamic visual device for the home treatment of benign paroxysmal positional vertigo. J Otolaryngol Head Neck Surg 2008 Jun;37(3):380-387. [Medline]
- Organ B, Liu H, Bromwich M. An iPhone-assisted particle repositioning maneuver for benign paroxysmal positional vertigo (BPPV): a prospective randomized study. J Am Board Fam Med 2015;28(1):118-120 [FREE Full text] [CrossRef] [Medline]
- Beauvais B, Rialle V, Sablier J. Myvigi: An android application to detect fall and wandering. 2012 Presented at: In: Proceedings of the sixth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies, UBICOMM; 2012; Barcelona, Spain p. 156-160.
- Sposaro F, Danielson J, Tyson G. iWander: An Android application for dementia patients. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.: IEEE; 2010 Presented at: Annual International Conference of the IEEE Engineering in Medicine and Biology; 2010; Buenos Aires, Argentina p. 3875-3878. [CrossRef]
- Baumel A, Faber K, Mathur N, Kane JM, Muench F. Enlight: A Comprehensive Quality and Therapeutic Potential Evaluation Tool for Mobile and Web-Based eHealth Interventions. J Med Internet Res 2017 Mar 21;19(3):e82 [FREE Full text] [CrossRef] [Medline]
- Wyatt JC. How can clinicians, specialty societies and others evaluate and improve the quality of apps for patient use? BMC Med 2018 Dec 03;16(1):225 [FREE Full text] [CrossRef] [Medline]
- BinDhim NF, Hawkey A, Trevena L. A systematic review of quality assessment methods for smartphone health apps. Telemed J E Health 2015 Feb;21(2):97-104. [CrossRef] [Medline]
- Nouri R, R Niakan Kalhori S, Ghazisaeedi M, Marchand G, Yasini M. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inform Assoc 2018 Aug 01;25(8):1089-1098. [CrossRef] [Medline]
- Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3(1):e27 [FREE Full text] [CrossRef] [Medline]
- Salazar A, de Sola H, Failde I, Moral-Munoz JA. Measuring the Quality of Mobile Apps for the Management of Pain: Systematic Search and Evaluation Using the Mobile App Rating Scale. JMIR Mhealth Uhealth 2018 Oct 25;6(10):e10718 [FREE Full text] [CrossRef] [Medline]
- Bardus M, van BSB, Smith JR, Abraham C. A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management. Int J Behav Nutr Phys Act 2016;13(1):35 [FREE Full text] [CrossRef] [Medline]
- Chavez S, Fedele D, Guo Y, Bernier A, Smith M, Warnick J, et al. Mobile Apps for the Management of Diabetes. Diabetes Care 2017 Oct;40(10):e145-e146. [CrossRef] [Medline]
- Grainger R, Townsley H, White B, Langlotz T, Taylor WJ. Apps for People With Rheumatoid Arthritis to Monitor Their Disease Activity: A Review of Apps for Best Practice and Quality. JMIR Mhealth Uhealth 2017 Feb 21;5(2):e7 [FREE Full text] [CrossRef] [Medline]
- Knitza J, Tascilar K, Messner E, Meyer M, Vossen D, Pulla A, et al. German Mobile Apps in Rheumatology: Review and Analysis Using the Mobile Application Rating Scale (MARS). JMIR Mhealth Uhealth 2019 Aug 05;7(8):e14991 [FREE Full text] [CrossRef] [Medline]
- Qu C, Sas C, Daudén Roquet C, Doherty G. Functionality of Top-Rated Mobile Apps for Depression: Systematic Search and Evaluation. JMIR Ment Health 2020 Jan 24;7(1):e15321 [FREE Full text] [CrossRef] [Medline]
- Alessa T, Hawley MS, Hock ES, de Witte L. Smartphone Apps to Support Self-Management of Hypertension: Review and Content Analysis. JMIR Mhealth Uhealth 2019 May 28;7(5):e13645 [FREE Full text] [CrossRef] [Medline]
- Sereda M, Smith S, Newton K, Stockdale D. Mobile Apps for Management of Tinnitus: Users' Survey, Quality Assessment, and Content Analysis. JMIR Mhealth Uhealth 2019 Jan 23;7(1):e10353 [FREE Full text] [CrossRef] [Medline]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010;8(5):336-341 [FREE Full text] [CrossRef] [Medline]
- Zhang H, She D, Qian Z. Android Root and its Providers: A Double-Edged Sword. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security.: ACM; 2015 Presented at: CCS'15: The 22nd ACM Conference on Computer and Communications Security; October 2015; Denver, Colorado, USA p. 1093-1104. [CrossRef]
- National Health Service (NHS) Apps Library Internet. NHS Apps Library - NHS. URL: https://www.nhs.uk/apps-library/ [accessed 2020-05-15]
- AppScript Internet. AppScript | Discover, Deliver & Track Digital Health. URL: https://www.appscript.net/ [accessed 2020-05-15]
- PatientView. MyHealthApps Internet. MyHealthApps - apps tried and tested by people like you. URL: https://myhealthapps.net/ [accessed 2020-05-15]
- Baxter C, Carroll JA, Keogh B, Vandelanotte C. Assessment of Mobile Health Apps Using Built-In Smartphone Sensors for Diagnosis and Treatment: Systematic Survey of Apps Listed in International Curated Health App Libraries. JMIR Mhealth Uhealth 2020 Feb 03;8(2):e16741 [FREE Full text] [CrossRef] [Medline]
- Vimeo, Inc. Vimeo Internet. Leanne Hides on Vimeo. URL: https://vimeo.com/244067710 [accessed 2020-05-15]
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977 Mar;33(1):159-174. [Medline]
- Tavakol M, Dennick R. Making sense of Cronbach's alpha. Int J Med Educ 2011 Jun 27;2:53-55 [FREE Full text] [CrossRef] [Medline]
- A basis for analyzing test-retest reliability. Psychometrika 1945;10:255-282. [CrossRef] [Medline]
- Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med 2016 Jun;15(2):155-163 [FREE Full text] [CrossRef] [Medline]
- PsyberGuide | A project of One Mind | A mental health App guide. One Mind. URL: https://onemindpsyberguide.org/ [accessed 2020-05-15]
- Rated Health Apps | Best Health Apps | ORCHA. ORCHA. URL: https://www.orcha.co.uk/ [accessed 2020-05-15]
- Weekly T, Walker N, Beck J, Akers S, Weaver M. A Review of Apps for Calming, Relaxation, and Mindfulness Interventions for Pediatric Palliative Care Patients. Children (Basel) 2018 Jan 26;5(2) [FREE Full text] [CrossRef] [Medline]
- Pryss R, Schlee W, Langguth B, Reichert M. Mobile Crowdsensing Services for Tinnitus Assessment and Patient Feedback. In: Proceedings of the 2017 IEEE International Conference on AI & Mobile Services (AIMS).: IEEE; 2017 Presented at: IEEE International Conference on AI & Mobile Services (AIMS); 2017; Honolulu, HI, USA p. 22-29. [CrossRef]
- Henry JA, Thielman E, Zaugg T, Kaelin C, Choma C, Chang B, et al. Development and field testing of a smartphone "App" for tinnitus management. Int J Audiol 2017 Oct;56(10):784-792. [CrossRef] [Medline]
- Probst T, Pryss R, Langguth B, Schlee W. Emotional states as mediators between tinnitus loudness and tinnitus distress in daily life: Results from the "TrackYourTinnitus" application. Sci Rep 2016 Feb 08;6:20382 [FREE Full text] [CrossRef] [Medline]
- Probst T, Pryss RC, Langguth B, Spiliopoulou M, Landgrebe M, Vesala M, et al. Outpatient Tinnitus Clinic, Self-Help Web Platform, or Mobile Application to Recruit Tinnitus Study Samples? Front Aging Neurosci 2017;9:113 [FREE Full text] [CrossRef] [Medline]
- Pryss R, Probst T, Schlee W, Schobel J, Langguth B, Neff P, et al. Prospective crowdsensing versus retrospective ratings of tinnitus variability and tinnitus–stress associations based on the TrackYourTinnitus mobile platform. Int J Data Sci Anal 2018 Mar 12;8(4):327-338. [CrossRef]
- Mosa ASM, Yoo I, Sheets L. A systematic review of healthcare applications for smartphones. BMC Med Inform Decis Mak 2012 Jul 10;12:67 [FREE Full text] [CrossRef] [Medline]
- Nagaraj MK, Prabhu P. Internet/smartphone-based applications for the treatment of tinnitus: a systematic review. Eur Arch Otorhinolaryngol 2020 Mar;277(3):649-657. [CrossRef] [Medline]
- Kalle S, Schlee W, Pryss RC, Probst T, Reichert M, Langguth B, et al. Review of Smart Services for Tinnitus Self-Help, Diagnostics and Treatments. Front Neurosci 2018;12:541 [FREE Full text] [CrossRef] [Medline]
- Lui JHL, Marcus DK, Barry CT. Evidence-based apps? A review of mental health mobile applications in a psychotherapy context. Professional Psychology: Research and Practice 2017 Jun;48(3):199-210. [CrossRef]
- Paglialonga A, Tognola G, Pinciroli F. Apps for Hearing Science and Care. Am J Audiol 2015 Sep;24(3):293-298. [CrossRef] [Medline]
- Smith S, Sereda M. Smartphone Apps for Tinnitus Management. The Hearing Journal 2020;73(2):20. [CrossRef]
- Levine DM, Co Z, Newmark LP, Groisser AR, Holmgren AJ, Haas JS, et al. Design and testing of a mobile health application rating tool. NPJ Digit Med 2020;3:74 [FREE Full text] [CrossRef] [Medline]
|ICC: intraclass correlation|
|MARS: Mobile Application Rating Scale|
|mHealth: mobile health|
Edited by G Eysenbach; submitted 24.06.20; peer-reviewed by T Alessa, F Palmieri; comments to author 11.07.20; revised version received 28.07.20; accepted 29.07.20; published 18.08.20
©Muntazir Mehdi, Michael Stach, Constanze Riha, Patrick Neff, Albi Dode, Rüdiger Pryss, Winfried Schlee, Manfred Reichert, Franz J Hauck. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 18.08.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.