Women’s menstrual cycles are important to their overall health [1,2] and are characterized by predictable and recurring symptoms. Continuous tracking of the menstrual cycle can aid in health management. Monitoring systems are needed to optimize menstrual health and provide easily accessible health information for women [3].

Mobile health (mHealth) apps facilitate personalized health monitoring and management [4], and menstrual apps are the most important mHealth apps for women. Such apps are typically highly accessible for most women and provide indicators relevant to various health domains [5].

Typically, menstrual apps also allow the user to log their symptoms, mood changes, and body temperature, and they visually represent statistical data via graphs and tables. Some apps offer professional-level information through communities and links, that is, they promote women’s health care by facilitating smooth communication with medical staff through information-sharing services [6]. The apps’ menstrual cycle tracking functions facilitate health care planning and management in various domains, including contraception, fertility, preparation with respect to pregnancy and ensuring adequate “menstrual supplies,” leisure activities, and travel [7].

Women use these functions for various purposes, such as tracking pregnancy, preventing pregnancy, and managing menstruation periods [8]. The functions desired by consumers depend on the intended purpose of the app. Currently, indicators to help consumers identify and select menstrual apps according to the desired functions are lacking [9]. Recently developed systems recommend apps based on consumer requirements [10,11]. Menstrual apps with various functions have been developed, and related research is being actively conducted. However, most of the existing studies are content reviews or expert evaluations [12-15]. Consumer-centered studies have also started to appear [5,16-18], but quality evaluations of menstrual apps remain scarce.

Menstrual apps are directly relevant to women’s health, so it is necessary for experts and health care providers to evaluate these apps [8]. Health care provider quality evaluations can contribute to the app development, which is important to ensure that consumers have access to high-quality apps [19]. Consumer quality evaluations provide feedback on apps, such as consumer preferences (eg, for easy-to-use content) [20,21]. Consistent use of an app is critical given the recurring nature of the menstrual cycle, but research on this topic is lacking from the developer’s standpoint. To promote sustained app use, evaluations from consumers and health care providers’ perspective are required. However, it is unclear whether current commercially available apps satisfy the quality standards of consumers and health care providers. If we find the differences between consumers and health care providers, they need to be discussed importantly to develop apps that can be used to promote women’s health that satisfy both consumers’ wants and health care providers’ needs.

This study examines currently available menstrual apps in term of their main contents and quality based on evaluations by health care providers and consumers. We also correlated the two sets of evaluation scores. The study’s findings could improve the utility and convenience of future mHealth apps.

We searched for keywords related to the development and evaluation of menstrual apps commonly included in previous studies, such as “period” and “menstrual cycle” in both English and Korean. The Google Play Store and the iOS App Store were searched from April 8 to April 15, 2021. Up to 150 apps were screened for each keyword. Since then, to secure the appropriated number of apps that can be statistically analyzed, the following app inclusion criteria have been set, based on previous studies [14,22,23]:

To examine the apps’ main contents, we performed a pilot study of 14 representative apps. The apps’ main contents were classified as follows: menstrual cycle management, education and knowledge, sharing information, and notifications. Frequency analysis was performed to determine the number of apps providing these functions.

The quality of 34 menstrual apps were evaluated using Mobile Application Rating Scale (MARS) and user version of the Mobile Application Rating Scale (uMARS). The MARS was developed to evaluate mobile apps; it is a reliable tool with a high internal consistency and interrater reliability [24]. The uMARS was subsequently developed, based on the MARS, to allow consumers to evaluate the quality of mHealth apps. uMARS has excellent internal consistency [25]. Both scales comprise the following five categories: engagement, aesthetics, functionality, information, and subjective quality (Table 1). The MARS has been used to evaluate various health care–related apps, such as apps related to chronic disease, COVID-19, and physical activity, as well as apps related to allergy, hepatitis treatment support, and breast cancer [14,15,19,22,23,26,27]. The uMARS has recently been used to evaluate various types of mHealth apps, including apps pertaining to weight loss and nutrition, rheumatic diseases, and the management of ankylosing spondylitis [28-31].

A total of 6 consumers completed the uMARS between July 9 and July 22, 2021, and 4 nurses (ie, health care providers) majoring in health care and working in medical centers completed the MARS between July 21 and July 30, 2021. Each evaluator was asked to use the app for more than 10 minutes every day and the evaluation was conducted in a blind test. The apps were randomly assigned to evaluators to prevent bias related to subjectivity. Each app was crossevaluated by at least two evaluators.

The MARS results represent health care providers’ perspectives, as stated above. The uMARS results, star ratings, and number of reviews were analyzed from the consumers’ perspective. After normalization, Pearson correlation was used to correlate app content, MARS and uMARS scores, star ratings, and reviews and to correlate perspectives of health care providers and consumers. We identified the top and bottom five apps based on the MARS and uMARS scores and compared app preferences between consumers and health care providers. P values <.05 were considered significant. R software (version 4.1.2; R Core Team,) was used for the analysis.

A total of 1127 menstrual apps were initially identified via the keyword search, and 34 apps (Android: n=28; iPhone: n=6) met all of the study criteria and were included in the final analysis (Figure 1).

The operating system (OS) of apps is linked with the app store, so that the app is updated simultaneously with updates of the OS [32]. Therefore, the app version and function may vary depending on the timing of the OS update. The number of reviews and star ratings differed among the apps in this study. For example, Bom Calendar had star ratings of 4.8 and 4.4 for the Android and iOS versions, respectively (23,437 and 76,258 reviews, respectively). In instances where the same apps were available for different OSs, each version was considered to be a unique app in the analysis.

Most apps (n=31, 91%) offered a menstrual cycle prediction function. Some apps (n=14, 41%) offered menstruation and fertility period notifications, while others had no specific functions (n=3, 9%). Most apps were confidential (n=29, 85%), allowed data export (n=28, 82%), and had a log-in function (n=25, 74%). However, few apps provided education or knowledge (n=10, 29%), screening-related information (n=2, 6%), or advice (n=4, 12%; Table 2).

The MARS and uMARS scores of all apps were obtained through the average of the evaluator’s evaluation scores. The average MARS and uMARS scores (ie, health care provider and consumer scores, respectively) were 3.06 (SD 0.62) and 3.33 (SD 0.57), respectively. The iPhone “Bom Calendar” app had the highest score for both the MARS (4.51, SD 0.22) and uMARS (4.23, SD 0.27). The Android “Period calendar—Women’s menstrual calendar❤” app had the second lowest score for both the MARS (2.05, SD 0.45) and uMARS (2.09, SD 0.05), despite its high star rating of 4.8. The Android “Menstrual calendar—ovulation & pregnancy calendar” app showed contrasting scores between the MARS (2.22, SD 0.07) and uMARS (4.15, SD 0.46); it had the third highest overall uMARS score, with an engagement score of 3.86 (SD 0.49), functionality score of 4.04 (SD 0.49), aesthetics score of 4.25 (SD 0.47), information score of 4.25 (SD 0.47), and subjective quality score of 4.37 (SD 0.41). However, for the MARS, its overall score was the third lowest, with an engagement score of 2.40 (SD 0.20), functionality score of 2.50 (SD 0.50), aesthetics score of 2.00 (SD 0.00), information score of 2.58 (SD 0.42), and subjective quality score of 1.63 (SD 0.38; Table 3 and Table 4). MARS and uMARS scores of all 34 apps are presented in Multimedia Appendix 1.

Table 5 shows the results of correlation analysis of the MARS and uMARS scores, star ratings, number of reviews, and app content. The number of reviews was not correlated with app content and menstrual cycle management (r=0.53; P=.001), and visualization (r=0.51; P=.002) had the highest correlation with star ratings. Among the evaluation scores, the highest correlation was found between uMARS and notification (r=0.39; P=.02) as well as between MARS and ovulation date management (r=0.49; P=.003).

Multimedia Appendix 2 shows content comparison between the top and bottom five apps in consumer and health care provider evaluation scores. Personalized alarms could be set in the top five apps. In addition, they provided a function to visualize all information through a calendar or to specify and manage ovulation days. On the contrary, in the bottom five apps did not have functions for managing or predicting the menstrual cycle.

Figure 2 shows the correlations among the MARS and uMARS scores, star ratings, and number of reviews to compare the perspective of health care providers and consumers. Figure 2 shows no correlation between MARS and uMARS scores of the health care providers and consumers (r=0.32; P=.06). uMARS scores and star rating (r=0.11; P=.54) as well as uMRAS scores and number of reviews (r=0.07; P=.67) also showed no significant correlations. The number of reviews and star rating (r=0.39; P=.02) showed a very low correlation.

Interest in mHealth apps has recently increased, and new apps are continuously being developed, including menstrual apps [33]. However, the needs of consumers and health care providers are different, and studies evaluating whether the available menstrual apps satisfy both of these groups are difficult to find. This study obtained quality evaluations of relevant apps from consumers and health care providers using the uMARS and MARS, respectively; the consistency of the evaluations of these two groups in terms of key app contents was analyzed. The health care providers valued engagement, functionality, and aesthetics when evaluating apps, while consumers valued aesthetics and information provision the most.

The MARS and uMARS scores were not correlated in this study. For example, the app with the third lowest MARS score had the third highest uMARS score. A significant difference was observed in app aesthetics scores between the consumers and health care providers; the scores for this attribute showed the largest group difference. Previous studies on health services have reported disparities between consumers and health care providers, and these results affect the implementation of consumer-centered services. Data from health care providers can provide a basis for high-quality apps [20], while consumer data serves as feedback on app quality [19]. To ensure high app quality and consumer satisfaction, app quality should be continuously monitored from the perspective of both health care providers and consumers. Monitoring can identify the needs of consumers and health care providers, which can in turn help in app development and update [10]. This study focused on evaluating app quality using MARS and uMARS for consistency, but could be extended to qualitative studies, including interviews, to collect in-depth answers in the future [34].

According to our findings, the uMARS (ie, consumer) scores, star ratings, and number of reviews were unrelated variables. The uMARS allows for direct assessment of mHealth apps and is a reliable measure of app quality [25]. However, reviews and star ratings are subjective indicators [24]. The currently available mHealth apps have been evaluated in a simplistic manner, such as through star ratings and reviews, even though they differ significantly with respect to content; thus, appropriate guidelines to aid app selection are lacking. By using the uMARS to guide app selection, the limitations of reviews and star ratings can be overcome such that consumers will likely select more useful apps to meet their particular needs. However, it is difficult for consumers to evaluate the quality of the apps using uMARS every time they download one. New indicators to guide app selection are needed so that consumers can make decisions based on objective evaluation results.

Most of the five apps that scored highly in the quality evaluation in this study included personalized monitoring functions. Menstrual apps are becoming increasingly popular, and apps that include self-monitoring functions and provide related information are continuously being developed [28]. Personalized monitoring can improve user well-being by encouraging them to check for signs and symptoms of health issues. Health-related information can also promote consumer health. mHealth apps that include personalized content of this nature are particularly useful for consumers [23,35]. However, only a few of the apps evaluated in this study facilitated consultations with specialists or provided information relating to women’s health. To increase the utility of apps, notifications, symptom recording functions, and the provision of knowledge should be prioritized.

mHealth apps should provide customized content for individual consumers [36]. However, the five bottom-scoring apps in this study did not meet the needs of the women who were using them. Consumers use apps to predict menstrual cycles and ovulation dates, and to monitor their general health [37]. However, most of the five bottom-scoring apps did not provide content enhancing consumer convenience, such as functions for menstrual cycle and ovulation day management, and some apps also lacked predictive functions. Such apps must be updated to include content allowing for the prediction and management of menstrual cycles based on accumulated menstrual cycle- and health-related information.

Menstrual apps collects personal information from consumers, such as name, date of birth, menstrual cycle, and medical history [38]. Personal data must be protected because it is sensitive information [12,39], but some apps do not provide locking functions, and few apps provide icon change functions for protecting personal information. Most fitness apps that record the number of steps do not consider privacy issues, and the data protection of mHealth apps related to women’s health is typically poor [39]. Therefore, regulations pertaining to app management of private data are necessary [40,41]. In fact, there are existing regulations protecting personal information, such as the European Unions’s General Data Protection Regulation, but no standard regulations are enforced worldwide. mHealth apps that protect personal information tend to be favored by consumers [21]. App developers should improve data protection–related functions to protect the personal information of consumers.

The MARS and uMARS were developed specifically for evaluating mHealth apps that aim to improve consumers’ health. Meanwhile, menstrual apps were designed to help consumers keep track of their current health rather than improve it. Health apps must provide solutions customized to individual consumers [36]; reliability may be key in this respect [42]. This study’s results indicate that current health apps do not fully meet consumers’ requirements or desires with respect to content. To better identify consumer objectives and take account of them during the development of menstrual apps, a new evaluation scale is needed to evaluate menstrual apps.

This study had several limitations. First, the results cannot be generalized to all menstrual apps because a small number of evaluators evaluated only the most popular apps. Second, the apps were selected based on App Store searches with a limited timeframe. Updates to apps may result in differences between the analyzed content and that in the future. Third, the database is not an electronic database but the App Store. The App Store’s app recommendation function may have compromised an inconsistent search accuracy.

In this study, consumer and health care provider ratings of menstrual apps were obtained using validated scales. Consumer preferred app had high scores of aesthetics and information, and evaluation scores differed between consumers and health care providers. The findings highlight the importance of consumer participation in menstrual app development and evaluation. This study is significant in that it is the first to compare health care providers’ and consumers’ menstrual app quality ratings. We expect our results to guide future mHealth app development and provide consumers with information on menstrual app content and quality. To provide high-quality apps for consumers, continuous quality evaluation research needs to be conducted, and the perspectives of both consumers and health care providers should be taken into account.