Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 11.02.20 in Vol 8, No 2 (2020): February

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/14661, first published Jun 01, 2019.

This paper is in the following e-collection/theme issue:

    Original Paper

    International ResearchKit App for Women with Menstrual Pain: Development, Access, and Engagement

    1Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Institute for Social Medicine, Epidemiology and Health Economics, Berlin, Germany

    2NICM Health Research Institute, Western Sydney University, Sydney, Australia

    3Center for Integrative Medicine, School of Medicine, University of Maryland, Baltimore, MD, United States

    4Institute of Medical Sociology, Centre for Health and Society, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany

    5School of Chinese Medicine, China Medical University, Taichung, Taiwan

    6Department of Chinese Medicine, China Medical University Hospital, Taichung, Taiwan

    7Integrative Medicine and Acupuncture Division, University Hospital, Federal University of Santa Catarina, Florianópolis, Santa Catarina, Brazil

    8Institute for Complementary and Integrative Medicine, University Zurich and University Hospital Zurich, Zurich, Switzerland

    *these authors contributed equally

    Corresponding Author:

    Daniel Pach, MD

    Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Institute for Social Medicine, Epidemiology and Health Economics

    Luisenstraße 57

    Berlin, 10117

    Germany

    Phone: 49 30450529011

    Email: daniel.pach@charite.de


    ABSTRACT

    Background: Primary dysmenorrhea is a common condition in women of reproductive age. A previous app-based study undertaken by our group demonstrated that a smartphone app supporting self-acupressure introduced by a health care professional can reduce menstrual pain.

    Objective: This study aims to evaluate whether a specific smartphone app is effective in reducing menstrual pain in 18- to 34-year-old women with primary dysmenorrhea in a self-care setting. One group of women has access to the full-featured study app and will be compared with 2 control groups who have access to fewer app features. Here, we report the trial design, app development, user access, and engagement.

    Methods: On the basis of the practical implications of the previous app-based study, we revised and reengineered the study app and included the ResearchKit (Apple Inc) framework. Behavior change techniques (BCTs) were implemented in the app and validated by expert ratings. User access was estimated by assessing recruitment progress over time. User evolution and baseline survey respondent rate were assessed to evaluate user engagement.

    Results: The development of the study app for a 3-armed randomized controlled trial required a multidisciplinary team. The app is accessible for the target population free of charge via the Apple App Store. In Germany, within 9 months, the app was downloaded 1458 times and 328 study participants were recruited using it without external advertising. A total of 98.27% (5157/5248) of the app-based baseline questions were answered. The correct classification of BCTs used in the app required psychological expertise.

    Conclusions: Conducting an innovative app study requires multidisciplinary effort. Easy access and engagement with such an app can be achieved by recruitment via the App Store. Future research is needed to investigate the determinants of user engagement, optimal BCT application, and potential clinical and self-care scenarios for app use.

    Trial Registration: ClinicalTrials.gov NCT03432611; https://clinicaltrials.gov/ct2/show/NCT03432611 (Archived by WebCite at http://www.webcitation.org/75LLAcnCQ).

    JMIR Mhealth Uhealth 2020;8(2):e14661

    doi:10.2196/14661

    KEYWORDS



    Introduction

    Background

    In recent years, increasing smartphone access has enabled the advancement and widespread use of smartphone apps [1,2]. Apps are a promising tool for people with a wide variety of health conditions and may be particularly useful to guide and support individuals in the self-management of these conditions [3,4]. A recent systematic review on apps in pain management concluded that apps might be beneficial for patients, particularly in an outpatient setting, but that there is a need for more scientific knowledge [5]. Furthermore, in an Australian national survey on mobile health (mHealth) in women with polycystic ovary syndrome [6], current evidence-based information was considered to be the most desirable app feature. Thus, an app with evidence-based information on menstrual pain might be of great value for patients suffering from this common problem.

    Menstrual disorders are highly prevalent among women of reproductive age, and especially in young women; they commonly include period pain and mood disturbances [7]. Primary dysmenorrhea is defined as menstrual pain in the absence of underlying pathology, with the pain commonly starting within 3 years of menarche (the first menstrual period) [8]. A characteristic symptom of primary dysmenorrhea is crampy, colicky spasms of pain below the belly button, occurring within 8 to 72 hours of menstruation and peaking within the first few days as menstrual flow increases [9]. Many women with dysmenorrhea also experience other menstrual-related symptoms such as back pain, headaches, bowel changes, nausea, and vomiting [9]. Primary dysmenorrhea has significant negative impacts on education [7] and productivity at work [10]. Current menstrual health literacy and understanding of effective self-care strategies for menstrual symptoms are often poor [11].

    In a previous randomized pragmatic trial (trial registration: ClinicalTrials.gov NCT01582724) [12] for women with menstrual pain, a total of 221 women were randomly assigned to 1 of 2 study groups. Both groups received the study app and a short introduction by a health care professional. Although the intervention group had access to acupressure-based features, including visual and written instructions on how to apply self-acupressure before and during menstruation, the control group did not. In addition, the app could send regular reminders to start the acupressure or to fill in questions. For both groups, the app was used to collect the study-related data and support the management of the menstrual period with a simplistic period calendar. Users in the self-acupressure group reported a significant reduction in the mean pain intensity and reported less pain medication intake in comparison with the usual care control group. In addition, two-thirds of the women still used the app and continued to apply self-acupressure after 6 months [12]. Owing to the fast-developing mHealth technology, it was difficult to keep these noteworthy study results relevant for actual implementation. This was, in part, because of the user experience and because the underlying technology soon became outdated. Therefore, a complete modernization and reengineering of the app and the development of a new corresponding trial that examines its effect over a longer duration than undertaken in the initial trial were necessary.

    In 2015, Apple Inc introduced ResearchKit as an open-source framework to support clinical researchers conducting structured mobile app–based health studies [1]. This free and reusable framework can simplify the integration of patient recruitment, the consent process, and the data collection in an mHealth study app. A modernization and reengineering of the previous study app using the ResearchKit framework, new software tools, and design guidelines for broader functionalities and an up-to-date interface would allow to verify the study results from our previous trial on a larger scale and in a real-life self-care setting in several different countries across the world. To our knowledge, no ResearchKit app-based interventional studies have been previously conducted targeting women with menstrual pain. By implementing this ResearchKit app, it would be possible to improve self-care for menstrual pain by encouraging users to change their behavior and regularly apply self-care activities, such as exercise, yoga, or self-acupressure.

    Michie et al defined the smallest, observable, replicable intervention component with the potential to bring about change in behavior as behavior change techniques (BCTs) [13]. BCTs have been widely applied in electronic health interventions. A prior ResearchKit app-based observational study evaluated the decision making in patients with acute anterior cruciate ligament ruptures [14] and suggested that it might be possible to maintain users’ motivation by providing instant feedback and relevant treatment information. In another study aimed at reducing alcohol consumption via an app [15], self-monitoring, goal setting, action planning, and feedback in relation to goals were identified as BCTs with the greatest potential to reduce alcohol use. A review on apps targeting persons with poor control of type 2 diabetes mellitus also suggests that the majority of BCTs employed are those for the promotion of self-regulatory behavior [16]. However, there is a lack of data for expert validation of BCTs implemented in apps for menstrual pain.

    From the recruitment perspective, previous ResearchKit-based studies predominantly used Web-based recruitment. Web-based recruitment has the potential advantage of reaching a broader population quickly, whereas conventional recruitment is usually time consuming and costly. However, the broad reach can potentially bring in people who are not the target population of a particular mHealth study [17]. In an interventional ResearchKit study, enrollment before eligibility screening (number of App Store visits and downloads) and after baseline questions are indicators for user engagement. However, this important measurement has not been widely reported in previous mHealth studies yet.

    Objectives

    To address the questions raised above and to gain a greater understanding for conducting mHealth trials, we report the development, user access, and user engagement of our ResearchKit-based study app for an ongoing pragmatic randomized controlled trial (RCT) [18] on menstrual pain.


    Methods

    Study App and Study Design

    Technical Development of the Study App

    The development of the app was started with the aim to modernize the design and technology of the study app AKUD (2012-2015) for a new 3-armed study in a self-recruitment self-care setting.

    The study app Luna. (Luna, period) was developed in a collaborative project by the Institute of Complementary and Integrative Medicine of the University of Zurich, Switzerland, the Institute for Social Medicine, Epidemiology and Health Economics, Charité – Universitätsmedizin Berlin, Germany, and Smart Mobile Factory, Berlin, Germany, based on Apple’s ResearchKit modular concept. The app was coded in Swift 4 with initial full support for English and German and prepared for easy deployment of other languages, such as simplified and traditional Chinese. The design followed the iOS Human Interface Guidelines (2017) and targets young women. The team involved in the development included iOS and back-end developers, designers, medical doctors, public health researchers, psychologists, and experts on integrative medicine and health.

    Behavior Change Techniques in the Study App

    The development of user interaction and feedback wording was based on the previous app. However, during the development of the new app, we used the BCT taxonomy (BCTTv1), according to Michie et al [19], to document BCTs employed in the app. For example, the BCT goal setting was implemented to promote the goal of completing certain self-care activities. In addition, bar charts that recorded change in pain and activities were set up based on the BCT self-monitoring. The app was developed in English. During the adaption to German and Chinese, the content of the app was always translated with care to ensure that the respective underlying BCTs were not affected.

    For the scientific description of an mHealth intervention, a proper description of BCTs implemented in the app is important. For this, expert validation is essential. At a later stage after the app development was completed, 2 psychologists who were not part of the development team independently rated the individual app features to validate the proper use of BCTs according to the BCTTv1 [19]. We compared the list of BCTs (that were intended to be implemented in the app) of 1 app development team member with the rating results of these 2 psychologists. Where there was disagreement regarding which BCT was used in the app, a final agreement was reached in a consensus meeting between the 3 raters (Table 1).

    Table 1. App features and corresponding behavior change techniques implemented.
    View this table
    Privacy and Data Security

    Privacy and data security were considered high priorities during app development. User data collected by the app are encrypted and transferred anonymously. We adhere to the principle of data minimization [20] and collect only data that are absolutely necessary to answer the research questions. Personally identifiable information (PII), such as the name and signature collected during the informed consent procedure provided by Apple ResearchKit, is stored only on the user’s iPhone and will not be sent to the back end. The individual person owning the iPhone (the study participant) will not be identifiable by the data transferred to the study server. A token will be created as an identifier to label the individual study data. Moreover, an app passcode is implemented to avoid unintended access to the app. Collection of information by the app can be stopped at any time by withdrawing from the study, using a specific button in the app’s settings, and uninstalling the app. Data will be collected anonymously. In addition, the study team of the coordinating office in Germany is supervised by the data protection officer of the Charité—Universitätsmedizin Berlin. The other participating centers are supervised by their respective institutions.

    Study Design

    We will conduct a 3-armed, randomized pragmatic trial [18,21] to evaluate whether the smartphone app is effective in reducing menstrual pain in 18- to 34-year-old women with primary dysmenorrhea. We will compare the group of women who has access to the full-featured study app with 2 control groups who have access to fewer app features. After within-app verification of eligibility for the study, eligible women will be randomly allocated to one of the 3 groups in a 1:1:1 ratio. The potential group allocations are as follows: full-featured app version (self-care information + self-acupressure feature), control intervention I (only self-care information feature), or control intervention II (only self-acupressure feature). The app contains the interventions for all 3 groups, but the content is only unlocked and presented to the user depending on their group allocation. Study participants can use the app for the whole study duration of 12 menstruation cycles. The primary outcome of the study is the mean pain intensity measured with the in-app numerical rating scale (NRS) ranging from 0, no pain, to 10, most intense pain imaginable, on the painful days during the sixth menstruation after starting the intervention (approximately 6 months from trial start depending on cycle length). It will be calculated by adding up the daily values from the start of the menstruation until the end of bleeding and then dividing them by the number of days with available values. NRS is a common measure of pain intensity that has been utilized in many previous studies [22-24], including studies of menstrual pain [25,26]. Secondary outcome measures are described in more detail on ClinicalTrials.gov (NCT03432611).

    The decisions on study design of this trial are based, in part, on decisions of the stakeholder advisory group from the corresponding previous trial and its results [12]. As no member of the study team was specialized in gynecology, this expertise was represented by a gynecologist appointed to the advisory group. Our stakeholder advisory group included a female gynecologist, a 16-year-old woman with dysmenorrhea, a female teacher, 2 acupuncture experts, and a mind-body medicine expert [27].

    Intervention Components

    Furthermore, 5 days before the anticipated start of the menstruation until the end of bleeding, notifications from the app will remind all the groups of participating women to complete questions and perform self-care activities, such as self-acupressure or yoga, depending on the group allocation.

    The self-care feature will offer information on self-care for menstrual pain, including evidence-based information about exercise, nutrition and dietary supplementation, heating pad/hot water bottle, yoga, and when to consult a doctor and regarding how primary dysmenorrhea is treated in most cases (see Multimedia Appendix 1).

    The acupressure feature will offer detailed written and multimedia descriptions of the acupressure to be used for menstrual pain (see Multimedia Appendix 2). A total of 3 acupressure points will be described that should be massaged bilaterally, if possible, twice a day, up to 5 times per day, starting from 5 days before menstruation until the end of menstruation. Each point should be massaged for 1 min (ie, altogether 6 min should be spent for 1 acupressure session). A visual timer for the acupressure will indicate desirable length of acupressure. In addition, an in-app notification on the app’s dashboard will remind users to practice acupressure during painful days at least twice daily.

    The acupressure intervention resulted from a written consensus process with international acupuncture experts from China, Germany, and the United States of America [27] and was already evaluated in an RCT previously conducted by our group demonstrating effectiveness of the intervention [12]. The acupuncture points SP6 (Sanyinjiao), LI4 (Hegu), and LR3 (Taichong) were chosen during this process.

    Participants are allowed to continue with their own usual care (medical and nonmedical) during the study.

    Participants and Group Allocation

    We aim to recruit 594 young women with primary dysmenorrhea. The sample size estimation is based on the comparison of the group receiving the full-featured app (self-care information + self-acupressure) with the group receiving the app version without the self-care information (control intervention II) regarding the primary outcome (NRS after 6 menstrual cycles) that will be treated as a continuous variable. Our previous study showed a mean group difference of 1.4 on the NRS and a standard deviation of 2.15 at the sixth menstrual cycle after the onset of the trial.

    Assuming that self-care information has a smaller impact on pain than acupressure, we hypothesize a difference of 0.8 on the NRS between groups. To detect a mean difference of 0.8 point on the NRS after 6 menstrual cycles between the group receiving the full-featured app (with a common standard deviation of 2.15 observed in our previous study) and control intervention II, applying a 2-sided t test with a power of 80% and an adjusted alpha of .025, a total of 139 participants will be needed per group (417 women for the 3 arms together). Taking into account a dropout rate of approximately 30% (based on our previous study after 6 cycles), 198 participants per group will be needed (total 594 women).

    The eligibility criteria resemble the criteria of our previous study. Women owning an iPhone will be included if they have primary dysmenorrhea, are between the ages of 18 and 34 years, report moderate or severe menstrual pain ≥6 on the NRS; 0=no pain at all, 10=most intense pain imaginable), and report no existing or planned pregnancy within the next 12 months. During the app-based eligibility screening, the inclusion and exclusion criteria will be assessed by 12 compulsory eligibility questions (Table 2). After the determination of eligibility and obtaining informed consent, participants will be asked to complete the baseline survey before they receive access to the app features depending upon the respective study group allocation. We will use a server-based randomization table created by a statistician using the RANUNI random number generator of the SAS/STAT version 9.2 (SAS Inc) [28]. Participating women will be randomized in a 1:1:1 ratio by block randomization with a fixed block length.

    Table 2. Eligibility questions.
    View this table
    Efficacy-Effectiveness Continuum

    From a methodological point of view, a clinical trial provides more evidence on the effectiveness of an intervention using a pragmatic trial design or on the efficacy side using an explanatory trial design [29,30]. Pragmatic trials are usually considered to study interventions in a real-world setting, whereas explanatory trials are usually designed to investigate interventions in an ideally controlled setting. The PRECIS-2 is a wheel-format tool that helps researchers to consider trial design as more effectiveness or efficacy focused including 9 domains: eligibility criteria, recruitment, setting, organization, flexibility (delivery), flexibility (adherence), follow-up, primary outcome, and primary analysis [31]. The PRECIS-2 scoring system ranges from 1 (most explanatory) to 5 (most pragmatic).

    During the design phase of the trial, PRECIS served as a tool to make better informed design decisions [32]. We used the PRECIS-2 tool to assess our app-based trial’s positioning on the pragmatic-explanatory continuum. The authors independently scored the 9 dimensions.

    User Enrollment

    The primary recruitment strategy focuses on self-referral through the Apple App Store. On the basis of our experience from the previous trial and the associated stakeholder engagement [12,27], we anticipate that an app-based study for menstrual pain would meet wide acceptance among young women in Germany. Furthermore, we assume that no external advertising (such as posters in public transport or on campus) will be needed for recruitment. A Web-based press release on the Charité university homepage was published on February 28, 2018, (in German and English language), highlighting the results of the previous trial, while also mentioning the new study with the updated app, including a link to the App Store. The media coverage of the app is observed regularly by the study team, using Google search with keywords “selfcare + period pain + Luna,” “selfcare + Luna,” “app + period pain,” “acupressure + period pain” (in German: “Selbsthilfe + Regelschmerzen + Luna,” “Selbsthilfe + Luna,” “app + Regelschmerzen,” “Akupressur + Regelschmerzen”).

    The app use will be free of charge; no financial compensation will be provided for participating in the study.

    Potential future recruitment strategies will include traditional and Web-based recruitment methods that are also adapted to the respective study sites. These will include information about the ongoing study on printed posters or information leaflets or in social media. In addition, if accepted by the Apple App Store editorial team, we will inform potential users about the study app with the App of the day feature option of the Apple App Store for the category Health and Fitness.

    User Engagement

    When users install and open the study app for the first time, they will be briefly introduced to the study and encouraged to participate. For potential participants who wish to continue, an app-based anonymous eligibility screening and more detailed information about the study will be provided. After the consent process, participants will finish the app-based baseline survey to unlock the intervention interface. This process is based on the onboarding process of Apple’s ResearchKit framework [33]. User flow and conversion rates will be calculated based on the number of downloads, the number of eligible users, and the number of users who finish the baseline survey and enter the study.

    In the baseline survey, general information relevant for menstrual pain will be assessed, such as age, education, individual exercise behavior, length of period and level of pain experienced during the period, and use of hormonal contraceptives and pain medications (Table 3). A skip button is available for a selection of questions and allows users to skip questions they do not want to answer. User engagement will be measured by usage of skip button and baseline survey respondent rate.

    Table 3. Baseline questions.
    View this table

    Statistical Analysis

    The PRECIS-2 score was calculated by summing up the means of each dimension based on the rating results of 11 raters; meanwhile, standard deviations were calculated to show the variability.

    For the BCT ratings, the interrater reliability among BCT raters was assessed by intraclass correlations (ICCs) [34,35].

    For the assessment of user access, we used the data generated by App Analytics [33] (Apple Inc) to descriptively show the source of product page views and number of downloads.

    To assess user engagement, the user conversion rate and the baseline survey response data were calculated using descriptive statistics (frequencies, percentages, means, and standard deviations). The baseline survey variables were extracted from the back-end database and only the missing values of the baseline survey (skipped questions) were used for the calculation of the proportion of actually skipped questions among all skippable questions to interpret the user engagement.

    All collected data were analyzed with SPSS version 22.0 (SPSS Inc).

    Ethics

    The app is prepared for international use and can be currently (October 2019) downloaded in the German App Store and will later be made available in the App Stores of the other participating centers. The study database, the app server, and the primary study center are based in Berlin, Germany. The study was approved by the university’s ethics committee (Charité—Universitätsmedizin Berlin approval Number EA1/364/16). The trial was registered at ClinicalTrials.gov (NCT03432611).

    The participation of the study sites in Taichung, Taiwan (approval letter Number CMUH107-REC1-120 by Ethics Committee of China Medical University and Hospital); Sydney, Australia (approval number H13175 by Western Sydney University Human Research Ethics Committee); Florianopolis, Brazil (approval number 3.583.066 by Ethics Committee of Federal University of Santa Catarina), and Baltimore, United States is currently being processed.


    Results

    Study App and Study Design

    The study app is a result of multidisciplinary efforts. The launch of the study app in the App Store will mark the beginning of the fully app-based study: users will be recruited via the Apple App Store, eligibility and consent will be processed by the study app, different self-care interventions will be guided by corresponding app features, and the follow-up will be recorded by app-based survey questions (Figure 1). Detailed screenshots, which depict the user flow in more detail, are listed in Multimedia Appendix 3.

    The study app will display the intervention components (self-acupressure and self-care information) selectively according to the group allocation. The core features Dashboard, Journal, and More (Figure 2) will be accessible to all users. The Dashboard will display feedback according to study progress and answers of survey questions and a prediction of the next period start date. The Journal feature will contain a period calendar and an overview of the progress on the survey questions. With the More feature, users will be able to set personal identification number lock and notification time. Users’ cycle information, the signed consent form, and a link to the privacy policy will also be displayed there.

    Figure 1. Study design. ITT: intention to treat; PP: per protocol.
    View this figure
    Figure 2. Screenshots of the study app.
    View this figure
    Behavior Change Technique Ratings

    To validate whether the BCTs implemented in the app were properly applied, a developer rating (JW) was compared with ratings of 2 psychologists with BCT expertise (CRP and AR) who had experienced the finalized full-featured app but who had not been part of the app development process. The interrater agreement between the 2 psychologists showed an excellent ICC (ICC=0.954; 95% CI 0.87-0.98). However, the overall interrater agreement including all raters was poor (ICC=0.442; 95% CI 0.07-0.78), that is, the ratings of the BCTs used during the development by the study team, did not correspond well with the ratings of the 2 psychologists. There was no significant difference between ICCs at the item level and the cluster level based on the BCTs taxonomy (v1) [19]. The final agreement that was reached in a consensus meeting is shown in Table 2. Overall, 12 BCTs were identified in the study app. The most frequently implemented BCTs are prompts/cues (5 times), instructions on how to perform behavior (4 times), and demonstration of the behavior (3 times).

    Efficacy-Effectiveness Continuum Rating

    On the basis of the rating results of all authors, all 9 dimensions of the PRECIS-2 tool are defined more on the pragmatic side (Figure 3). Thus, this app-based RCT can be considered as a pragmatic trial.

    Figure 3. PRECIS-2 rating results of the study design.
    View this figure

    User Enrollment

    Trial recruitment started in February 2018 with the launch of the ResearchKit-based study app in the German App Store. The Web-based press release was well received by the public and the media. By observation of media coverage via Google search during the following 10 weeks, 65 articles or blog entries of pharmacy or health-related websites citing the press releases in English and German could be detected. Overall, 2 printed newspapers reported about this app-based study in German. An increase of media coverage could be observed from March to May 2018. In the weeks following the press release, the app showed continuous increase in both downloads and the number of users (Figure 4).

    After 38 weeks in the app store (from February 19, 2018, to November 13, 2018), there were 1458 downloads and 328 users were included into the study (22.5%). On average, we recruited around 8 study participants per week with a peak between May and June after the press release (22 new users per week). Approximately 60% (195/328) of the participants were recruited within these 2 months.

    Figure 4. App downloads and new users per day.
    View this figure

    User Engagement

    During the first 38 weeks of recruitment, the App Store’s preview of the app was viewed 1885 times. Although 75% of the app’s product page viewers found the app by searching the App Store, 25% found the app by App Store browsing, app referral, or Web referral. The app was downloaded 1458 times. A total of 388 (27%) users passed the 12-question eligibility screening and agreed to consent; 328 of the 388 users (85%) completed the 16-question baseline survey and were recruited to the study. Figure 5 displays the user evolution [14].

    For 11 of 16 baseline questions, the skip button can be used because these questions are either not related to the primary outcome of the study or their data are not essential for the proper functioning of the app. The usage of the skip button of a study sample (328 users) was calculated to evaluate the user engagement in the app-based survey.

    Almost all questions of the baseline survey were answered (data completeness of 98.27%; 5157/5248). A total of 276 users (276/328, 84.1%) answered all 16 baseline questions and never used the skip button. Only 3% of the data based on the skippable questions were missing. The question asking for discomfort/symptoms during the period was answered by all users (response rate 100%). For free-text fields, 105 (105/328, 32.0%) of the users provided details about their discomfort/symptoms during their period; 269 (269/328, 82.0%) users provided details about their medication for the question asking about the period pain-related medical history.

    Figure 5. User evolution.
    View this figure

    Discussion

    Principal Findings

    By using the ResearchKit framework, we successfully developed a study app for a fully app-based pragmatic RCT for young women with primary dysmenorrhea. The app is easily accessible via self-referral and can be used as a self-care and study tool for a highly relevant condition. The available data already indicate a high level of user engagement with the study app. We also realized that the early involvement of behavioral science experts is of great importance for the development of app-based trials.

    In a young population that widely uses smartphones, a digital intervention, such as the study app, provides low entry barriers. It offers easy access to evidence-based self-care information for menstrual pain and tools to improve healthy behavior. We believe that recruitment is not only influenced by the app itself but also by the way of communicating the study. We observed a substantial increase in recruitment rates following the publication of a press release on our university’s websites and corresponding media coverage. A causal relationship in the recruitment increase seems to be very probable. After 5 months without actively communicating the study with media or information material, we still could observe a basic recruitment of about 1 new study participant per day.

    Almost all research or self-care apps include BCT elements, such as prompts/cues to fill in questionnaires (self-monitoring) or to engage in app- specific intervention components. Dialog boxes are also used to give feedback on behavior or to promote self-belief [19]. However, the adequate implementation and the proper description of the applied BCTs are not easy to achieve. Therefore, it is important to involve psychologists or behavioral scientists in the design and development of an app [36,37]. The review of the use of the BCT taxonomy during the development of the trial revealed some discrepancies between the study team members and the psychologists that were involved in the ratings. For future studies, the behavior change wheel framework by Michie et al [38] will be applied before the app development to improve the design and implementation of app-based interventions [39]. Moreover, the mechanisms and efficacy of BCTs implemented need to be further explored in mHealth research settings.

    As in our previous mHealth studies, the app and trial simultaneously shaped each other during the trial design and app development process. In conventional RCTs, the trial intervention and outcomes are usually very standardized as they are described in the study protocol. However, during the development and coding process of the study app, we regularly made adaptations of the study protocol because of technical and design aspects. For example, during the development process, we realized that the digitally collected data can be used to give the users an overview of study progress and symptom improvement that subsequently became part of the intervention strategy. Branching within a question (the answer of an item impacts the next question choices) and combining different question types were not possible with the standard ResearchKit framework. Moreover, baseline questions had to be limited to reduce the time spent until finalization of onboarding, that is, the whole process from introduction, eligibility screening, and participant consent until completion of baseline survey and the random allocation to the respective intervention group. However, the final onboarding process in our research app was longer than what users of consumer apps might usually accept. This could have resulted in a loss of potential study participants. Some baseline questions typical for research studies, such as questions about partnership and income, were omitted because of privacy concerns. It was not necessary to collect body weight and height as PII data, as they were only used for BMI calculation on the user’s iPhone and not transferred to the study backend. The study design also impacted some technical decisions. For instance, to limit recall bias, questions that required daily answers before and during the period will expire after 7 days. Moreover, the way symptoms are measured or tracked in an app is limited to validated and commonly used outcomes. NRS or Likert scales are used instead of more consumer-oriented approaches, such as individualized icons or emojis to record mood or pain. This might limit the user experience.

    Limitations

    In addition to the limitation of the development process already described above, several other related limitations have to be taken into account. The decision to focus on Apple’s iOS only enabled the use of Apple ResearchKit and avoided the difficulties associated with developing for 2 operating systems simultaneously, as was done in our previous trials [12,27]. Owing to this decision, only women using an iPhone can participate in the study, which consequently might introduce selection bias and therefore limit the generalizability of the study. Moreover, the impact of technical updates (ie, ResearchKit and iOS updates) or other potential adaptations of the app during the course of the study is not clear yet, but these adaptations will be thoroughly reported in the results paper of the trial.

    Our study is also subject to some limitations from the access perspective. The numbers of App Store’s visitors and downloads are generated by Apple’s App Analytics, which we do not control. This is the only source to estimate the number of subjects interested in our study because of our anonymous study design. However, we think that it is important to also include App Analytics’ data despite its nonstudy purpose. Taking advantage of these resources from the mHealth ecosystem might help future app-based studies. To be eligible to use our study app, individuals who downloaded the app had to pass our 12-question eligibility screening that is based on our relatively strict inclusion and exclusion criteria. However, for the assessment of user evolution, we could only record the number of eligible users who gave consent because of ResearchKit’s design restrictions and our privacy rules. As a result, we lack knowledge about the reasons for ineligibility. In addition, although the participant’s eligibility and survey data underwent comparably strict plausibility checks that we have implemented in the app, fake users and fraud registration for the study cannot be completely ruled out. However, our fully remote study allows user behavior in a real-life setting [12,21,40]. Additional plausibility checks will be developed before the analysis of the results. Another way to access the app could be on recommendation of a gynecologist and/or family physician within a therapeutic setting. Further study is required to make a definite conclusion about the extent to which the app might be of use in such a setting.

    Data on user engagement in our study are limited so far. The only indicator we currently use for assessing engagement is based on the completion and response of the baseline questions. Commercial apps often use analytic tools to track user interaction with the app. These data can be used for the evaluation of engagement [41], the optimization of the app, or the addition of new features. In a study setting with strict privacy considerations, we do not use these tools. In addition, adherence would be a good measurement for user engagement. Data about adherence is not available yet but will be considered as an outcome of the study.

    Comparison With Prior Studies

    The study app and the app-based trial result from adaptation and amendments of our previous AKUD trial [12,27]. The inclusion criteria of the research population are based on the previous trial but were modified to meet the necessities of remote recruitment. In the previous AKUD trial, participants were recruited through onsite recruitment by 1 study center in Berlin, Germany facilitated by advertisements (posters, flyers, leaflets, students email lists, and subway advertisements). Baseline data were recorded with paper-and-pencil surveys. This way, it took 20 months to reach the recruitment target of 221 participants [12,27]. With the ResearchKit-based study app, we are now able to reach participants across Germany.

    For the assessment of access of app studies, Anguera et al [40] reported the recruitment number, whereas Zens et al [14] reported the consent/download rate. The percentage of consented participants (27%) in our trial is lower than in other ResearchKit studies [14,42,43]. The mPower study [42] reported 35% consent/download rate, whereas the Back on Track study [14] reported 58%. The differences might be explained by the observational character of these studies and the application of a comparably strict eligibility process in our study with 12 eligibility questions.

    User adherence and survey response rate are usually considered to be the measurements for evaluating engagement in app studies [40,44,45]. However, as adherence data of the current trial are not available in the current stage of the study, the baseline survey response is used as a proxy for engagement.

    The ResearchKit framework has been used for studies for many health conditions, such as asthma [43], acute anterior cruciate ligament ruptures [14], and Parkinson disease [42] since its launch in 2015 [46]. To our knowledge, no ResearchKit clinical trial for pain conditions has been conducted yet. Thurnheer et al [5] reported 15 studies without ResearchKit and their efficacy in a systematic review of app-based studies for pain management. App-based studies have been conducted both for acute pain such as acute needle stick pain [47] and acute pain before coronarography [48] and for chronic pain such as chronic cancer pain [49], neck pain [50], and low back pain [24,51]. No mHealth-based interventional trial has been conducted for examining the influence of an app promoting health behavior and the use of acupressure on menstrual pain [5,52] so far. Regarding the high prevalence of menstrual pain and the increasing ownership of smartphones [12,53,54], our trial might provide data that can have practical public health implications.

    Conclusions

    Conducting an evidence-based and up-to-date app study requires multidisciplinary efforts. The resulting ResearchKit-based study app for menstrual pain is accessible for the target population with positive user engagement. However, future research is necessary to investigate the determinants of user engagement, optimal BCT application, and potential clinical scenarios for app use.

    Acknowledgments

    We thank Beatrice Eden, Iris Bartsch, and Stefanie Helmer for app testing, and all participating women for their study participation. This is an investigator-initiated trial. The app development has been supported by a starting grant of the University of Zurich to Claudia Witt. Mike Armour and Caroline Smith: As a medical research institute, the National Institute of Complementary Medicine (NICM) Health Research Institute receives research grants and donations from foundations, universities, government agencies, and industry. Sponsors and donors provide untied and tied funding for work to advance the vision and mission of the Institute. This study was not specifically supported by donor or sponsor funding to NICM.

    Conflicts of Interest

    The app has been developed for research purposes and is not a commercial product. The authors do not have any financial stake in the success of the app.

    Multimedia Appendix 1

    Self care feature.

    PDF File (Adobe PDF File), 216 KB

    Multimedia Appendix 2

    Acupressure feature.

    PDF File (Adobe PDF File), 275 KB

    Multimedia Appendix 3

    Screenshots and user flow to enter the study.

    PDF File (Adobe PDF File), 1343 KB

    Multimedia Appendix 4

    CONSORT-EHEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 577 KB

    References

    1. Chan YY, Bot BM, Zweig M, Tignor N, Ma W, Suver C, et al. The asthma mobile health study, smartphone data collected using ResearchKit. Sci Data 2018 May 22;5:180096 [FREE Full text] [CrossRef] [Medline]
    2. Firth J, Torous J, Nicholas J, Carney R, Pratap A, Rosenbaum S, et al. The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry 2017 Oct;16(3):287-298 [FREE Full text] [CrossRef] [Medline]
    3. McKay FH, Cheng C, Wright A, Shill J, Stephens H, Uccellini M. Evaluating mobile phone applications for health behaviour change: a systematic review. J Telemed Telecare 2018 Jan;24(1):22-30. [CrossRef] [Medline]
    4. Lucht M, Bredenkamp R, Boeker M, Kramer U. Uniklinik Freiburg. Freiburg; 2013. Health and Supply Apps   URL: https:/​/www.​uniklinik-freiburg.de/​fileadmin/​mediapool/​09_zentren/​studienzentrum/​pdf/​Studien/​150331_TK-Gesamtbericht_Gesundheits-und_Versorgungs-Apps.​pdf [accessed 2019-12-13]
    5. Thurnheer SE, Gravestock I, Pichierri G, Steurer J, Burgstaller JM. Benefits of mobile apps in pain management: systematic review. JMIR Mhealth Uhealth 2018 Oct 22;6(10):e11231 [FREE Full text] [CrossRef] [Medline]
    6. Boyle JA, Xu R, Gilbert E, Kuczynska-Burggraf M, Tan B, Teede H, et al. Ask PCOS: identifying need to inform evidence-based app development for polycystic ovary syndrome. Semin Reprod Med 2018 Jan;36(1):59-65. [CrossRef] [Medline]
    7. Armour M, Parry K, Manohar N, Holmes K, Ferfolja T, Curry C, et al. The prevalence and academic impact of dysmenorrhea in 21,573 young women: a systematic review and meta-analysis. J Womens Health (Larchmt) 2019 Aug;28(8):1161-1171. [CrossRef] [Medline]
    8. Coco AS. Primary dysmenorrhea. Am Fam Physician 1999 Aug;60(2):489-496 [FREE Full text] [Medline]
    9. Proctor M, Farquhar C. Diagnosis and management of dysmenorrhoea. Br Med J 2006 May 13;332(7550):1134-1138 [FREE Full text] [CrossRef] [Medline]
    10. Schoep ME, Adang EM, Maas JW, de Bie B, Aarts JW, Nieboer TE. Productivity loss due to menstruation-related symptoms: a nationwide cross-sectional survey among 32 748 women. BMJ Open 2019 Jun 27;9(6):e026186 [FREE Full text] [CrossRef] [Medline]
    11. Armour M, Parry K, Al-Dabbas MA, Curry C, Holmes K, MacMillan F, et al. Self-care strategies and sources of knowledge on menstruation in 12,526 young women with dysmenorrhea: A systematic review and meta-analysis. PLoS One 2019;14(7):e0220103 [FREE Full text] [CrossRef] [Medline]
    12. Blödt S, Pach D, Eisenhart-Rothe SV, Lotz F, Roll S, Icke K, et al. Effectiveness of app-based self-acupressure for women with menstrual pain compared to usual care: a randomized pragmatic trial. Am J Obstet Gynecol 2018 Feb;218(2):227.e1-227.e9 [FREE Full text] [CrossRef] [Medline]
    13. Michie S, Abraham C, Eccles MP, Francis JJ, Hardeman W, Johnston M. Strengthening evaluation and implementation by specifying components of behaviour change interventions: a study protocol. Implement Sci 2011 Feb 07;6:10. [CrossRef] [Medline]
    14. Zens M, Woias P, Suedkamp NP, Niemeyer P. 'Back on Track': A mobile app observational study using Apple's ResearchKit framework. JMIR Mhealth Uhealth 2017 Feb 28;5(2):e23 [FREE Full text] [CrossRef] [Medline]
    15. Garnett C, Crane D, West R, Brown J, Michie S. Identification of behavior change techniques and engagement strategies to design a smartphone app to reduce alcohol consumption using a formal consensus method. JMIR Mhealth Uhealth 2015 Jun 29;3(2):e73 [FREE Full text] [CrossRef] [Medline]
    16. Kebede MM, Liedtke TP, Möllers T, Pischke CR. Characterizing active ingredients of eHealth interventions targeting persons with poorly controlled type 2 diabetes mellitus using the behavior change techniques taxonomy: scoping review. J Med Internet Res 2017 Oct 12;19(10):e348 [FREE Full text] [CrossRef] [Medline]
    17. Lane TS, Armin J, Gordon JS. Online recruitment methods for web-based and mobile health studies: a review of the literature. J Med Internet Res 2015 Jul 22;17(7):e183 [FREE Full text] [CrossRef] [Medline]
    18. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, CONSORT group, Pragmatic Trials in Healthcare (Practihc) group. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. Br Med J 2008 Nov 11;337:a2390 [FREE Full text] [CrossRef] [Medline]
    19. Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med 2013 Aug;46(1):81-95. [CrossRef] [Medline]
    20. European Parliament and Council of the European Union. EUR-Lex. 2016 Apr 27. REGULATION (EU) 2016/679 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation)   URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32016R0679 [accessed 2019-12-13] [WebCite Cache]
    21. Witt CM. Efficacy, effectiveness, pragmatic trials--guidance on terminology and the advantages of pragmatic trials. Forsch Komplementmed 2009 Oct;16(5):292-294 [FREE Full text] [CrossRef] [Medline]
    22. Heapy AA, Higgins DM, Goulet JL, LaChappelle KM, Driscoll MA, Czlapinski RA, et al. Interactive voice response-based self-management for chronic back pain: the COPES noninferiority randomized trial. JAMA Intern Med 2017 Jun 1;177(6):765-773 [FREE Full text] [CrossRef] [Medline]
    23. Choi S, O'Hare T, Gollish J, Paul JE, Kreder H, Thorpe KE, et al. Optimizing pain and rehabilitation after knee arthroplasty: a two-center, randomized trial. Anesth Analg 2016 Nov;123(5):1316-1324. [CrossRef] [Medline]
    24. Huber S, Priebe JA, Baumann K, Plidschun A, Schiessl C, Tölle TR. Treatment of low back pain with a digital multidisciplinary pain treatment app: short-term results. JMIR Rehabil Assist Technol 2017 Dec 4;4(2):e11 [FREE Full text] [CrossRef] [Medline]
    25. Witt CM, Reinhold T, Brinkhaus B, Roll S, Jena S, Willich SN. Acupuncture in patients with dysmenorrhea: a randomized study on clinical effectiveness and cost-effectiveness in usual care. Am J Obstet Gynecol 2008 Feb;198(2):166.e1-166.e8. [CrossRef] [Medline]
    26. Wade C, Wang L, Zhao WJ, Cardini F, Kronenberg F, Gui SQ, et al. Acupuncture point injection treatment of primary dysmenorrhoea: a randomised, double blind, controlled study. BMJ Open 2016 Jan 5;6(1):e008166 [FREE Full text] [CrossRef] [Medline]
    27. Blödt S, Schützler L, Huang W, Pach D, Brinkhaus B, Hummelsberger J, et al. Effectiveness of additional self-care acupressure for women with menstrual pain compared to usual care alone: using stakeholder engagement to design a pragmatic randomized trial and study protocol. Trials 2013 Apr 11;14:99 [FREE Full text] [CrossRef] [Medline]
    28. SAS Support. Cary, NC, USA SAS/STAT version 9.2   URL: http://support.sas.com/rnd/app/stat/9.2/stat92resources.html [accessed 2019-12-13]
    29. Witt CM, Aickin M, Baca T, Cherkin D, Haan MN, Hammerschlag R, Effectiveness Guidance Documents Collaborators. Effectiveness Guidance Document (EGD) for acupuncture research - a consensus document for conducting trials. BMC Complement Altern Med 2012 Sep 6;12:148 [FREE Full text] [CrossRef] [Medline]
    30. Wieland LS, Berman BM, Altman DG, Barth J, Bouter LM, D'Adamo CR, et al. Rating of included trials on the efficacy-effectiveness spectrum: development of a new tool for systematic reviews. J Clin Epidemiol 2017 Apr;84:95-104 [FREE Full text] [CrossRef] [Medline]
    31. Loudon K, Zwarenstein M, Sullivan FM, Donnan PT, Gágyor I, Hobbelen HJ, et al. The PRECIS-2 tool has good interrater reliability and modest discriminant validity. J Clin Epidemiol 2017 Aug;88:113-121. [CrossRef] [Medline]
    32. Bellmann-Strobl J, Pach D, Chang Y, Pasura L, Liu B, Jäger S, et al. The effectiveness of acupuncture and mindfulness-based stress reduction (MBSR) for patients with multiple sclerosis associated fatigue – A study protocol and its rationale for a randomized controlled trial. Eur J Integr Med 2018;20(5):6-15. [CrossRef]
    33. Apple Developer. Cupertino, CA, USA ResearchKit Human Interface Guidelines   URL: https://developer.apple.com/design/human-interface-guidelines/researchkit/overview/introduction/ [accessed 2019-05-07] [WebCite Cache]
    34. Michie S, Wood CE, Johnston M, Abraham C, Francis JJ, Hardeman W. Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data). Health Technol Assess 2015 Nov;19(99):1-188 [FREE Full text] [CrossRef] [Medline]
    35. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016 Jun;15(2):155-163 [FREE Full text] [CrossRef] [Medline]
    36. Kataria S, Ravindran V. Digital health: a new dimension in rheumatology patient care. Rheumatol Int 2018 Nov;38(11):1949-1957. [CrossRef] [Medline]
    37. Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, et al. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med 2014 Dec;29(12):1692-1701 [FREE Full text] [CrossRef] [Medline]
    38. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011 Apr 23;6:42 [FREE Full text] [CrossRef] [Medline]
    39. Fulton EA, Brown KE, Kwah KL, Wild S. StopApp: using the behaviour change wheel to develop an app to increase uptake and attendance at NHS stop smoking services. Healthcare (Basel) 2016 Jun 8;4(2):pii: E31 [FREE Full text] [CrossRef] [Medline]
    40. Anguera JA, Jordan JT, Castaneda D, Gazzaley A, Areán PA. Conducting a fully mobile and randomised clinical trial for depression: access, engagement and expense. BMJ Innov 2016 Jan;2(1):14-21 [FREE Full text] [CrossRef] [Medline]
    41. Osborn CY, van Ginkel JR, Rodbard D, Heyman M, Marrero DG, Huddleston B, et al. One Drop | Mobile: An Evaluation of Hemoglobin A1c Improvement Linked to App Engagement. JMIR Diabetes 2017 Aug 24;2(2):e21 [FREE Full text] [CrossRef] [Medline]
    42. Bot BM, Suver C, Neto EC, Kellen M, Klein A, Bare C, et al. The mPower study, Parkinson disease mobile data collected using ResearchKit. Sci Data 2016 Mar 3;3:160011 [FREE Full text] [CrossRef] [Medline]
    43. Chan YY, Wang P, Rogers L, Tignor N, Zweig M, Hershman SG, et al. The Asthma Mobile Health Study, a large-scale clinical observational study using ResearchKit. Nat Biotechnol 2017 Apr;35(4):354-362 [FREE Full text] [CrossRef] [Medline]
    44. Bakker D, Rickard N. Engagement in mobile phone app for self-monitoring of emotional wellbeing predicts changes in mental health: MoodPrism. J Affect Disord 2018 Feb;227:432-442. [CrossRef] [Medline]
    45. Torous J, Staples P, Slaters L, Adams J, Sandoval L, Onnela JP, et al. Characterizing smartphone engagement for schizophrenia: results of a naturalist mobile health study. Clin Schizophr Relat Psychoses 2017 Aug 4. [CrossRef] [Medline]
    46. Dorsey ER, Chan YF, McConnell MV, Shaw SY, Trister AD, Friend SH. The use of smartphones for health research. Acad Med 2017 Feb;92(2):157-160. [CrossRef] [Medline]
    47. Stinley NE, Norris DO, Hinds PS. Creating mandalas for the management of acute pain symptoms in pediatric patients. Art Therapy 2015;32(2):46-53. [CrossRef]
    48. Guétin S, Brun L, Deniaud M, Clerc J, Thayer JF, Koenig J. Smartphone-based music listening to reduce pain and anxiety before coronarography: a focus on sex differences. Altern Ther Health Med 2016 Jul;22(4):60-63. [Medline]
    49. Sun Y, Jiang F, Gu JJ, Wang YK, Hua H, Li J, et al. Development and testing of an intelligent pain management system (IPMS) on mobile phones through a randomized trial among Chinese cancer patients: a new approach in cancer pain management. JMIR Mhealth Uhealth 2017 Jul 25;5(7):e108 [FREE Full text] [CrossRef] [Medline]
    50. Lee M, Lee SH, Kim T, Yoo H, Kim SH, Suh D, et al. Feasibility of a smartphone-based exercise program for office workers with neck pain: an individualized approach using a self-classification algorithm. Arch Phys Med Rehabil 2017 Jan;98(1):80-87. [CrossRef] [Medline]
    51. Irvine AB, Russell H, Manocchia M, Mino DE, Glassen TC, Morgan R, et al. Mobile-Web app to self-manage low back pain: randomized controlled trial. J Med Internet Res 2015 Jan 2;17(1):e1 [FREE Full text] [CrossRef] [Medline]
    52. Portelli P, Eldred C. A quality review of smartphone applications for the management of pain. Br J Pain 2016 Aug;10(3):135-140 [FREE Full text] [CrossRef] [Medline]
    53. Götz FM, Stieger S, Reips U. Users of the main smartphone operating systems (iOS, Android) differ only little in personality. PLoS One 2017;12(5):e0176921 [FREE Full text] [CrossRef] [Medline]
    54. Mikolasek M, Witt CM, Barth J. Adherence to a mindfulness and relaxation self-care app for cancer patients: mixed-methods feasibility study. JMIR Mhealth Uhealth 2018 Dec 6;6(12):e11271 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    BCT: behavior change technique
    ICC: intraclass correlation
    mHealth: mobile health
    NICM: National Institute of Complementary Medicine
    NRS: numerical rating scale
    PII: personally identifiable information
    RCT: randomized controlled trial


    Edited by G Eysenbach; submitted 01.06.19; peer-reviewed by C Reis, T Janmohamed; comments to author 30.08.19; revised version received 21.10.19; accepted 12.11.19; published 11.02.20

    ©Jiani Wang, Alizé A Rogge, Mike Armour, Caroline A Smith, Christopher R D’Adamo, Claudia R Pischke, Hung-Rong Yen, Mei-Yao Wu, Ari Ojeda Ocampo Moré, Claudia M Witt, Daniel Pach. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 11.02.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.