This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Primary dysmenorrhea is a common condition in women of reproductive age. A previous app-based study undertaken by our group demonstrated that a smartphone app supporting self-acupressure introduced by a health care professional can reduce menstrual pain.
This study aims to evaluate whether a specific smartphone app is effective in reducing menstrual pain in 18- to 34-year-old women with primary dysmenorrhea in a self-care setting. One group of women has access to the full-featured study app and will be compared with 2 control groups who have access to fewer app features. Here, we report the trial design, app development, user access, and engagement.
On the basis of the practical implications of the previous app-based study, we revised and reengineered the study app and included the ResearchKit (Apple Inc) framework. Behavior change techniques (BCTs) were implemented in the app and validated by expert ratings. User access was estimated by assessing recruitment progress over time. User evolution and baseline survey respondent rate were assessed to evaluate user engagement.
The development of the study app for a 3-armed randomized controlled trial required a multidisciplinary team. The app is accessible for the target population free of charge via the Apple App Store. In Germany, within 9 months, the app was downloaded 1458 times and 328 study participants were recruited using it without external advertising. A total of 98.27% (5157/5248) of the app-based baseline questions were answered. The correct classification of BCTs used in the app required psychological expertise.
Conducting an innovative app study requires multidisciplinary effort. Easy access and engagement with such an app can be achieved by recruitment via the App Store. Future research is needed to investigate the determinants of user engagement, optimal BCT application, and potential clinical and self-care scenarios for app use.
ClinicalTrials.gov NCT03432611; https://clinicaltrials.gov/ct2/show/NCT03432611 (Archived by WebCite at http://www.webcitation.org/75LLAcnCQ).
In recent years, increasing smartphone access has enabled the advancement and widespread use of smartphone apps [
Menstrual disorders are highly prevalent among women of reproductive age, and especially in young women; they commonly include period pain and mood disturbances [
In a previous randomized pragmatic trial (trial registration: ClinicalTrials.gov NCT01582724) [
In 2015, Apple Inc introduced ResearchKit as an open-source framework to support clinical researchers conducting structured mobile app–based health studies [
Michie et al defined the smallest, observable, replicable intervention component with the potential to bring about change in behavior as behavior change techniques (BCTs) [
From the recruitment perspective, previous ResearchKit-based studies predominantly used Web-based recruitment. Web-based recruitment has the potential advantage of reaching a broader population quickly, whereas conventional recruitment is usually time consuming and costly. However, the broad reach can potentially bring in people who are not the target population of a particular mHealth study [
To address the questions raised above and to gain a greater understanding for conducting mHealth trials, we report the development, user access, and user engagement of our ResearchKit-based study app for an ongoing pragmatic randomized controlled trial (RCT) [
The development of the app was started with the aim to modernize the design and technology of the study app
The study app
The development of user interaction and feedback wording was based on the previous app. However, during the development of the new app, we used the BCT taxonomy (BCTTv1), according to Michie et al [
For the scientific description of an mHealth intervention, a proper description of BCTs implemented in the app is important. For this, expert validation is essential. At a later stage after the app development was completed, 2 psychologists who were not part of the development team independently rated the individual app features to validate the proper use of BCTs according to the BCTTv1 [
App features and corresponding behavior change techniques implemented.
App features | Wording and app content | BCTsa (rating) |
Introduction to baseline survey | “Hello! To get to know you better, we would like to ask you some more questions. All of your data will be kept strictly confidential and anonymous.” | No BCTs |
Baseline survey finished | “Thank you for your patience. Now we have all the necessary baseline information. You can start with the study.” | No BCTs |
Notification of doing interventions/fulfilling surveys | “Time to do some activities for your period pain and record your progress.” | Prompts/cues (7.1) |
When a survey has been finished | “Well Done!” | Social reward (10.4) |
In-app reminder of finishing survey during task days | “Missing Answers. Keep going with the questions, this can help you see your progress.” | Prompts/cues (7.1) |
In-app reminder for acupressure | “Apply acupressure. On days where you have pain, we recommend at least twice a day.” | Prompts/cues (7.1) |
When the timer for acupressure finished (for all 6 points) | “Well Done! Keep on taking care of yourself.” | Social reward (10.4) |
Guide for nontask days | “New questions will appear five days before your next period.” | Prompts/cues (7.1) |
Instructions of when to apply acupressure | When to Apply Acupressure. Instructions of when to apply acupressure (time, frequency). | Goal setting (behavior) (1.1); action planning (1.4) |
Instructions of how to apply acupressure | How to Apply Acupressure. Instructions of how to apply acupressure (position, strength, and feeling). | Instructions on how to perform a behavior (4.1) |
An image and location for each acupressure point | Image and description of locations of acupressure 3 points: spleen 6, liver 3, large intestine 4. | Instructions on how to perform a behavior (4.1); demonstration of behavior (6.1) |
Instruction video for self-acupressure | An instruction animation for self-acupressure on 3 points: spleen 6, liver 3, large intestine 4. | Instructions on how to perform a behavior (4.1); demonstration of behavior (6.1) |
Self-care recommendation | “Evidence-based information with references of 5 self-care recommendations: exercises; dietary supplementations; heating pad/hot water bottle; yoga; medication.” | Information about health consequences (5.1); credible source (9.1) |
Timer for self-acupressure: 1 minute for each point | A counting down timer with a picture of the corresponding acupressure point. | Goal setting (behavior) (1.1); instructions on how to perform a behavior (4.1); demonstration of behavior (6.1) |
Dashboard screen | Dashboard screen, including period calendar, diagrams, and charts reviewing pain and survey questions, and a function button for period start/end. | Feedback on behavior (2.2); self-monitoring of behavior (2.3); self-monitoring of outcome(s) of behavior (2.4); feedback on outcome(s) of behavior (2.7) |
Journal screen: calendar | Journal screen in calendar view, including period calendar that also displays the completion of survey questions. | Prompts/cues (7.1) |
Journal screen: questions | Journal screen in questions view, including a list of survey questions with the date. | No BCTs |
Self-care screen | Self-care screen, including a list and icon images for 5 self-care recommendations. | No BCTs |
aBCT: behavior change technique.
Privacy and data security were considered high priorities during app development. User data collected by the app are encrypted and transferred anonymously. We adhere to the principle of data minimization [
We will conduct a 3-armed, randomized pragmatic trial [
The decisions on study design of this trial are based, in part, on decisions of the stakeholder advisory group from the corresponding previous trial and its results [
Furthermore, 5 days before the anticipated start of the menstruation until the end of bleeding, notifications from the app will remind all the groups of participating women to complete questions and perform self-care activities, such as self-acupressure or yoga, depending on the group allocation.
The self-care feature will offer information on self-care for menstrual pain, including evidence-based information about exercise, nutrition and dietary supplementation, heating pad/hot water bottle, yoga, and when to consult a doctor and regarding how primary dysmenorrhea is treated in most cases (see
The acupressure feature will offer detailed written and multimedia descriptions of the acupressure to be used for menstrual pain (see
The acupressure intervention resulted from a written consensus process with international acupuncture experts from China, Germany, and the United States of America [
Participants are allowed to continue with their own usual care (medical and nonmedical) during the study.
We aim to recruit 594 young women with primary dysmenorrhea. The sample size estimation is based on the comparison of the group receiving the full-featured app (self-care information + self-acupressure) with the group receiving the app version without the self-care information (control intervention II) regarding the primary outcome (NRS after 6 menstrual cycles) that will be treated as a continuous variable. Our previous study showed a mean group difference of 1.4 on the NRS and a standard deviation of 2.15 at the sixth menstrual cycle after the onset of the trial.
Assuming that self-care information has a smaller impact on pain than acupressure, we hypothesize a difference of 0.8 on the NRS between groups. To detect a mean difference of 0.8 point on the NRS after 6 menstrual cycles between the group receiving the full-featured app (with a common standard deviation of 2.15 observed in our previous study) and control intervention II, applying a 2-sided
The eligibility criteria resemble the criteria of our previous study. Women owning an iPhone will be included if they have primary dysmenorrhea, are between the ages of 18 and 34 years, report moderate or severe menstrual pain ≥6 on the NRS; 0=
Eligibility questions.
Eligibility questions | Question type | Criteria |
Are you a woman over 18 and below 35 years old? | Yes/no | If no, exclude |
Do you suffer from period pain or menstrual cramps during every menstrual cycle? | Yes/no | If no, exclude |
Do you suffer from your period pain on more than 5 days outside the period? | Yes/no | If yes, exclude |
Do you think your pain started during your teenage years? | Yes/no | If no, exclude |
Do you have any prior history of a gynecological disease that is known to be a reason for your period pain? | Yes/no | If yes, exclude |
Did you have a period within the last 6 weeks? | Yes/no | If no, exclude |
Is your cycle length between 3 and 6 weeks? | Yes/no | If no, exclude |
How strong was the most severe pain without medication during your last period? | Numerical on a pain scale from 0 to 10 | If <6, exclude |
Are you willing to see a doctor when (1) your pain is getting worse than usual, (2) pain medication is not helping, and (3) when you have pain well before or well after the period? | Yes/no | If no, exclude |
Are you pregnant? | Yes/no | If yes, exclude |
Do you plan to be pregnant within the next 12 months? | Yes/no | If yes, exclude |
Is this your iPhone? | Yes/no | If no, exclude and message the user because of data protection, the app should be used only on your own iPhone |
From a methodological point of view, a clinical trial provides more evidence on the effectiveness of an intervention using a pragmatic trial design or on the efficacy side using an explanatory trial design [
During the design phase of the trial, PRECIS served as a tool to make better informed design decisions [
The primary recruitment strategy focuses on self-referral through the Apple App Store. On the basis of our experience from the previous trial and the associated stakeholder engagement [
The app use will be free of charge; no financial compensation will be provided for participating in the study.
Potential future recruitment strategies will include traditional and Web-based recruitment methods that are also adapted to the respective study sites. These will include information about the ongoing study on printed posters or information leaflets or in social media. In addition, if accepted by the Apple App Store editorial team, we will inform potential users about the study app with the
When users install and open the study app for the first time, they will be briefly introduced to the study and encouraged to participate. For potential participants who wish to continue, an app-based anonymous eligibility screening and more detailed information about the study will be provided. After the consent process, participants will finish the app-based baseline survey to unlock the intervention interface. This process is based on the onboarding process of Apple’s ResearchKit framework [
In the baseline survey, general information relevant for menstrual pain will be assessed, such as age, education, individual exercise behavior, length of period and level of pain experienced during the period, and use of hormonal contraceptives and pain medications (
Baseline questions.
Baseline questions and answer field | Skip button | ||||
|
|
||||
|
_____ years |
|
|||
|
|
||||
|
Your height: ____ cm |
|
|||
|
Your body weight: ____ kg |
|
|||
|
|
||||
|
High school or above |
|
|||
|
Other |
|
|||
|
|
||||
|
_____ days |
|
|||
|
|
||||
|
_____ days |
|
|||
|
|
||||
|
Stomach cramps |
|
|||
|
General pain in lower belly |
|
|||
|
Lower back pain |
|
|||
|
Headache |
|
|||
|
Nausea/Vomiting |
|
|||
|
Other symptoms, namely _____ |
|
|||
|
|
||||
|
No |
|
|||
|
|
|
|||
|
|
|
|
||
|
|
|
I use hormonal contraceptives because of my period pain. |
|
|
|
|
|
I use hormonal contraceptives for contraception. |
|
|
|
|
|
I use hormonal contraceptives because of other reasons (for example, acne). |
|
|
|
If yes, which hormonal contraceptives are you using? ______ |
|
|||
|
If yes, how long have you been using hormonal contraceptives? for ____ months and ____ years |
|
|||
|
|
||||
|
No |
|
|||
|
Yes |
|
|||
|
If yes, number of pregnancies:____ |
|
|||
|
If yes, number of births:____ |
|
|||
|
|
||||
|
0 1 2 3 4 5 6 7 8 9 10 (0=no pain at all, 10=most intense pain imaginable) |
|
|||
|
|
||||
|
0 1 2 3 4 5 6 7 8 9 10 (0=no pain at all, 10=most intense pain imaginable) |
|
|||
|
|
||||
|
_____ days |
|
|||
|
|
||||
|
_____ days |
|
|||
|
|
||||
|
No | X | |||
Yes ->if yes, which one: ______ |
|
||||
|
|
||||
|
No actions |
|
|||
|
Fitness/Gymnastics |
|
|||
|
Jogging/Running |
|
|||
|
Acupressure |
|
|||
|
Yoga |
|
|||
|
Autogenic training |
|
|||
|
Herbal medicine |
|
|||
|
Meditation/Relaxation |
|
|||
|
Homeopathy |
|
|||
|
Local supply of heat |
|
|||
|
Food supplements |
|
|||
|
Tea |
|
|||
|
Others: ______ |
|
|||
|
|
||||
|
No actions |
|
|||
|
Fitness/gymnastics |
|
|||
|
Jogging/running |
|
|||
|
Acupressure |
|
|||
|
Yoga |
|
|||
|
Autogenic training |
|
|||
|
Herbal medicine |
|
|||
|
Meditation/relaxation |
|
|||
|
Homeopathy |
|
|||
|
Local supply of heat |
|
|||
|
Food supplements |
|
|||
|
Tea |
|
|||
|
Others: ______ |
|
|||
|
|
||||
|
__.__.____ |
|
aX: skip button enabled.
b—: skip button disabled.
cIUD: intrauterine device.
The PRECIS-2 score was calculated by summing up the means of each dimension based on the rating results of 11 raters; meanwhile, standard deviations were calculated to show the variability.
For the BCT ratings, the interrater reliability among BCT raters was assessed by intraclass correlations (ICCs) [
For the assessment of user access, we used the data generated by App Analytics [
To assess user engagement, the user conversion rate and the baseline survey response data were calculated using descriptive statistics (frequencies, percentages, means, and standard deviations). The baseline survey variables were extracted from the back-end database and only the missing values of the baseline survey (
All collected data were analyzed with SPSS version 22.0 (SPSS Inc).
The app is prepared for international use and can be currently (October 2019) downloaded in the German App Store and will later be made available in the App Stores of the other participating centers. The study database, the app server, and the primary study center are based in Berlin, Germany. The study was approved by the university’s ethics committee (Charité—Universitätsmedizin Berlin approval Number EA1/364/16). The trial was registered at ClinicalTrials.gov (NCT03432611).
The participation of the study sites in Taichung, Taiwan (approval letter Number CMUH107-REC1-120 by Ethics Committee of China Medical University and Hospital); Sydney, Australia (approval number H13175 by Western Sydney University Human Research Ethics Committee); Florianopolis, Brazil (approval number 3.583.066 by Ethics Committee of Federal University of Santa Catarina), and Baltimore, United States is currently being processed.
The study app is a result of multidisciplinary efforts. The launch of the study app in the App Store will mark the beginning of the fully app-based study: users will be recruited via the Apple App Store, eligibility and consent will be processed by the study app, different self-care interventions will be guided by corresponding app features, and the follow-up will be recorded by app-based survey questions (
The study app will display the intervention components (self-acupressure and self-care information) selectively according to the group allocation. The core features
Study design. ITT: intention to treat; PP: per protocol.
Screenshots of the study app.
To validate whether the BCTs implemented in the app were properly applied, a developer rating (JW) was compared with ratings of 2 psychologists with BCT expertise (CRP and AR) who had experienced the finalized full-featured app but who had not been part of the app development process. The interrater agreement between the 2 psychologists showed an excellent ICC (ICC=0.954; 95% CI 0.87-0.98). However, the overall interrater agreement including all raters was poor (ICC=0.442; 95% CI 0.07-0.78), that is, the ratings of the BCTs used during the development by the study team, did not correspond well with the ratings of the 2 psychologists. There was no significant difference between ICCs at the item level and the cluster level based on the BCTs taxonomy (v1) [
On the basis of the rating results of all authors, all 9 dimensions of the PRECIS-2 tool are defined more on the pragmatic side (
PRECIS-2 rating results of the study design.
Trial recruitment started in February 2018 with the launch of the ResearchKit-based study app in the German App Store. The Web-based press release was well received by the public and the media. By observation of media coverage via Google search during the following 10 weeks, 65 articles or blog entries of pharmacy or health-related websites citing the press releases in English and German could be detected. Overall, 2 printed newspapers reported about this app-based study in German. An increase of media coverage could be observed from March to May 2018. In the weeks following the press release, the app showed continuous increase in both downloads and the number of users (
After 38 weeks in the app store (from February 19, 2018, to November 13, 2018), there were 1458 downloads and 328 users were included into the study (22.5%). On average, we recruited around 8 study participants per week with a peak between May and June after the press release (22 new users per week). Approximately 60% (195/328) of the participants were recruited within these 2 months.
App downloads and new users per day.
During the first 38 weeks of recruitment, the App Store’s preview of the app was viewed 1885 times. Although 75% of the app’s product page viewers found the app by searching the App Store, 25% found the app by App Store browsing, app referral, or Web referral. The app was downloaded 1458 times. A total of 388 (27%) users passed the 12-question eligibility screening and agreed to consent; 328 of the 388 users (85%) completed the 16-question baseline survey and were recruited to the study.
For 11 of 16 baseline questions, the
Almost all questions of the baseline survey were answered (data completeness of 98.27%; 5157/5248). A total of 276 users (276/328, 84.1%) answered all 16 baseline questions and never used the skip button. Only 3% of the data based on the skippable questions were missing. The question asking for discomfort/symptoms during the period was answered by all users (response rate 100%). For free-text fields, 105 (105/328, 32.0%) of the users provided details about their discomfort/symptoms during their period; 269 (269/328, 82.0%) users provided details about their medication for the question asking about the period pain-related medical history.
User evolution.
By using the ResearchKit framework, we successfully developed a study app for a fully app-based pragmatic RCT for young women with primary dysmenorrhea. The app is easily accessible via self-referral and can be used as a self-care and study tool for a highly relevant condition. The available data already indicate a high level of user engagement with the study app. We also realized that the early involvement of behavioral science experts is of great importance for the development of app-based trials.
In a young population that widely uses smartphones, a digital intervention, such as the study app, provides low entry barriers. It offers easy access to evidence-based self-care information for menstrual pain and tools to improve healthy behavior. We believe that recruitment is not only influenced by the app itself but also by the way of communicating the study. We observed a substantial increase in recruitment rates following the publication of a press release on our university’s websites and corresponding media coverage. A causal relationship in the recruitment increase seems to be very probable. After 5 months without actively communicating the study with media or information material, we still could observe a basic recruitment of about 1 new study participant per day.
Almost all research or self-care apps include BCT elements, such as prompts/cues to fill in questionnaires (self-monitoring) or to engage in app- specific intervention components. Dialog boxes are also used to give feedback on behavior or to promote self-belief [
As in our previous mHealth studies, the app and trial simultaneously shaped each other during the trial design and app development process. In conventional RCTs, the trial intervention and outcomes are usually very standardized as they are described in the study protocol. However, during the development and coding process of the study app, we regularly made adaptations of the study protocol because of technical and design aspects. For example, during the development process, we realized that the digitally collected data can be used to give the users an overview of study progress and symptom improvement that subsequently became part of the intervention strategy. Branching within a question (the answer of an item impacts the next question choices) and combining different question types were not possible with the standard ResearchKit framework. Moreover, baseline questions had to be limited to reduce the time spent until finalization of onboarding, that is, the whole process from introduction, eligibility screening, and participant consent until completion of baseline survey and the random allocation to the respective intervention group. However, the final onboarding process in our research app was longer than what users of consumer apps might usually accept. This could have resulted in a loss of potential study participants. Some baseline questions typical for research studies, such as questions about partnership and income, were omitted because of privacy concerns. It was not necessary to collect body weight and height as PII data, as they were only used for BMI calculation on the user’s iPhone and not transferred to the study backend. The study design also impacted some technical decisions. For instance, to limit recall bias, questions that required daily answers before and during the period will expire after 7 days. Moreover, the way symptoms are measured or tracked in an app is limited to validated and commonly used outcomes. NRS or Likert scales are used instead of more consumer-oriented approaches, such as individualized icons or emojis to record mood or pain. This might limit the user experience.
In addition to the limitation of the development process already described above, several other related limitations have to be taken into account. The decision to focus on Apple’s iOS only enabled the use of Apple ResearchKit and avoided the difficulties associated with developing for 2 operating systems simultaneously, as was done in our previous trials [
Our study is also subject to some limitations from the access perspective. The numbers of App Store’s visitors and downloads are generated by Apple’s App Analytics, which we do not control. This is the only source to estimate the number of subjects interested in our study because of our anonymous study design. However, we think that it is important to also include App Analytics’ data despite its nonstudy purpose. Taking advantage of these resources from the mHealth ecosystem might help future app-based studies. To be eligible to use our study app, individuals who downloaded the app had to pass our 12-question eligibility screening that is based on our relatively strict inclusion and exclusion criteria. However, for the assessment of user evolution, we could only record the number of eligible users who gave consent because of ResearchKit’s design restrictions and our privacy rules. As a result, we lack knowledge about the reasons for ineligibility. In addition, although the participant’s eligibility and survey data underwent comparably strict plausibility checks that we have implemented in the app, fake users and fraud registration for the study cannot be completely ruled out. However, our fully remote study allows user behavior in a real-life setting [
Data on user engagement in our study are limited so far. The only indicator we currently use for assessing engagement is based on the completion and response of the baseline questions. Commercial apps often use analytic tools to track user interaction with the app. These data can be used for the evaluation of engagement [
The study app and the app-based trial result from adaptation and amendments of our previous
For the assessment of access of app studies, Anguera et al [
User adherence and survey response rate are usually considered to be the measurements for evaluating engagement in app studies [
The ResearchKit framework has been used for studies for many health conditions, such as asthma [
Conducting an evidence-based and up-to-date app study requires multidisciplinary efforts. The resulting ResearchKit-based study app for menstrual pain is accessible for the target population with positive user engagement. However, future research is necessary to investigate the determinants of user engagement, optimal BCT application, and potential clinical scenarios for app use.
Self care feature.
Acupressure feature.
Screenshots and user flow to enter the study.
CONSORT-EHEALTH checklist (V 1.6.1).
behavior change technique
intraclass correlation
mobile health
National Institute of Complementary Medicine
numerical rating scale
personally identifiable information
randomized controlled trial
We thank Beatrice Eden, Iris Bartsch, and Stefanie Helmer for app testing, and all participating women for their study participation. This is an investigator-initiated trial. The app development has been supported by a starting grant of the University of Zurich to Claudia Witt. Mike Armour and Caroline Smith: As a medical research institute, the National Institute of Complementary Medicine (NICM) Health Research Institute receives research grants and donations from foundations, universities, government agencies, and industry. Sponsors and donors provide untied and tied funding for work to advance the vision and mission of the Institute. This study was not specifically supported by donor or sponsor funding to NICM.
The app has been developed for research purposes and is not a commercial product. The authors do not have any financial stake in the success of the app.