JMIR Publications

JMIR mHealth and uHealth


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 08.05.15 in Vol 3, No 2 (2015): Apr-Jun

This paper is in the following e-collection/theme issue:

    Original Paper

    Testing the Feasibility and Psychometric Properties of a Mobile Diary (myWHI) in Adolescents and Young Adults With Headaches

    1IWK Health Centre, Centre for Research in Family Health, Halifax, NS, Canada

    2Dalhousie University, Department of Community Health and Epidemiology, Halifax, NS, Canada

    3Dalhousie University, Departments of Community Health and Epidemiology, Pediatrics and Psychiatry, Halifax, NS, Canada

    4Nova Scotia Health Authority, Halifax, NS, Canada

    5Dalhousie University, Department of Psychology and Neuroscience, Halifax, NS, Canada

    6The Hospital for Sick Children, Child Health Evaluative Sciences, Toronto, ON, Canada

    7University of Toronto, Lawrence S Bloomberg Faculty of Nursing, Toronto, ON, Canada

    8Memorial University of Newfoundland, Faculty of Medicine, Health Sciences Centre, St John's, NL, Canada

    *all authors contributed equally

    Corresponding Author:

    Anna Huguet, PhD

    IWK Health Centre

    Centre for Research in Family Health

    5850/5980 University Avenue, 8th Floor Children’s Building, Room K8527, PO Box 9

    Halifax, NS, B3K 6R8


    Phone: 1 902 470 7521

    Fax:1 902 470 6534



    Background: Headaches are prevalent among teens and young adults. Self-monitoring is essential for managing headaches and can be accomplished with the help of electronic headache diaries. An increasing number of electronic headache diaries exist, yet the absence of quality standards compromises their use for research and clinical purposes.

    Objective: Our goal was to develop and test the usability, feasibility, and psychometric properties of an electronic diary iPhone application for self-monitoring by adolescents and young adults with headaches.

    Methods: We used an iterative participatory design to develop and test our electronic headache diary. Participants aged 14-28 years old with recurrent headaches were recruited internationally. Screening and consent were conducted online. Following completion of an online pre-questionnaire, participants downloaded the diary to use in their natural environment for 14 days. An online post-questionnaire was completed following testing. The diary’s usability and feasibility were tested first and determined to be complete when improvements to the diary did not result in a statistically significant impact on indicators of feasibility and adherence. Interviews were conducted with participants of usability and feasibility testing. The psychometric properties of the diary were then tested, and a case study analysis of one participant was completed.

    Results: Three cycles to test the usability and feasibility were conducted. Each cycle included 11-19 unique participants ranging in age from 16 to 28 years. Following the testing period for each cycle, 15% to 25% of participants took part in the post-cycle interview. Participants perceived the final version of the diary as useful, easy to learn, and efficient to use. Psychometric properties were then tested with a sample of 65 participants (6 aged 14-17 years old; 59 aged 18-28 years old). All items in the diary had substantial between- and within-subjects variability (percent of variance for the two participant groups ranged from 20.64 to 75.60 and 23.74 to 79.21, respectively). Moreover, the Migraine Disability Assessment (MIDAS) included in the diary had adequate between-subjects reliability (R1F=0.66, RKF=0.98), but low within-subjects reliability (RC=0.51). Critical elements of the diary demonstrated adequate convergent and concurrent validity, particularly in the older age group (18-28 years). The validity of some critical elements of the diary could not be explored in the younger age group due to the small subgroup size. The case study provides an example of the potential utility of the diary.

    Conclusions: Our electronic headache diary was shown to be a usable and feasible self-monitoring tool when used by adolescents and young adults with headaches for 14 days. This study provides preliminary support of its psychometric properties. Our diary has the potential for helping users to better understand their headaches and, consequently, to change behaviors to improve self-management of their headaches. Its effectiveness as a component of an intervention will be the focus of future research.

    JMIR mHealth uHealth 2015;3(2):e39




    Tension-type headache (HA) and migraine are ranked as the second and third most common diseases in the world, affecting 20.8% and 14.7% of the world population, respectively [1]. Migraine is ranked as the eighth leading cause for disability [1]. Due to the burden of these conditions, effort toward improving care is warranted.

    Consistent with the International Headache Society guidelines [2], health care professionals often advise diary use to self-monitor headaches [3,4]. Self-monitoring enables recognition of temporal behavior patterns, allows individuals to become informed and actively self-manage headaches, facilitates treatment decision-making and treatment tailoring, and offers a measure of treatment efficacy [5]. Self-monitoring is particularly useful for people with recurrent headaches, whose episodes usually occur in response to unrecognized triggers [6]. Diaries can help individuals understand headache patterns and identify triggers [7], which is a basic treatment strategy for headaches [8,9]. Behavioral management of these triggers can result in fewer headaches [10]. Findings from meta-analyses indicate that behavioral interventions, usually including self-monitoring, are also effective at reducing headaches [11].

    In addition to the clinical advantages of self-monitoring with headache management, diaries also have research benefits. Diaries allow researchers to test hypotheses of within-subject relations over time as an extension of prior cross-sectional research (eg, finding whether headache episodes more likely to occur in the context of a putative trigger) [12,13].

    While paper diaries have long been used for self-monitoring headaches [14], advances in technology have afforded widespread use of electronic diaries (e-diaries) [10,15-18]. E-diaries offer several advantages, including increased adherence, accuracy, acceptability, and efficiency [14,18-21]. Two recent systematic reviews identified 5 e-diaries used in research [22] and 38 in Canadian mobile app stores for iOS and Android platforms [23]. The quality of these self-monitoring tools is questionable in the absence of any existing standards. Current headache e-diaries have several limitations, including not using a participatory design [24], recording insufficient data to provide understanding of headache patterns, lacking evidence that demonstrates feasibility and psychometric properties, and lacking research on the impact of these diaries on health outcomes [11,22].

    Our overall goal was to create a usable, feasible, and psychometrically sound electronic headache diary for people aged between 14 and 28 years old who have recurrent headaches that also overcame some of the identified weaknesses of existing diaries. The specific objectives of this study were to: (1) test and improve the usability and feasibility of a new iPhone-based diary in terms of adherence, learnability, acceptability, efficiency, and accuracy through the use of iterative cycles; (2) test the psychometric properties (eg, reliability/convergent and concurrent validity) of this diary when used for assessment purposes; and (3) illustrate its potential utility with a case study analysis.



    Participants were eligible if they self-identified as: (1) being aged 14-28; (2) having proficiency with English speaking, reading, and writing; (3) having a headache frequency of 2 or more episodes each month for the last 3 months; and (4) owning an iPhone with a data plan or wireless Internet access. Participants were excluded if they self-identified as having: (1) cognitive and/or developmental delays; (2) not visited a physician to exclude an organic disorder or traumatic injury as the cause for their headaches; and/or (3) significant visual impairment or blindness. Participants who consented but did not complete the pre-assessment questionnaire were also excluded.

    Concept and Development Process of the Electronic Headache Diary

    Several steps were taken to create our electronic headache diary application (see Figure 1). Once diary items and features were well-defined, an iterative process that involved 3 cycles of designing, testing, reviewing, and refining the e-diary app was followed. Initially, low-fidelity paper diary prototypes, followed by high-fidelity software-based diary prototypes, were tested with volunteers in the lab. Potential end users were then involved in 3 iterative cycles of testing high-fidelity, software-based diary prototypes in participants’ natural environments. The final high-fidelity prototype was used to test the psychometric properties of the diary. The evaluation of the diary with potential end users during the iterative cycle phase is the focus of this manuscript.

    Figure 1. Defining the concept of myWHI diary.
    View this figure

    Evaluation of the Usability, Feasibility, and Psychometric Properties of the Diary

    Participants involved in usability/feasibility testing and psychometric properties testing followed the same procedure unless indicated. Participants were recruited internationally through online advertisements on social networks, classified ads, and mailing lists. We used an automated Web-based system for efficient online screening and consent that closely mimicked the electronic signup process that will be followed when the application is released to the public. Potential participants were asked screening questions by the automated system. Those identified as eligible were automatically asked to provide online consent (assent and parent authorization if between 14-16 years old). If consent was provided, an online pre-questionnaire was completed. The researcher then emailed each new participant with instructions for downloading the iPhone application along with the request to use the diary for 14 days. After this testing period, an online post-questionnaire was completed and 15% to 25% of participants involved in the usability/feasibility testing were randomly selected for a 45-minute, end-of-study interview via phone or Skype. Participants involved in psychometric testing were not interviewed. Participants were reimbursed for their time. This study was approved by the IWK Health Centre’s Research Ethics Board.


    Electronic Headache Diary

    Our electronic headache diary was called the myWireless Headache Intervention diary (myWHI diary). It was designed to help users become more aware of headache symptoms and patterns. It tracks temporal, sensory, and affective aspects of headaches, and headaches’ impact on daily life, potential triggers, and coping behaviors. The diary incorporates ad-hoc and validated paper measures (ie, MIDAS/PedMIDAS and NRS-11). It includes outcome measures recommended by IMMPACT, PedIMMPACT [25-27], and guidelines for both pharmacological and behavioral clinical trials of headache [28-30] to facilitate use of the diary in a scientific trial context. The diary encourages users to complete a headache entry for every headache. First, users report occurrence by specifying start time, initial intensity, starting location, potential trigger(s), and prior symptoms (Figure 2.1). At the end of the headache, users are encouraged to report ending time, headache quality, highest pain intensity, level of unpleasantness, associated symptoms, change in headache location, and medication taken or strategies utilized to cope with the headache. At the end of each day, regardless of whether they had a headache, users are asked to enter additional information into the daily diary (eg, overall mood, hours and quality of sleep; see Figure 2.2). In addition, participants record headache impact on daily activities if a headache occurred that day. As all items are optional, users can keep track of information most relevant to them. The diary provides visual graphs of headaches over time in terms of occurrence, intensity, duration, and level of headache-related interference in daily functioning. To highlight potential causal relationships, users can also see how these parameters are related with the tracked potential triggers (Figure 2.3).

    Figure 2. Screenshots of the myWHI diary interface. 2.1 Headache entry 2.2 Daily diary entry 2.3 Reports.
    View this figure


    A closed-ended questionnaire gathered demographics, iPhone usage, and headache characteristics. Based on headache characteristics and considering the International Headache Classification (IHC) criteria [31], participants were classified as having migraine-like headaches, tension-like headaches, mixed headaches (meeting criteria for both migraine-like and tension-like headaches), or unclassified headaches.


    A closed-ended questionnaire was administered to participants of usability/feasibility testing and psychometric testing. The questions posed to the 2 testing groups were different, as explained below.

    Post-Questionnaire for Usability/feasibility Testing

    This questionnaire evaluated the diary in terms of helpfulness, usefulness, efficiency, visual appeal, and ability to be understood through ad-hoc questions.

    Post-Questionnaire for Psychometric Testing

    This questionnaire included the following standardized tools:

    1. The Numerical Rating Scale (NRS-11): An 11-point self-report pain intensity scale commonly used. It has been psychometrically tested in children aged 8 years and above, adolescents [32], and adults [33,34].
    2. The Mental Health Inventory-5 (MHI-5): A 5-item self-report measure to assess psychological distress during the past month [35]. A higher score indicates better mental health. MHI-5 has repeatedly been shown to be valid and reliable in adolescents and adults [36,37].
    3. The Patient Reported Outcomes Information Measurement System (PROMIS) Short Form—Sleep Disturbance (adult version): A measure developed to assess the qualitative aspects of sleep. Its psychometric properties have been shown to be valid for adults [38]. This scale was administered to participants aged 18 years and over.
    4. The PROMIS Short Form—Pain Interference (adult version): An 8-item self-report measure to assess pain-related interference with: (1) physical functioning, (2) psychological functioning, and (3) social functioning. It has been tested in adults [39]. This scale was administered to participants aged 18 years and over.
    5. The Sleep/Wake Behavior Problems Scale of the Sleep Habits Survey, a scale that evaluates erratic sleep/wake behaviors [40], and the PROMIS Pediatric Short Form—Pain Interference (child version), an 8-item self-report measure to assess pain-related interference with functioning [41], were administered to participants between age 14 and 17 years old. However, this information was not used in our analyses due to the reduced number of participants between the ages of 14 and 17 years old (see results).

    End-of-study interview

    A semi-structured interview administered to 15% to 25% of participants involved in each cycle of usability/feasibility testing was facilitated by one researcher guided by a script. The questions focused on the participants’ experiences using the diary, including barriers, usefulness, burdens, goals, and suggestions for improvement.

    Data Analytic Strategy

    Usability and Feasibility Testing of the Diary

    Analysis was exploratory in nature. Using SPSS 20 predictive analytics software, descriptive statistics (median and range) were calculated for pre- and post-assessment data. Usage of the diary was automatically collected from the system. Mann-Whitney U and Chi-square tests were used to evaluate differences between consecutive testing cycles for continuous and categorical variables used as indicators of feasibility and adherence. We used non-parametric methods and report the median (mdn) instead of the mean because of our small sample size. Qualitative data collected through end-of-study interviews were transcribed, coded, and analyzed using an inductive thematic analysis [42].

    Psychometric Properties Testing of the Diary

    We hypothesized that the diary would be reliable and the most essential components of the diary would be valid. We hypothesized that convergent construct validity would be supported by high correlations between the data derived from the items of the diary that assess headache occurrence, intensity, unpleasantness, mood, and sleep and headache impact, with data obtained through questions asking for the same information retrospectively mostly using well-established single-point measures. We also hypothesized that if the most essential parts of diary had concurrent validity then a good number of variables collected through the diary would be correlated with other single-point measures that assess related constructs. Specifically, we hypothesized that the total number of headache episodes, total headache time, highest headache intensity, and unpleasantness of headaches would be moderately to strongly related to levels of pain-related interference, sleep-related impairment, and emotional functioning as assessed retrospectively through well-established single-point measures.

    Reliability of most variables measured by the diary was assessed using generalizability theory analysis. Following Cranford et al [43] we used generalizability theory to decompose the variance of daily variables and to calculate reliability estimates using VARCOMP procedures in SPSS 20. For single-item variables, variance was decomposed into person, day, and person-by-day components. The person component represents the between-subjects portion of the variance that remains stable across all 14 days, and the person-by-day component represents the within-subjects component that varies by day. Thus, a variable with a high proportion of person variance is highly stable over time, which is analogous to having high test-retest reliability. Because MIDAS/PedMIDAS, which measures the impact of headache on the subject’s daily functioning, is a multi-item measure, the variance was decomposed into person, day, item, person-by-day, person-by-item, day-by-item, and error variability. This information is used to calculate 3 estimates of between- and within-subjects reliability [43]. R1F is equivalent to calculating Cronbach’s alpha for each of the 14 days separately, and taking the average of all 14 alphas. RKF values are equivalent to averaging all 14 days of data into a single composite index, then calculating Cronbach’s alpha once. RC values represent within-subjects reliability, which represents the precision of the measure for measuring systematic variance from day-to-day.

    The convergent construct validity and criterion validity of the diary were assessed by correlating several items of the diary with well-established 1-point measures. Because diagnostics (ie, kurtosis, skewness, histograms, P-P plots) suggested many variables were non-normally distributed, we used Spearman’s rank-order correlation coefficient (rs). Cohen’s criterion (small = 0.10; medium = 0.30; large = 0.50; Cohen et al [44]) was used to interpret effect sizes. Variables from the diary were combined across all 14 days into a single value per participant using various functions (eg, sums, averages, maximum value) to facilitate comparison with variables measured only once. A full list of all convergent and criterion validity tests can be found in Table 7. The minimum required sample size to achieve 80% power assuming to find at least medium-to-large effect size (rs=0.40) and an alpha of 0.05 was 46 people.

    Case Study Analysis

    A representative example of a participant of psychometric properties testing was selected.


    Table 1 provides the descriptive statistics of the participants involved in evaluation of the diary. In the usability/feasibility testing cycles, the majority of participants were young adult females with median ages of 24 to 25 years old (range, 16-28 years). The most frequent type of headache was migraine-like headache. In psychometric testing, the median age was similar, 22 years old (range, 14-28); only 6 of the 65 participants were 14 to 17 years old, and 59 were 18 to 28 years old. As in usability/feasibility testing, the majority of participants were female and the most common type of headache was migraine-like headache.

    Table 1. Descriptive statistics of participants involved in the evaluation of the myWHI diary.
    View this table

    Figure 3 shows the flowchart of participation in the evaluation of the diary. Results of usability/feasibility testing are based on participants who used the diary application at least once during Cycle 1 (n=11), Cycle 2 (n=19), and Cycle 3 (n=13). Results of psychometric testing are based on participants who completed the post-questionnaire (n=65).

    Figure 3. Flowchart of participation in the evaluation of the myWHI diary.
    View this figure

    Usability and Feasibility Testing of the Diary


    Table 2 shows indicators of adherence. In Cycle 1, headache entries tended to be entered a long time after reported start time (mdn=13.59 h; range, 1.73-101.28 hours). Headaches were often not recorded on the actual day that the episode was reported to have occurred. However, most important headache items were answered with each entry. A minority of participants completed all 14 daily diary entries (18%, n=2) or 75% of the 14 daily diary entries (18%, n=2). The items encouraged to be answered every day were usually entered the day that the daily entry was made. Following Cycle 1, changes were made to the diary to make the application clearer, which we expected would increase adherence. The most significant changes are shown in Table 4.

    In Cycle 2, statistically significant improvements in adherence with headache entries were found following refinements to the first prototype. Participants in Cycle 2 completed their headache entries closer to the time pain began than did participants in Cycle 1 (Cycle 2 mdn=3.83 h; range, 0.09-19.92 hours, vs Cycle 1 mdn=13.59 h; U=32.00, z=-3.01, P=.003). Adherence with the daily diary entries also improved, but did not reach statistical significance (26% of participants, n=5, completed all 14 daily entries in Cycle 2 vs 18% of participants, n=2, in Cycle 1; χ2(1)=0.26,P=.69; 53% of participants, n=10, completed 75% of the 14 daily diary entries in Cycle 2 vs 18% of participants, n=2, in Cycle 1; χ2(1)=3.44,P=.12). As observed in Cycle 1, participants in Cycle 2 also tended to answer all of the items when completing a headache or daily entry (see Table 2). Following Cycle 2, minor changes were made to the diary primarily to increase adherence (see Table 4).

    In Cycle 3, the level of participant adherence utilizing the diary for headache entry remained acceptable with no statistically significant differences found between Cycle 2 and Cycle 3 (see Table 2). As observed in Cycle 2, the majority of participants’ headache entries during Cycle 3 were made on the same day that the episode occurred. Once participants created the headache entry, they tended to report initial information about their headache right away. The level of adherence of participants utilizing the diary for entering the daily diary entries was not statistically different from Cycle 2. Participants completed the majority of daily entries in real-time with only a minority of daily entries entered retrospectively. Because significant improvements in feasibility indicators of the diary were not observed in Cycle 3, we decided not to make further changes. This was the final version of the diary used to test the psychometric properties.

    Table 2. . Median level of adherence with the myWHI diary for Cycles 1-3a.
    View this table

    Usage of Diary Features

    Table 3 summarizes how participants used the features of the diary.

    Table 3. Usage of diary features for Cycles 1-3.
    View this table

    Learnability, Acceptability, and Efficiency

    Information Collected Through the Online Post-Questionnaires

    Figure 4 shows participants’ opinions on attributes of the diary. In Cycle 1, 7 of 11 participants completed the post-questionnaire. In terms of acceptability, all participants reported that they would recommend this diary to others. Five participants (71%) expressed interest in continuing to use the diary (eg, for 6 more weeks). Six participants (55%) continued using the diary for 2 weeks following their 14-day trial, with 3 (27%) continuing to use it beyond that. This behavior occurred in the absence of any incentives or encouragement. In terms of efficiency, the headache diary item that was repeatedly reported to be the most difficult to complete was “potential triggers” (n=4, 57%).

    In Cycle 2, 17 of 19 participants completed the post-questionnaire. Participants’ opinions on the feasibility of the diary remained positive and improved when contrasted with opinions in Cycle 1 (Figure 4). The levels of acceptability and efficiency between the first 2 cycles were not statistically different, with the exception of fewer participants reporting trouble recording “potential triggers” for headache entries during Cycle 2, the item with the most problems for participants in Cycle 1 (Cycle 1: 57%, 4 of 7 participants vs Cycle 2: 0%, 0 of 19 participants; χ2(1)=11.66, P<.001). For learnability, participants in Cycle 2 reported the diary to be easier to learn how to use than did participants in Cycle 1 (z=-2.22, P=.03).

    In Cycle 3, 12 of 13 participants completed the post-questionnaire. The learnability, acceptability, and efficiency of the diary remained as positive as users’ experience during Cycle 2, and no statistically significant improvements were found (Figure 4).

    Figure 4. Participants’ feedback on attributes of the myWHI diary (mdn values). aNo participant used the paper-format diary in Cycle 1.
    View this figure
    Information Collected Through the End-of-Study Interviews

    In Cycle 1, 4 participants were interviewed. Thematic analysis revealed 3 distinct themes:

    1. Poor understanding: Many participants neglected to use features of the diary because they either did not know how the features worked, or were unaware that they existed (eg, participants were unaware that they could customize the triggers tracked on a daily basis). This lack of understanding was the prevailing message of the first cycle. As a potential solution, participants suggested the addition of tutorials.

    2. General endorsements, likes, and dislikes: Participants felt satisfied with the diary and perceived it as useful because it taught them what to pay attention to; helped increase awareness of headaches, identification of triggers, and effectiveness of medications; helped guide self-care behaviors; and helped in reporting headaches to physicians and family. Participants felt that the diary was easy to use overall, but identified some difficult and confusing situations, such as: setting the start and end time of headaches as wake and sleep times, respectively; including not applicable items (eg, symptoms before and after the headache) for those with constant headache; having to develop the habit of using the diary; identifying potential headache triggers within the diary’s hierarchical presentation; and the slowing of the application due to network connections.

    3. Suggestions for improvement: To improve ease of use, participants suggested: setting default answers to daily items, allowing participants to remove unused items, adding more reminders to complete either ongoing headache entries or daily entries, and providing the most frequently entered options at the top of lists to expedite data entry. They also suggested providing additional reports and adding the ability to export diary data.

    In Cycle 2, 3 participants were interviewed with 3 distinct themes identified:

    1. Good understanding: Participants demonstrated a good understanding of the functionality of the application. They found the tutorials that were added after Cycle 1 to be clear and useful.

    2. General endorsements, likes, and dislikes: Participants found the application easy to use, relevant, and useful. They reported that using the application gave them a more accurate idea of their headaches (ie, type of pain, intensity, and potential triggers). However, they did not speak highly of some sections or functionalities, such as: no interviewed participant reported using the Web-based reports that were added after Cycle 1, instead emphasizing the convenience of reviewing reports on the phone and indicating that they were likely to consult the reports more often after using the diary for a longer period of time. They did consult the iPhone reports, but not frequently. Participants also disliked being unable to enter daily diary entries retrospectively, being unable to enter daily entries that extended past midnight without starting a new day, or having to enter day-long headaches for constant headache.

    3. Suggestions for improvement: For ease-of-use improvements, participants suggested: adding the ability to be reminded about incomplete items in headache entries and the ability to see the day of the week when entering dates.

    In Cycle 3, 2 participants were interviewed and the following theme was identified:

    1. General endorsements, likes, and dislikes: Both participants liked the application, reporting that it was clear, easy to use, and useful. Despite their satisfaction with the overall application, a few features were not always perceived as useful. Whereas 1 participant used the comment section throughout headache episodes to track perceived changes, the other reported that the section was not useful. While 1 participant found the reports appealing and easy to interpret, the other found them confusing. Other aspects that were raised as a source of dissatisfaction or not used were: the loading time, the difficulty deciding what daily triggers to track due to the large number of available options, and the Web-based reports. They expressed again that access to the reports from within the mobile application would be more beneficial than accessing them through the Web.

    Summary of Diary Changes

    As part of the iterative design process, the diary was refined following Cycles 1 and 2 based on participant feedback. Table 4 summarized the most important changes made.

    Table 4. Most important changes made to the diary during testing.
    View this table

    Psychometric Properties Testing of the Diary

    The 65 participants completed a mean of 10.32 daily diary entries (SD=4.94) during the 14-day period. A total of 33 participants (65%) completed all 14 daily diary entries and 44 (67%) completed more than 75% of the daily diary entries.

    Generalizability theory results for single-item measures recorded using the NRS-11 scale are located in Table 5. The largest portion of variance for headache start intensity, maximum intensity, and duration was explained by person variability (62.79% to 75.60%) suggesting that these variables are highly stable across individuals across 14 days. In contrast, unpleasantness, amount of sleep, quality of sleep, and mood was primarily characterized by person-by-day variability (60.27% to 79.21%) suggesting that these variables tended to vary substantially from day to day. However, all variables had substantial person and person-by-day variability suggesting that these variables capture both trait-like individual differences and state-like daily fluctuations. Reliability for PedMIDAS was not calculated due to the small sample size of users between the ages of 14 and 17 years old. Generalizability theory results for MIDAS (Table 6) shows that MIDAS has substantial person and person-by-day variance, but also has a substantial amount of measurement error. The various estimates of reliability were excellent (RKF=0.98), adequate (R1F=0.66), and somewhat low (RC=0.51), depending on the measure used. These results suggest that MIDAS has good reliability if data from all 14 days are aggregated to create a single estimate for a person, but may not be suitable for measuring day-to-day variations in headache impact.

    Table 5. Variance components for single-item measures of the diary.
    View this table
    Table 6. Variance components for multi-item measures of the diary.
    View this table

    Table 7 summarizes the correlations used to validate several sections of the diary. Although we administered The Sleep/Wake Behavior Problems Scale of the Sleep Habits Survey [40] and the child version of the PROMIS Pediatric Short Form—Pain Interference [41] this information was not used in our analyses due to the reduced number of participants between the ages of 14 and 17 years old (n=6). All convergent construct validity correlations were statistically significant, and 4 of 5 had large effect sizes (rs>|0.50|). These correlations ranged from -0.26 (average sleep quality recorded with the PROMIS Sleep Disturbance scale) to 0.81 (highest headache pain recorded on the diary with retrospective recall of highest intensity). A total of 8 of the 12 concurrent validity correlations were statistically significant with medium to large effect sizes, ranging from -0.22 (average headache unpleasantness and MHI-5 emotional functioning) to 0.55 (highest headache pain intensity on the diary and the adult PROMIS Pain Interference scale). Due to a very small sample of adolescent participants (n=6), 10 planned correlations using adolescent measures (for participants between 14 and 17 years old) were omitted.

    Table 7. Convergent and concurrent validity Spearman rank-order correlations for sections of the diary.
    View this table

    Case Study Analysis

    A case study analysis of 1 participant of psychometric properties testing was conducted. The results are presented to illustrate how it is possible to improve the quality of self-reported data, consequently helping users and health care professionals to better understand headaches and improve health care decisions. Figure 5 shows screenshots of the Web-based reports of a 21-year-old female participant. This participant experienced migraine-like headaches, and recorded 6 episodes mainly of moderate headache pain over the 14 days. Most episodes lasted less than 1 hour, 2 episodes lasted around 2 and 8 hours, respectively, and caused mild to moderate interference in her life. The participant reported symptoms such as dizziness or hearing changes before the onset of half of the episodes, and often reported accompanying symptoms such as vomiting and light or noise sensitivity. The participant daily kept track of potential triggers, including: menstrual period, stress, and tiredness. There may be relationships between her stress and the occurrence of headache episodes, but it is too premature to extract conclusions from these limited observations. Besides taking ibuprofen on 2 occasions, which did not seem to help ease the pain in a timely manner, the participant reported different strategies to cope with her headache episodes, and perceived eating breakfast, rest, and sleep as the most effective. However, because of the limited duration of using the headache diary, again no conclusions about the effectiveness of these strategies can be drawn, however, this may be clinically helpful.

    Figure 5. Screenshots of the Web-based reports of one of the participants.
    View this figure


    Principal Results

    After 3 iterations of designing, testing, reviewing, and refining, we created a feasible electronic headache diary. At the end of this iterative process, young adults with headaches perceived the application as useful, easy to learn, and efficient to use. Although adherence with the final version evaluated in Cycle 3 was not perfect, most participants made active use of the application throughout the testing period. The option for participants to omit items when completing the diary could have had drawbacks to this research by threatening maintenance of internal consistency while diminishing the usefulness of the diary in clinical settings. However, participants typically provided an answer to each item included in the diary when either completing a daily entry or recording a headache episode during the testing period. Moreover, most participants completed a majority of the daily entries and recorded most headache episodes on the day they occurred. Substantial differences have been observed in levels of adherence with electronic headache diaries, with rates of 75%-98% [22]. There is no consensus on the acceptable minimum adherence level in self-monitoring systems. The ultimate goal when developing a diary such as this is to provide individuals with headaches (and their health care providers) with the greatest insight into their condition while imposing minimum time and effort on maintaining a diary. This delicate balance poses a challenge. It is possible that the comprehensiveness of our application, although recognized as a strength by most participants, may have reduced adherence due to the time and effort required to complete all of the items. However, it may also take time to develop a habit of using the diary. We found that 35% of participants were still creating diary entries 2 weeks after the conclusion of their 14-day participation, and 18% created more diary entries after the study period than they had within it. This suggests that level of adherence with the diary may be maintained or even improved over time. This is most apparent with the Web-based reports, which can provide enhanced value after regular use of the application over time. It is challenging to motivate users to regularly enter data when the benefits are uncertain and meaningful reports may be weeks or even months away. Therefore, despite the possible issues with the time and effort required due to the comprehensiveness of our application, we do not plan to simplify this application, mainly because simplicity does not automatically imply that this will help patients improve adherence, and most importantly, we do not know the minimum level of adherence required to achieve better outcomes. In addition to the observed levels of participant adherence when filling in the diary, we also observed that the features incorporated in the diary (ie, the reminders, the comment sections, and the customizable list of potential triggers for the users to track their exposure on a daily basis and explore their potential relevance) were commonly used by the participants, with the exception of the Web-based report generation system that was external to the application.

    We assessed reliability of measures included in the final version of the diary using generalizability theory analyses. For single-item measures, these analyses revealed that headache intensity and duration measures were primarily trait-like variables that were highly stable over time. In contrast, headache unpleasantness, amount/quality of sleep, and mood measures were more state-like, and varied considerably from day-to-day. The MIDAS had excellent internal consistency when all 14 days are combined together into a single composite variable; however, it does not appear to be a good measure for reliably measuring variability in headache-related interference from day to day. This said, poor within-subjects reliability is typical of many measures that otherwise have excellent psychometric properties [43]. Consequently, we recommend that patients and health care providers who use the diary not to examine day-to-day fluctuations in MIDAS scores. Instead, they should calculate an overall index of headache-related interference by averaging MIDAS scores across many measurement occasions and look for changes over longer periods of time using small-N, A-B-A designs. Finally, we also explored convergent and criterion validity for some critical elements of the myWHI diary, especially when used by the oldest subgroup.

    Strengths and Weaknesses of the Mywhi Application

    The myWHI diary overcomes several drawbacks identified in the electronic diaries described and used in the scientific literature [22] or available on the market [23]. It allows users to track information for both headache and headache-free days, and has undergone formal usability, feasibility, and psychometric testing. Unlike the myWHI diary, the usability, feasibility, and reliability of current headache diaries are often unknown. Available diaries have been fundamentally designed to only log headaches and the main variables associated with them (ie, intensity, duration, and timing). Diaries often do not collect information in the absence of headaches, as they are event-contingent, making it difficult for the user (or health care professionals) to understand what is precipitating the headaches, and what strategies may prevent new headaches; the digital headache diary (DHD) is one exception [45]. The myWHI diary meets the criteria, which our team defined a priori, for an “ideal” diary application; it is intended to help individuals with headache to better understand and manage their headaches, while providing relevant data to health professionals [23]. The myWHI diary was created with the help of headache experts, it measures clinically relevant headache variables, it allows customization to make the application clinically relevant to the individual user, and it generates reports displaying relationships between variables.

    Despite the high levels of satisfaction with the application, we also identified weaknesses. There were weaknesses with our Web-based report generation system that is external to the application, which was not used by the participants. Incorporating a robust set of reports directly in the mobile application is something to consider. The relatively slow responsiveness of the application influences user experience; we stored data on a remote server for research purposes, which introduced delays users are not accustomed to when using mobile apps. This could be resolved in a new version of the application that stores data locally, with the potential for a data synchronization system to maintain availability of usage data to researchers. We acknowledge the need to adapt the diary for other mobile phone platforms or Web-based application. Finally, taking into account that the MIDAS scores seem inappropriate for day-to-day reporting, it would be desirable to replace this measure with a new one that could examine daily changes. However, more psychometric work is required to develop a new measure.

    Limitations of the Study

    This study has some limitations. First, participants across a wide range of ages were included. We encountered recruitment difficulties for participants between 14 and 17 years old, which limits generalizability of our psychometric findings to this age group. Second, introducing a routine such as using this diary every day is neither easy nor fast. When testing usability and feasibility of the diary, we offered participants a prototype version of the diary to use for 14 days. With this time period, which is shorter than the 28-day period that is often recommended for assessment in headache studies [28,29,46], the use of the diary was not effortlessly adapted as daily routine. This could have negatively affected use of the diary, quality of the suggestions, and even bias modifications taken as result of the collected data. A longer trial duration could have minimized these problems and provide a more accurate picture of feasibility data when this diary is used as consistently recommended in the literature. Third, findings from interviews conducted in each cycle of usability/feasibility were not representative of the participating sample in each cycle as only 15%-25% of participants were involved in the interviews. Despite this inherent characteristic of qualitative research, the data helped to provide richer insight of some participant experiences using the diary. Similarly, the case study is not generalizable; however, it provides an example of how this diary may be helpful. Finally, allowing participants to freely omit some items on the questionnaire may result in greater levels of missing data. In a clinical context, practitioners should work with patients closely to determine which questionnaire items are most important to track on a case-by-case basis.

    Conclusions and Next Steps

    This research represents the necessary first steps toward creating a feasible and psychometrically sound electronic diary for young adults with headaches. Users should be continuously involved during the design of applications [47] and we involved users from the onset [48]. The application was evaluated using formal testing cycles with participants once we had a high-fidelity and fully functional prototype of the application to provide an accurate and realistic setting for evaluation, following Prinz’s recommendations [49]. This diary may be useful not only for individuals with headaches but also for medical doctors who want to collect accurate and thorough information. Without a diary, medical doctors may be forced to make care decisions on the basis of limited retrospective information collected during brief and sporadic encounters. They will now have an accurate picture of their patients’ headaches in order to tailor headache treatments to each patient, which ultimately may improve treatment outcomes. The diary records of frequency, severity, duration, location, qualities of headaches, the level of disability, and its associated symptoms can help to consolidate headache diagnosis. The exposure to potential triggers and perceived precipitating factors can help identify headache triggers, and its records of frequency of analgesic use and other coping strategies can help to determine the best methods for managing pain.

    Since there is still no evidence indicating the multiple potential benefits of electronic diaries in the field of headache [50] our plan for the future is to evaluate the impact of this diary at multiple levels (eg, does the use of myWHI save clinician time? does it facilitate diagnosis? does it improve patient outcomes?). The authors plan to make this diary publically available soon at the myWHI website.


    The authors thank Carl von Bayer, PhD, Mark Connelly, PhD, Bo Larsson, MD, Mattias Linde, MD, Tonya Palermo, PhD, Laurie Pereles, MD, Allan Purdy, MD, and 10 anonymous individuals with headache for their participation in the review of an initial draft of the diary items. The authors would also like to thank Zack Miller for his technical assistance in helping us to extract data from the diary system for data analysis. Dr. McGrath’s research is supported by a Canada Research Chair. This project has been funded by a Canadian Institutes of Health Research (CIHR) operating grant (grant #97981).

    Conflicts of Interest

    The authors have no conflicts of interest to declare.


    1. Vos T, Flaxman AD, Naghavi M, Lozano R, Michaud C, Ezzati M, de Vaccaro Karen Courville, Des Jarlais Don C, Fowkes Francis Gerry R, Narayan K M Venkat, De Leòn Felipe Rodriguez, Stapelberg Nicolas J C, van der Werf Marieke J, Williams Sean R M, Zaidi Anita K M, Murray Christopher J L, et al. Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet 2012 Dec 15;380(9859):2163-2196. [CrossRef] [Medline]
    2. Headache Classification Committee of the International Headache Society (IHS). The International Classification of Headache Disorders, 3rd edition (beta version). Cephalalgia 2013 Jul;33(9):629-808. [CrossRef] [Medline]
    3. Dowson AJ, Sender J, Lipscombe S, Cady RK, Tepper SJ, Smith R, van Duijn N P, et al. Establishing principles for migraine management in primary care. Int J Clin Pract 2003;57(6):493-507. [Medline]
    4. Peng KP, Wang SJ. Migraine diagnosis: screening items, instruments, and scales. Acta Anaesthesiol Taiwan 2012 Jun;50(2):69-73. [CrossRef] [Medline]
    5. Winkler R, Underwood P, Fatovich B, James R, Gray D. A clinical trial of a self-care approach to the management of chronic headache in general practice. Soc Sci Med 1989;29(2):213-219. [Medline]
    6. Baldacci F, Vedovello M, Ulivi M, Vergallo A, Poletti M, Borelli P, et al. How aware are migraineurs of their triggers? Headache 2013 May;53(5):834-837. [CrossRef] [Medline]
    7. Moloney MF, Aycock DM, Cotsonis GA, Myerburg S, Farino C, Lentz M. An Internet-based migraine headache diary: issues in Internet-based research. Headache 2009 May;49(5):673-686. [Medline]
    8. Nicholson RA, Buse DC, Andrasik F, Lipton RB. Nonpharmacologic treatments for migraine and tension-type headache: how to choose and when to use. Curr Treat Options Neurol 2011 Feb;13(1):28-40. [CrossRef] [Medline]
    9. WHO. Neurological Disorders: Public Health Challenges. Geneva, Switzerland: World Health Organization; 2006.
    10. Martin PR. Behavioral management of migraine headache triggers: learning to cope with triggers. Curr Pain Headache Rep 2010 Jun;14(3):221-227. [CrossRef] [Medline]
    11. Huguet A, McGrath PJ, Stinson J, Tougas ME, Doucette S. Efficacy of psychological treatment for headaches: an overview of systematic reviews and analysis of potential modifiers of treatment efficacy. Clin J Pain 2014 Apr;30(4):353-369. [CrossRef] [Medline]
    12. Connelly M, Bickel J. An electronic daily diary process study of stress and health behavior triggers of primary headaches in children. J Pediatr Psychol 2011 Sep;36(8):852-862 [FREE Full text] [CrossRef] [Medline]
    13. van den Brink M, Bandell-Hoekstra EN, Abu-Saad HH. The occurrence of recall bias in pediatric headache: a comparison of questionnaire and diary data. Headache 2001 Jan;41(1):11-20. [Medline]
    14. Sorbi MJ, Mak SB, Houtveen JH, Kleiboer AM, van Doornen Lorenz J P. Mobile Web-based monitoring and coaching: feasibility in chronic migraine. J Med Internet Res 2007;9(5):e38 [FREE Full text] [CrossRef] [Medline]
    15. Stinson JN, Jibb LA, Nguyen C, Nathan PC, Maloney AM, Dupuis LL, et al. Development and testing of a multidimensional iPhone pain assessment application for adolescents with cancer. J Med Internet Res 2013;15(3):e51 [FREE Full text] [CrossRef] [Medline]
    16. Palermo TM, Valenzuela D, Stork PP. A randomized trial of electronic versus paper pain diaries in children: impact on compliance, accuracy, and acceptability. Pain 2004 Feb;107(3):213-219. [Medline]
    17. Stone AA, Shiffman S. Capturing momentary, self-report data: a proposal for reporting guidelines. Ann Behav Med 2002;24(3):236-243. [Medline]
    18. Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient compliance with paper and electronic diaries. Control Clin Trials 2003 Apr;24(2):182-199. [Medline]
    19. Affleck G, Zautra A, Tennen H, Armeli S. Multilevel daily process designs for consulting and clinical psychology: a preface for the perplexed. J Consult Clin Psychol 1999 Oct;67(5):746-754. [Medline]
    20. Burton C, Weller D, Sharpe M. Are electronic diaries useful for symptoms research? A systematic review. J Psychosom Res 2007 May;62(5):553-561. [CrossRef] [Medline]
    21. Lewandowski AS, Palermo TM, Kirchner HL, Drotar D. Comparing diary and retrospective reports of pain and activity restriction in children and adolescents with chronic pain conditions. Clin J Pain 2009 May;25(4):299-306 [FREE Full text] [CrossRef] [Medline]
    22. Stinson JN, Huguet A, McGrath P, Rosenbloom B, Soobiah C, White M, et al. A qualitative review of the psychometric properties and feasibility of electronic headache diaries for children and adults: where we are and where we need to go. Pain Res Manag 2013;18(3):142-152 [FREE Full text] [Medline]
    23. Hundert AS, Huguet A, McGrath PJ, Stinson JN, Wheaton M. Commercially available mobile phone headache diary apps: a systematic review. JMIR Mhealth Uhealth 2014;2(3):e36 [FREE Full text] [CrossRef] [Medline]
    24. Skeels MM, Pratt W. Participatory design with health consumers. AMIA Annu Symp Proc 2008:1136. [Medline]
    25. Dworkin RH, Turk DC, Farrar JT, Haythornthwaite JA, Jensen MP, Katz NP, IMMPACT. Core outcome measures for chronic pain clinical trials: IMMPACT recommendations. Pain 2005 Jan;113(1-2):9-19. [CrossRef] [Medline]
    26. McGrath PJ, Walco GA, Turk DC, Dworkin RH, Brown MT, Davidson K, von Baeyer Carl L, PedIMMPACT. Core outcome domains and measures for pediatric acute and chronic/recurrent pain clinical trials: PedIMMPACT recommendations. J Pain 2008 Sep;9(9):771-783. [CrossRef] [Medline]
    27. Turk DC, Dworkin RH, Allen RR, Bellamy N, Brandenburg N, Carr DB, et al. Core outcome domains for chronic pain clinical trials: IMMPACT recommendations. Pain 2003 Dec;106(3):337-345. [Medline]
    28. Bendtsen L, Bigal ME, Cerbo R, Diener HC, Holroyd K, Lampl C, International Headache Society Clinical Trials Subcommittee. Guidelines for controlled trials of drugs in tension-type headache: second edition. Cephalalgia 2010 Jan;30(1):1-16. [CrossRef] [Medline]
    29. Penzien DB, Andrasik F, Freidenberg BM, Houle TT, Lake AE, Lipchik GL, American Headache Society Behavioral Clinical Trials Workgroup. Guidelines for trials of behavioral treatments for recurrent headache, first edition: American Headache Society Behavioral Clinical Trials Workgroup. Headache 2005 May;45 Suppl 2:S110-S132. [CrossRef] [Medline]
    30. Tfelt-Hansen P, Block G, Dahlöf C, Diener HC, Ferrari MD, Goadsby PJ, International Headache Society Clinical Trials Subcommittee. Guidelines for controlled trials of drugs in migraine: second edition. Cephalalgia 2000 Nov;20(9):765-786. [Medline]
    31. von Baeyer Carl L, Spagrud LJ, McCormick JC, Choo E, Neville K, Connelly MA. Three new datasets supporting use of the Numerical Rating Scale (NRS-11) for children's self-reports of pain intensity. Pain 2009 Jun;143(3):223-227. [CrossRef] [Medline]
    32. Jensen MP, Turner JA, Romano JM, Fisher LD. Comparative reliability and validity of chronic pain intensity measures. Pain 1999 Nov;83(2):157-162. [Medline]
    33. Jensen MP, Karoly P, Braver S. The measurement of clinical pain intensity: a comparison of six methods. Pain 1986 Oct;27(1):117-126. [Medline]
    34. Berwick DM, Murphy JM, Goldman PA, Ware JE, Barsky AJ, Weinstein MC. Performance of a five-item mental health screening test. Med Care 1991 Feb;29(2):169-176. [Medline]
    35. Rumpf HJ, Meyer C, Hapke U, John U. Screening for mental health: validity of the MHI-5 using DSM-IV Axis I psychiatric disorders as gold standard. Psychiatry Res 2001 Dec 31;105(3):243-253. [Medline]
    36. Strand BH, Dalgard OS, Tambs K, Rognerud M. Measuring the mental health status of the Norwegian population: a comparison of the instruments SCL-25, SCL-10, SCL-5 and MHI-5 (SF-36). Nord J Psychiatry 2003;57(2):113-118. [CrossRef] [Medline]
    37. Buysse DJ, Yu L, Moul DE, Germain A, Stover A, Dodds NE, et al. Development and validation of patient-reported outcome measures for sleep disturbance and sleep-related impairments. Sleep 2010 Jun;33(6):781-792 [FREE Full text] [Medline]
    38. Amtmann D, Cook KF, Jensen MP, Chen WH, Choi S, Revicki D, et al. Development of a PROMIS item bank to measure pain interference. Pain 2010 Jul;150(1):173-182 [FREE Full text] [CrossRef] [Medline]
    39. Wolfson AR, Carskadon MA. Sleep schedules and daytime functioning in adolescents. Child Dev 1998 Aug;69(4):875-887. [Medline]
    40. Headache Classification Subcommittee of the International Headache Society. The International Classification of Headache Disorders: 2nd edition. Cephalalgia 2004;24 Suppl 1:9-160. [Medline]
    41. Varni JW, Stucky BD, Thissen D, Dewitt EM, Irwin DE, Lai JS, et al. PROMIS Pediatric Pain Interference Scale: an item response theory analysis of the pediatric pain item bank. J Pain 2010 Nov;11(11):1109-1119 [FREE Full text] [CrossRef] [Medline]
    42. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006 Jan;3(2):77-101. [CrossRef]
    43. Cranford JA, Shrout PE, Iida M, Rafaeli E, Yip T, Bolger N. A procedure for evaluating sensitivity to within-person change: can mood measures in diary studies detect change reliably? Pers Soc Psychol Bull 2006 Jul;32(7):917-929 [FREE Full text] [CrossRef] [Medline]
    44. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale, NJ: L Erlbaum Associates; 1988.
    45. Allena M, Cuzzoni MG, Tassorelli C, Nappi G, Antonaci F. An electronic diary on a palm device for headache monitoring: a preliminary experience. J Headache Pain 2012 Oct;13(7):537-541 [FREE Full text] [CrossRef] [Medline]
    46. Silberstein S, Tfelt-Hansen P, Dodick DW, Limmroth V, Lipton RB, Pascual J, Task Force of the International Headache Society Clinical Trials Subcommittee. Guidelines for controlled trials of prophylactic treatment of chronic migraine in adults. Cephalalgia 2008 May;28(5):484-495. [CrossRef] [Medline]
    47. Lynch T, Gregor S. User participation in decision support systems development: Influencing system outcomes. Eur J Inf Syst 2004 Nov 02;13(4):286-301. [CrossRef]
    48. Huguet A, Stinson J, Mackay B, Watters C, Tougas M, White M, et al. Bringing psychosocial support to headache sufferers using information and communication technology: lessons learned from asking potential users what they want. Pain Res Manag 2014;19(1):e1-e8 [FREE Full text] [Medline]
    49. Prinz W, Mark G, Pankoke-Babatz U. Designing groupware for congruency in use. In: CSCW '98: computer supported cooperative work: ACM 1998 Conference on Computer Supported Cooperative Work: proceedings, Seattle, Washington, November 14-18, 1998. New York, NY: The Association for Computing Machinery; 1998:373-382.
    50. Marceau LD, Link C, Jamison RN, Carolan S. Electronic diaries as a tool to improve pain management: is there any evidence? Pain Med 2007 Oct;8 Suppl 3:S101-S109. [CrossRef] [Medline]


    DHD: daily headache diary
    HA: Headache
    MIDAS: Migraine Disability Assessment Scale
    Mdn: Median
    myWHI diary: myWireless Headache Intervention diary
    PROMIS: Patient Reported Outcomes Measurement Information System

    Edited by G Eysenbach; submitted 20.09.14; peer-reviewed by N Maleki, L Simons; comments to author 24.12.14; revised version received 14.02.15; accepted 02.03.15; published 08.05.15

    ©Anna Huguet, Patrick J McGrath, Michael Wheaton, Sean P Mackinnon, Sharlene Rozario, Michelle E Tougas, Jennifer N Stinson, Cathy MacLean. Originally published in JMIR Mhealth and Uhealth (, 08.05.2015.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.