Given that no standards exist for evaluating these apps, the authors consensually defined a set of criteria for an ideal headache diary app intended to help headache sufferers better understand and manage their headaches, while providing relevant data to health professionals. Based on the authors’ judgment, an ideal headache diary app should (1) be created with clinical and/or scientific headache expertise, (2) have undergone testing to ensure the diary is a feasible and reliable method of data collection, (3) measure clinically relevant headache variables, (4) be usable, (5) include customizable answer options and reports, (6) include reports linking multiple variables, and (7) have the ability to export headache data from the app. See below for how each of these criteria, intended as minimum requirements for an acceptable headache diary app, were evaluated.

An appropriate app does not necessarily need to be developed by headache experts themselves, but it is important that experts be involved at least in advising development. For this reason, we a priori defined that an ideal app be created with headache expertise. The app description available in the app store and any websites linked to the developer, creator, or institution affiliated with app development were examined for scientific or clinical headache expertise. The apps found to be supported by academic or clinical institutions, or created by individuals with MDs or PhDs practicing or doing research in the fields of neurology or pain were considered to have been created with headache expertise. The method used to identify expert involvement was chosen as a feasible strategy. It is possible that headache experts may have been involved in development but not identified in the app descriptions or associated websites. We also acknowledge that headache sufferers can be considered experts in creating diary apps. However, they were not included in this criterion as it was not possible to reliably confirm whether the app creators held this type of expertise.

To examine whether the feasibility—described in terms of adherence, acceptability, learnability, efficiency, or accuracy—and psychometric properties of the existing apps could have been formally tested, a search of the following databases was conducted: PubMed, Web of Knowledge, and PsychINFO (2000 to October 24, 2013). The search did not include publications prior to 2000; the oldest app versions included in this review were released in 2010. The search terms included “headache or migraine or cephalalgia” and “diary or diaries”. A total of 1442 abstracts were retrieved from our search strategy. Two reviewers (ASH, AH) independently screened all retrieved abstracts (n=723 after removing duplicates) for e-diaries or mobile phone diaries matching the names of the apps, their developers’ names, or descriptions of the content of the apps included in this review. The systematic review of the headache e-dairies developed and used in the academic setting was conducted recently by several members of this research team and was taken into account [8]. Using Cohen’s kappa, the level of agreement between the 2 reviewers screening the abstracts was 1.00, indicating perfect agreement [19,20]. We also acknowledge that the apps may have undergone psychometric or feasibility testing that was not published in the scientific literature. However, it was not possible to verify whether such testing had occurred.

There is no consensus on a standard set of core variables that should be assessed in a headache diary. Consequently, the authors created and conducted an online survey among headache experts to define what the most clinically relevant headache variables for a headache diary app should be. Headache experts were required to (1) have an MD or PhD, (2) be affiliated with recognized universities, (3) be currently conducting research and/or practicing in the field of neurology or pain, and (4) be published in peer-reviewed journals on the topic of headaches. We identified and invited 35 headache experts to participate. Of the 35 experts contacted, 10 responded. Experts were independently asked to create a list including all variables they believed should be measured in a mobile headache diary. Responses were compiled and comparable responses grouped under the same variable (eg, headache severity, headache intensity, and pain level were grouped together). For a complete list of headache variables recommended by the experts, see Table 1. Those variables suggested by 50% or more of the headache experts were considered clinically relevant. A reviewer (ASH) assessed each of the apps for inclusion of the clinically relevant variables suggested by the experts.

An ideal app was expected to be usable. Usability is a qualitative attribute which assesses how easy user interfaces are to use and understand [21]. Usability was assessed using a heuristic evaluation, which consists of a small number of expert evaluators assessing the user interface against a list of heuristics, defined as general principles for interaction design [21]. Heuristic evaluation is one of the most common methods of usability assessment. It benefits from being an efficient evaluation method for obtaining high-quality results in a short amount of time, and at a low cost [21-23]. Usability can also be assessed using a variety of methods by users themselves, such as the think aloud protocol, which consists of verbal reports from users [23].

In the current review, each app user interface was systematically inspected, and its compliance with a common list of 10 well-established usability heuristics (see Table 2 for a description of each heuristic) was judged by trained reviewers [21]. Each app user interface was rated on a scale of 1 to 5 (1=poor, 5=excellent) against each of the 10 heuristics and a total usability score was obtained by summing the individual scores (maximum score of 50). The calculated usability score for each app was not intended to be used as a precise indicator of its usability; instead it was intended to be used as an approximate indicator, with higher scores indicating higher perceived usability. One reviewer (ASH) was trained for usability evaluation and evaluated all included apps. For the purpose of exploring interrater reliability, a second reviewer (MW), a software developer with expertise in developing medical apps and testing usability, evaluated the usability of a random selection (9/38, 24%) of the apps. Usability scores are subjective and slight variation between reviewers is expected. Interrater reliability of the total usability scores was assessed using a two-way mixed, absolute agreement, average-measures intraclass correlation (ICC) [24]. Unlike kappa, ICC incorporates magnitudes of disagreement, making it more suitable for evaluating interrater reliability of ratio variables [24]. The resulting ICC was .95, indicating excellent agreement between reviewers [25]. Given that strong agreement was identified between reviewers, it was not considered necessary for the second reviewer (MW) to evaluate more than 24% of the apps. For all of the apps, the first reviewer’s (ASH) scores were used in the presented data. Usability scores of 75% (equivalent to a score of 37.5 out of a maximum score of 50) or higher were considered acceptable for meeting the app criteria.

Customizable answer options are important in making the apps relevant to each user. This feature allows users to create their own inputs when filling out a diary entry. For example, the possibility for the user to add a custom trigger (eg, chocolate, caffeine, or stress) in case the desired trigger does not appear in the default list. To meet this criterion, the apps were required to have at least one headache variable answer option input be customizable and to contain some level of customization in the reports. Customizable reports allow users to better understand their headaches by allowing them to examine the trends that are a concern to them. Examples of customizable reports include controlling the time span of a report or choosing the variables contained in a report. One reviewer (ASH) extracted the required information by reviewing the content of the apps.

Reports allow users to understand trends associated with their headaches. This criterion required that the apps include reports simultaneously linking multiple variables in tables or graphs. For example, a report displaying information about both time of day and headache occurrence was considered to be a report linking multiple variables (time of day and headache occurrence). One reviewer (ASH) extracted the required information by reviewing the content of the apps.

The final app criterion required that the apps include an export feature, allowing users to export logged headache data directly to email, PDF, etc, and allowing the data to be viewed and saved outside the app. This feature is important as it facilitates sharing users’ headache data with their health care professionals. One reviewer (ASH) extracted the required information by reviewing the content of the apps.

The quality of the apps was determined by how many app criteria were met. The apps with the highest quality were iHeadache (developed by Better QOL), ecoHeadache (developed by ecoTouchMedia), and Headache Diary Pro (developed by Froggyware), each of which met 5 of the 7 app criteria. See Figure 1 for a screenshot of the 3 highest scoring apps. Only 7 of the 38 available apps met 4 or more of the app criteria. The median number of app criteria met was 2. Table 4 shows the number of apps meeting each criterion.

Only 7 apps (18%) were found to have been created with scientific or clinical expertise and met criterion #1. None of the apps in this review were found in the scientific literature search, and as a result none of the apps were considered to have undergone formal psychometric or feasibility testing (criterion #2).

Of the 38 apps, 7 of them (18%) measured all 7 clinically relevant headache variables as defined by app criterion #3. The average number of headache variables measured in each app was 7 out of 24. The average number of variables measured per app that were identified as clinically relevant by the experts was 5 out of 7. The most common variable measured was headache intensity (37/38, 97%), followed by medication usage (30/38, 79%), triggers (27/38, 71%), time of headache (27/38, 71%), notes/comments (26/38, 68%), other headache symptoms (25/38, 66%), headache duration (25/38, 66%), location of headache (21/38, 55%), headache disability (12/38, 32%), headache quality (11/38, 29%), and other coping strategies (11/38, 29%). Other variables less frequently measured were geographical location, weather, mood, and headache type. For a complete list of headache variables measured by those apps meeting 4 or more app criteria, see Table 5.

Of the 38 apps, 24 (63%) met this criterion, which consisted of scoring a total usability score of at least 75%. Usability scores ranged from 0% to 94% with a median score of 80%.

Most of the apps (27/38, 71%) contained reports on headache data, with 58% (22/38) of the apps containing reports linking multiple variables, while customizable reports were less common (9/38, 24%). The ability to modify existing lists of answer options for a headache variable (eg, adding a new trigger to the preexisting list) was seen in 63% (24/38) of the apps. Many of the apps (25/38, 66%) also allowed data entered into the diary to be exported, often via email or by creating PDF documents.