This paper is in the following e-collection/theme issue:

Evaluation and Research Methodology for mHealth (210) mHealth for Symptom and Disease Monitoring, Chronic Disease Management (1428) mHealth for Data Collection and Research (981) mHealth for Treatment Adherence (404)

Published on in Vol 10, No 6 (2022): June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/31491, first published .
Validation Parameters of Patient-Generated Data for Digitally Recorded Allergic Rhinitis Symptom and Medication Scores in the @IT.2020 Project: Exploratory Study

Validation Parameters of Patient-Generated Data for Digitally Recorded Allergic Rhinitis Symptom and Medication Scores in the @IT.2020 Project: Exploratory Study

Validation Parameters of Patient-Generated Data for Digitally Recorded Allergic Rhinitis Symptom and Medication Scores in the @IT.2020 Project: Exploratory Study

Authors of this article:

Stephanie Dramburg1 Author Orcid Image ;   Serena Perna1 Author Orcid Image ;   Marco Di Fraia1 Author Orcid Image ;   Salvatore Tripodi2, 3 Author Orcid Image ;   Stefania Arasi1, 4 Author Orcid Image ;   Sveva Castelli1 Author Orcid Image ;   Danilo Villalta5 Author Orcid Image ;   Francesca Buzzulini5 Author Orcid Image ;   Ifigenia Sfika2 Author Orcid Image ;   Valeria Villella2 Author Orcid Image ;   Ekaterina Potapova1 Author Orcid Image ;   Maria Antonia Brighetti6 Author Orcid Image ;   Alessandro Travaglini6 Author Orcid Image ;   Pierluigi Verardo7 Author Orcid Image ;   Simone Pelosi8 Author Orcid Image ;   Paolo Maria Matricardi1 Author Orcid Image
Article Authors Cited by (2) Tweetations (2) Metrics

    Original Paper

    1Department of Pediatric Respiratory Medicine, Immunology and Critical Care Medicine, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany

    2Pediatric Allergology Unit, Sandro Pertini Hospital, Rome, Italy

    3Allergology Service, Policlinico Casilino, Rome, Italy

    4Translational Research in Pediatric Specialities Area, Division of Allergy, Bambino Gesù Children's Hospital, Scientific Institute for Research, Hospitalization and Healthcare, Rome, Italy

    5Department of Immunology and Allergy, Santa Maria degli Angeli Hospital, Pordenone, Italy

    6Department of Biology, University of Rome Tor Vergata, Rome, Italy

    7Center of Aerobiology, Regional Agency for the Protection of the Environment, Pordenone, Italy

    8TPS Production, Rome, Italy

    Corresponding Author:

    Paolo Maria Matricardi, MD

    Department of Pediatric Respiratory Medicine, Immunology and Critical Care Medicine

    Charité-Universitätsmedizin Berlin

    Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin

    Augustenburgerplatz, 1

    Berlin, 13353

    Germany

    Phone: 49 30 450 566 406

    Email: paolo.matricardi@charite.de


    Background: Mobile health technologies enable allergists to monitor disease trends by collecting daily patient-reported outcomes of allergic rhinitis. To this end, patients with allergies are usually required to enter their symptoms and medication repetitively over long time periods, which may present a risk to data completeness and quality in the case of insufficient effort reporting. Completeness of patient’s recording is easily measured. In contrast, the intrinsic quality and accuracy of the data entered by the patients are more elusive.

    Objective: The aim of this study was to explore the association of adherence to digital symptom recording with a predefined set of parameters of the patient-generated symptom and medication scores and to identify parameters that may serve as proxy measure of the quality and reliability of the information recorded by the patient.

    Methods: The @IT.2020 project investigates the diagnostic synergy of mobile health and molecular allergology in patients with seasonal allergic rhinitis. In its pilot phase, 101 children with seasonal allergic rhinitis were recruited in Rome and instructed to record their symptoms, medication intake, and general conditions daily via a mobile app (AllergyMonitor) during the relevant pollen season. We measured adherence to daily recording as the percentage of days with data recording in the observation period. We examined the patient’s trajectories of 3 disease indices (Rhinoconjunctivitis Total Symptom Score [RTSS], Combined Symptom and Medication Score [CSMS], and Visual Analogue Scale [VAS]) as putative proxies of data quality with the following 4 parameters: (1) intravariation index, (2) percentage of zero values, (3) coefficient of variation, and (4) percentage of changes in trend. Lastly, we examined the relationship between adherence to recording and each of the 4 proxy measures.

    Results: Adherence to recording ranged from 20% (11/56) to 100% (56/56), with 64.4% (65/101) and 35.6% (36/101) of the patients’ values above (highly adherent patients) or below (low adherent patients) the threshold of 80%, respectively. The percentage of zero values, the coefficient of variation, and the intravariation index did not significantly change with the adherence to recording. By contrast, the proportion of changes in trend was significantly higher among highly adherent patients, independently from the analyzed score (RTSS, CSMS, and VAS).

    Conclusions: The percentage of changes in the trend of RTSS, CSMS, and VAS is a valuable candidate to validate the quality and accuracy of the data recorded by patients with allergic rhinitis during the pollen season. The performance of this parameter must be further investigated in real-life conditions before it can be recommended for routine use in apps and electronic diaries devoted to the management of patients with allergic rhinitis.

    JMIR Mhealth Uhealth 2022;10(6):e31491

    doi:10.2196/31491

    Keywords

    allergic rhinitis; symptom scores; patient-generated data; patient-reported outcomes; mHealth; mobile health; health applications; allergies; allergy monitor; digital health; medication scores 


    Digital and mobile health technologies are creating new perspectives in many areas of research and medical care. One important aspect in both fields is the ability to easily collect patient-generated data via smartphone apps and connected devices such as wearables, diagnostic tools, or environmental sensors [1-5]. Although the use of patient-reported outcomes has become popular over the last decade [6-9], it is not sure how accurately the collected data represent the patient’s state, as recording is done without supervision [10,11]. In particular, daily reporting over a longer time period may be perceived as challenging and cause a certain degree of reporting fatigue. The risk of potentially lower quality owing to disengaged survey respondents has been described with different terms, most recently in the field of psychology as “insufficient effort reporting” [11,12]. However, consensus on methodologies assessing the quantity and quality of entered data is still missing. Proposed methodologies include the (1) response pattern approach [13], (2) response time approach [14], (3) infrequency approach [15], (4) inconsistency approach [16], and (5) the number of unanswered questions. Most of these methodologies, however, refer to single points of data collection and extended questionnaires, which make their application in a setting with daily data recording via smartphone apps difficult to impossible. As digital methods of data collection via openly available mobile apps usually generate very large data sets, new challenges for the analysis and interpretation apply, such as the lack of standard measures [17]. The importance of unified approaches to data recording has recently been underlined in the context of patient adherence, and computational solution approaches were published to support uniform data formats [18]. A representative example for the daily acquisition of patient-generated data are mobile apps related to seasonal allergic rhinitis [19,20]. A variety of apps has been published for patients with pollen allergies, providing exposure forecasts, individualized symptom prediction, symptom and medication diaries, and in some cases, the possibility of exchanging recorded data with the attending physician [3,10,21-23]. Although several studies have shown the potential of mobile technologies for research purposes and clinical disease management, only few address the topic of data quality and validation [24,25]. The purpose of this study is to retrospectively investigate 4 putative validation criteria to assess the quality of data longitudinally collected by patients with seasonal allergic rhinitis and defined as follows: (1) intravariation index, (2) percentage of zero values, (3) coefficient of variation, and (4) percentage of changes in trends. To this end, we have taken advantage of the patient-generated symptom and medication data set, which has been acquired via a mobile app in a cohort of patients with seasonal allergic rhinitis in the context of the @IT.2020 pilot project [26,27].


    Study Design

    The @IT.2020 pilot project is an observational clinical study on the impact of component-resolved diagnosis and digital symptom recording on the diagnosis of pollen allergy. In the context of this project, 101 patients experiencing seasonal allergic rhinitis were recruited in the Sandro Pertini Hospital in Rome. The detailed study protocol has been published previously [26,27]. Briefly, recruited patients underwent a medical examination first (T0), including skin prick testing, blood sampling, and clinical questionnaires. At the end of the visits, patients were instructed on the use of the AllergyMonitor (TPS Software Production) mobile app to monitor their symptoms of the eyes, nose, and lungs, as well as medication intake and the impact of allergy symptoms on their daily activities during an individual study period. After the monitoring period, all patients underwent a second medical examination (T1), including a repetition of the initial clinical questionnaires focused on the past pollen season.

    Ethics Approval

    The study design and procedures were approved by the local ethics committee “Comitato Etico Indipendente Lazio 2” (study 10-16, Protocol 9871—01/02/2016).

    AllergyMonitor App

    AllergyMonitor is a CE-certified smartphone app designed for the daily reporting of allergic symptoms of the eyes, nose, and lungs. Further, the impact of allergic symptoms on daily activities and sleep as well as the medication intake were recorded. In order to facilitate the correct recording of medication intake, the study doctor registered the patients’ individual medication via the back end of the app, and the patient’s tailored drug name and administration schedule appeared in the app’s front end.

    Symptom and Medication Scores

    The following symptom and symptom medication scores were used in this study: Rhinoconjunctivitis Total Symptom Score (RTSS, 0-18 points) [28], Combined Symptom and Medication Score (CSMS, 0-6 points) [29], and Visual Analogue Scale (VAS, 0-10 points) [30]. RTSS and CSMS were calculated automatically by the app for every reporting day on the basis of 4 questions on nasal symptoms (sneezing, rhinorrhea, nasal pruritus, nasal congestion), 2 questions on ocular symptoms (itchy eyes, watery eyes), and 3 questions on medication intake (antihistaminic drugs, local steroids, systemic steroids). The severity of each of the symptoms was also measured by the patient using 4 different emoticons (Figure 1), each one representing a distinct severity grade (no symptoms, mild, moderate, or severe). Overall, severity was also measured using VAS in response to the question “How do you feel in relation to your allergic symptoms today?”

    ‎
    Figure 1. Screenshots of the AllergyMonitor app, indicating the emoticons used to assess symptom severity.
    View this figure

    Adherence to Electronic Diary Recording

    Retrospective reporting of symptoms for missed days was only possible within 48 hours. After 2 days of missed reporting, an automated reminder appeared on the patient’s phone. After 3 days of missed recording, the patient received an individual email or phone call from the study center to ensure that no technical problems had occurred. Adherence was measured as the percentage of days with completed electronic diary recording in the monitoring period [26]. Patients with adherence to recording above or below the arbitrary threshold of 80% were defined as high or low adherence, respectively.

    Validation Parameters

    Data retrospectively obtained were summarized as numbers (n) and frequencies (%) if they were categorical and as mean (SD) or median (IQR) if quantitative. The prevalence of atopic sensitization to airborne allergens was evaluated for every patient via skin prick test results (cutoff for positivity: a wheal size of ≥3 mm). For every pollen period considered, adherence values were calculated for each patient. Four tentative parameters for validation of the data quality were calculated as follows.

    Intravariation Index

    For each ith subject (i = 1,,,,n), (1) the percentage of the number of variations between 2 consecutive days was calculated, , where t= 1,,,,,,, T is the indicator of time point considered, y is the value of the symptom medication score considered, and I is a binary variable, which is 1 when the condition in the brackets is verified, 0 otherwise; (2) the individual variation range was calculated (rVAR = (max(y1, …, yT) – min(y1, …, yT))/S, where S represents the unit increase for each symptom medication score, that is, S=1 for VAS and RTSS and S=0.167 for CSMS); and (3) the average of all individual intravariation index values was calculated, , for each symptom score or symptom medication score.

    Percentage of Zero Values

    Average of individual percentages of zero values (100*number of zero compiled values/actual compiled days)

    Coefficient of Variation

    Average of individual coefficient of variation (100*SD/mean)

    Percentage of Changes in Trend

    Average of individual percentages of changes in severity trends (worsening=plus; stability=zero; improvement=minus) between 2 consecutive values of symptom medication score

    Statistical Analysis

    Spearman rank correlation coefficient was used to investigate the relationship between individual averages for each symptom medication score considered. The nonparametric Mann-Whitney U test was applied to compare the average values of quality indexes between 2 groups of subjects divided by their adherence of recording (<80% vs ≥80%). P<.05 was considered statistically significant. Statistical analyses were performed with R Core Team (2018) version 3.5.2 (The R Project for Statistical Computing).


    Study Population and Pollen Season

    This analysis consisted of 101 children (mean age 13.7 [SD 2.8] years) meeting the inclusion criteria for the @IT.2020 pilot study. Male gender was slightly more frequent (63/101, 62.4%), and the population was characterized by predominantly persistent allergic rhinitis symptoms by Allergic Rhinitis and its Impact on Asthma criteria, as assessed by retrospective questionnaire during T0. In addition to persistence, the severity of symptoms for 39.6% (40/101) of the patients was classified as moderate-to-severe (Table 1). The rate of patients with moderate-to-severe persistent symptoms increased to 70.3% (64/101) at the final study visit when being asked the same questionnaire concerning the past pollen season. The most frequent allergic comorbidities were oral allergy syndrome (32/101, 31.7%), atopic dermatitis (28/101, 27.7%), and allergic asthma (28/101, 27.7%). Most patients were sensitized to grass pollen, with 97% (98/101) having a positive skin prick test to Timothy grass and 90.1% (91/101) reacting to Bermuda grass (Table S1 of Multimedia Appendix 1). Grass pollen concentrations ranged from 0 to 199 grains/m3 air. Season criteria of the European Academy of Allergy and Clinical Immunology [31] were adapted to the local setting, and the dates of whole grass pollen season from April 13 to July 28 as well as the peak grass pollen season between May 4 and June 28, 2016 are reported in this study (Figure S1 of Multimedia Appendix 1).

    Table 1. Characteristics of the study population (N=101).
    CharacteristicsValue
    Males, n (%)63 (62.4)
    Age (years), mean (SD)13.7 (2.8)
    Allergic rhinitis

    		Age at onset (years), median (IQR)6 (4-8)

    		Allergic rhinitis and its impact on asthma classification at first medical examination, n (%)


    		Mild intermittent19 (18.8)


    		Mild persistent31 (30.7)


    		Moderate/severe intermittent11 (10.9)


    		Moderate/severe persistent40 (39.6)

    		Allergic rhinitis and its impact on asthma classification at second medical examination, n (%)


    		Mild intermittent6 (6.6)


    		Mild persistent17 (18.7)


    		Moderate/severe intermittent4 (4.4)


    		Moderate/severe persistent64 (70.3)
    Other allergic comorbidities, n (%)

    		Allergic asthma28 (27.7)

    		Oral Allergy Syndrome32 (31.7)

    		Urticaria/angioedema19 (19.2)

    		Atopic dermatitis28 (27.7)

    		Gastrointestinal disorders4 (4)

    		Anaphylaxis episode10 (10.1)

    		Other5 (5.1)

    Adherence to Recording

    During the grass pollen season (May 4 to June 28, 2016), 4003 single reports were collected, equaling an average adherence to recording of 70.8% (4003/5654). Over the period of 56 days, the individual number of filled questionnaires ranged from 11 (20%) to 56 (100%); 65 of the 101 patients (64.4%) were highly adherent to data collection. A delayed reporting start or an anticipated end [27] was observed, with 9 patients starting the monitoring 3 days or more after the start of the prescribed period and 12 patients ending the reporting ≥3 days before the prescribed end. Figure S2 of Multimedia Appendix 1 shows that 53 patients had a prescribed monitoring starting before the grass pollen season.

    Interrelation Among RTSS, CSMS, and VAS

    The RTSS and CSMS correlate well at a population level over time (Figures 2 and 3). Although these 2 scores are calculated based on identical symptom questions with and without the integration of symptomatic medication, the VAS depicted in the bottom panel takes information from a separate question, filled with the same frequency and showing a similar trend over time. At the individual level, the average VAS score correlated well with both—the average symptom score (RTSS) and the average symptom-medication score (CSMS) (Figure 3A and B).

    ‎
    Figure 2. Average population values of Rhinoconjunctivitis Total Symptom Score (0-18 points), Combined Symptom and Medication Score (0-6 points) (both top panel), and Visual Analogue Scale (0-10 points) on impact of allergic symptoms on daily life (bottom panel) over time. CSMS: Combined Symptom and Medication Score; RTSS: Rhinoconjunctivitis Total Symptom Score; VAS: Visual Analogue Scale.
    View this figure
    ‎
    Figure 3. Correlation between individual averages of (A) Rhinoconjunctivitis Total Symptom Score versus Visual Analogue Scale, (B) Combined Symptom and Medication Score versus Visual Analogue Scale, and (C) Combined Symptom and Medication Score versus Rhinoconjunctivitis Total Symptom Score. CSMS: Combined Symptom and Medication Score; RTSS: Rhinoconjunctivitis Total Symptom Score; VAS: Visual Analogue Scale.
    View this figure

    Quality Indices in RTSS, VAS, and CSMS

    In order to assess the quality of recorded data, 4 different parameters were investigated within each of the 3 scores (RTSS, VAS, CSMS) (Table 2), and average values between highly and poorly adherent patients were compared (Table 3, Figure 4).

    The highest diversity of data was observed in the VAS and the RTSS, as expressed by the intravariation index and the coefficient of variation. However, the CSMS was more homogeneous over time. As expected, the percentage of zero values was the lowest in the CSMS, whose average values were almost half of those observed in the RTSS and VAS. By contrast, the percentage of changes in the trend was quite similar and high for all the 3 scores, with values oscillating around 30% (Table 2).

    Table 2. Quality of symptom and symptom-medication scores.
    Quality indexRhinoconjunctivitis Total Symptom Score, mean (95% CI)Visual Analogue Scale, mean (95% CI)Combined Symptom and Medication Score, mean (95% CI)
    Intravariation indexa5.1 (4.5-5.6)6.1 (5.5-6.7)2.9 (2.6-3.2)
    % of zero valuesb38.9 (33.3-44.5)43.3 (37.2-49.3)20.4 (16.0-24.8)
    Coefficient of variationc134.8 (116.5-153.0)138.1 (115.8-160.5)86.3 (72.3-100.3)
    % of changes in trendsd31.3 (28.2-34.4)28.9 (25.6-32.3)32.7 (29.6-35.7)

    aAverage of intravariation index by subjects; for each ith subject, (1) the percentage of the number of variations between 2 consecutive days is calculated, , where t= 1,,,,,,, T is the indicator of time point considered, y is the value of the symptom medication score considered, and I is a binary variable, which is 1 when the condition in the brackets is verified, 0 otherwise; (2) the individual variation range has been calculated (rVAR = (max(y1, …,yT) – min(y1, …, yT)) + 1/S, where S represents the unit increase for each symptom medication score, that is, S=1 for Visual Analogue Scale and Rhinoconjunctivitis Total Symptom Score and S=0.167 for Combined Symptom and Medication Score); and (3) the average of all individual intravariation index values was calculated, , for each symptom score or symptom medication score.

    b100*number of zero compiled values/actual compiled days.

    cAverage of individual coefficient of variation (100*SD/mean).

    dNumber of changes in trends (plus/minus/stable) within ith differences between 2 consecutive values of symptom medication score.

    Table 3. Association between adherence and quality indexes.

    		Rhinoconjunctivitis Total Symptom ScoreVisual Analogue ScaleCombined Symptom and Medication Score

    		Adha<80% (n=36), median (IQR)Adh≥80% (n=65), median (IQR)P valuebAdh<80% (n=36), median (IQR)Adh≥80% (n=65), median (IQR)P valueAdh<80% (n=36), median (IQR)Adh≥80% (n=65), median (IQR)P value
    Intravariation indexc4 (3-6)5 (4-6).085 (4-8)7 (4-8).183 (2-3)3 (2-4).32
    % of zero valuesd27 (12-72)38 (17-55).6622 (16-60)43 (22-65).3113 (0-31)16 (4-28).61
    Coefficient of variatione105 (79-197)99 (79-131).7684 (60-115)115 (76-158).0677 (53-95)74 (48-101).67
    % of changes in trendf25 (11-30)34 (26-51)<.00123 (11-31)33 (16-45).00426 (18-30)36 (28-51)<.001

    aAdh: adherence.

    bMann-Whitney U test was used to compare means among the 2 groups.

    cAverage of intravariation index by subjects; for each ith subject, (1) the number of variations between 2 consecutive days has been calculated, , where t= 1,,,,,,, T is the indicator of the time point considered, y is the value of the symptom medication score considered, and I is a dummy variable, which is 1 when the condition in the brackets is verified, 0 otherwise; (2) the individual variation range has been calculated (rVAR = (max(y1, …,yT) – min(y1, …, yT)) + 1/S, where S represents the unit increase for each symptom medication score, that is, S=1 for Visual Analogue Scale and Rhinoconjunctivitis Total Symptom Score and S=0.167 for Combined Symptom and Medication Score); and (3) the average of all individual intravariation index values was calculated, , for each symptom score or symptom medication score.

    d100*number of zero compiled values/actual compiled days.

    eAverage of individual coefficient of variation (100*SD/mean).

    fNumber of changes in trends (plus/minus/stable) within the differences between 2 consecutive values of symptom medication score.

    ‎
    Figure 4. Changes in trends (positive/negative changes) of recorded (A) Rhinoconjunctivitis Total Symptom Score, (B) Visual Analogue Scale, and (C) Combined Symptom and Medication Score values among patients with adherence to recording of <80% versus ≥80%. ADH: adherence. **P<.01; ***P<.001.
    View this figure

    Relationship of Quality Index Values With Adherence to Recording

    No significant differences were observed between the groups of highly and poorly adherent patients with regard to the average intravariation index, percentage of zero values, and the coefficient of variation. Of note, the coefficient of variation of the VAS score was higher among highly than poorly adherent patients, but the difference was only marginally significant (P=.06). Similarly, the average intravariation index of the RTSS was higher among patients with adherence to recording of 80% or more, but the difference did not reach statistical significance (P=.08). In contrast, the percentage of changes in trend was significantly lower among patients who recorded their symptoms on less than 80% of the prescribed days compared to highly adherent patients, considering each of the 3 indexes: RTSS (P<.001), VAS (P<.005), and CSMS (P<.001) (Table 3, Figure 3).


    We retrospectively analyzed patient-generated data recorded during the grass pollen season by patients with seasonal allergic rhinitis to grasses. The collected data contained information on daily symptoms (RTSS), a combination of symptoms and medication intake (CSMS), and the overall impact of pollen allergy on daily life (VAS). Our analysis shows that (1) RTSS, CSMS, and VAS trajectories correlate well over time at population level; (2) VAS average values correlate well with average values of RTSS and CSMS at individual level; (3) the percentage of days with a change in trend during the observation period is higher in patients with high adherence to recording; and (4) other investigated parameters such as the percentage of zero values, the coefficient of variation, and the intravariation index are not significantly different among highly versus poorly adherent patients. Overall, our results suggest that the percentage of days with a change in trend deserves further investigation in a prospective study as a proxy of data quality in patients monitoring their pollen allergy with an electronic diary app.

    Electronic diaries are increasingly produced and used in medicine, particularly in allergology. Nevertheless, studies focusing on the accuracy and completeness of the patient-generated information collected via electronic diaries are substantially missing. This is of high priority, as data validation is a prerequisite of any scientific or clinical use of the information collected through mobile apps from patients. A recent study demonstrated that daily monitoring with a VAS score has a high intrarater reliability and medium-high validity, reliability, and responsiveness, suggesting the validity of this simple methodology in monitoring disease impact on the patient’s daily life [32]. Along the same line of evidence, our study demonstrates that VAS correlates well with complex measurements such as RTSS and CSMS, both in terms of trajectory at the population level (Figure 1) and as average values at the individual level (Figure 2).

    The percentage of days with changes in trend within the registration period is an interesting parameter, as it can be examined within the context of any trajectory, independently from the structure of the algorithm generating the clinical score. Therefore, this parameter can be applied to VAS, RTSS, CSMS, or any other index that will be generated and validated in the future. We speculate that patients whose personal interest in the recording of their electronic diary is lower, may still be adherent but inaccurate, replicating the same pattern of values every day. The day-to-day variability of pollen counts coupled with daily variability of exposure to pollen as well as the use of preventative medication may impact markedly on day-to-day variability in symptom score. A patient highly adherent to recording is better placed to record the symptom variability and thus more likely to report a higher number of changes in trend. With regard to the use of electronic diaries in clinical practice and research, a tool to predict the quantity and quality of expected data collection would be helpful. Unfortunately, to our knowledge, no such tools exist at the moment. In a previous approach to assess and predict the adherence of patients to symptom recording of patients with pollen allergies, we observed an association between the reporting behavior between the 7th and 21st day of recording compared to the rest of the monitoring period (up to 70 days) [27].

    The lack of a previously established methodology also justifies our explorative approach in the attempt to identify new statistical methods or methods generated in other contexts to address a novel research question. We therefore speculate that future studies will adopt similar approaches and generate new and even more precise methodologies to answer the same research questions. Further, an expansion to other chronic diseases for which digital data collection has been well adopted, for example, asthma, will be of great value.

    We are aware of the limitations of our analysis. First of all, we retrospectively examined in an opportunistic approach a database already generated with different targets. Consequently, our paper can only generate the hypothesis that the percentage of changes in trend is a valuable parameter measuring the quality of patient-recorded data. Unfortunately, other important parameters such as clinical validity could not be investigated within this data set, as this parameter should be investigated independently from the adherence to reporting. Therefore, before any use of changes in trend as a parameter in clinical practice, our hypothesis must be prospectively proven in studies designed with this specific scope. Second, we limited our investigation period to a maximum of 56 days of recording and in the context of a clinical investigation. The generalizability of our conclusions to a real-life setting and to longer periods of monitoring are also to be proven in real-world studies and longer observation periods. Third, our study population was composed of children, whose electronic diary recording is partially (in general until the age of 14 years) performed with the assistance of parents and whose influence on the reliability of data should also be accounted for. Fourth, we have used adherence to recording as a reference parameter under the assumption that patients more compliant in regularly filling their electronic diaries are also those whose data are more reliable. This assumption also should be proven in a new prospective study by adopting external quality standards not affected by recording patterns. Fifth, there is no possibility of correlating medication usage with adherence to digital symptom recording.

    In conclusion, our retrospective analysis identifies the percentage of changes in trend in the trajectory of RTSS, CSMS, and VAS as a parameter, intrinsic to the trajectory itself, thereby representing a valuable candidate as proxy measure of data quality. This hypothesis deserves now to be investigated in prospective studies.

    Acknowledgments

    This study was supported by an unrestricted educational grant from Euroimmun (code 118583). We thank all participants and the entire study team. SA was supported by the European Academy of Allergy and Clinical Immunology Fellowship Award. This study has been supported by an unrestricted grant from Euroimmun. The Informatics Platform AllergyCARD and the AllergyMonitor app have been kindly provided by TPS Software Production.

    Conflicts of Interest

    PMM reports grants and personal fees from TPS Software Production, outside the submitted work. ST is a cofounder of TPS Software Production, which has developed the AllergyMonitor app. S Pelosi is a cofounder of TPS Software Production. All other authors declare no conflicts of interest.

    ‎
    Multimedia Appendix 1

    Supplementary data.

    DOCX File , 266 KB
    1. Nahum JL, Fu MR, Scagliola J, Rodorigo M, Tobik S, Guth A, et al. Real-time electronic patient evaluation of lymphedema symptoms, referral, and satisfaction: a cross-sectional study. Mhealth 2021;7:20 [FREE Full text] [CrossRef] [Medline]
    2. Gomis-Pastor M, Mirabet Perez S, Roig Minguell E, Brossa Loidi V, Lopez Lopez L, Ros Abarca S, et al. Mobile Health to Improve Adherence and Patient Experience in Heart Transplantation Recipients: The mHeart Trial. Healthcare (Basel) 2021 Apr 14;9(4):463 [FREE Full text] [CrossRef] [Medline]
    3. Tripodi S, Giannone A, Sfika I, Pelosi S, Dramburg S, Bianchi A, et al. Digital technologies for an improved management of respiratory allergic diseases: 10 years of clinical studies using an online platform for patients and physicians. Ital J Pediatr 2020 Jul 25;46(1):105 [FREE Full text] [CrossRef] [Medline]
    4. Khusial RJ, Honkoop PJ, Usmani O, Soares M, Simpson A, Biddiscombe M, myAirCoach study group. Effectiveness of myAirCoach: A mHealth Self-Management System in Asthma. J Allergy Clin Immunol Pract 2020 Jun;8(6):1972-1979.e8 [FREE Full text] [CrossRef] [Medline]
    5. Steel PAD, Bodnar D, Bonito M, Torres-Lavoro J, Eid DB, Jacobowitz A, et al. MyEDCare: Evaluation of a Smartphone-Based Emergency Department Discharge Process. Appl Clin Inform 2021 Mar;12(2):362-371. [CrossRef] [Medline]
    6. Richter JG, Nannen C, Chehab G, Acar H, Becker A, Willers R, et al. Mobile App-based documentation of patient-reported outcomes - 3-months results from a proof-of-concept study on modern rheumatology patient management. Arthritis Res Ther 2021 Apr 19;23(1):121 [FREE Full text] [CrossRef] [Medline]
    7. Benson T. Person-specific outcome measure (PSO) for use in primary and community care. BMJ Open Qual 2021 Apr;10(2):1767-1768 [FREE Full text] [CrossRef] [Medline]
    8. Avery P. Using e-health tools and PROMs to support self-management in patients with inflammatory bowel disease. Br J Nurs 2021 Apr 08;30(7):394-402 [FREE Full text] [CrossRef] [Medline]
    9. Austin E, Lee JR, Amtmann D, Bloch R, Lawrence SO, McCall D, et al. Use of patient-generated health data across healthcare settings: implications for health systems. JAMIA Open 2020 Apr;3(1):70-76 [FREE Full text] [CrossRef] [Medline]
    10. Matricardi PM, Dramburg S, Alvarez-Perea A, Antolín-Amérigo D, Apfelbacher C, Atanaskovic-Markovic M, Cavkaytar, et al. The role of mobile health technologies in allergy care: An EAACI position paper. Allergy 2020 Feb;75(2):259-272. [CrossRef] [Medline]
    11. Bousquet J, Ansotegui IJ, Anto JM, Arnavielhe S, Bachert C, Basagaña X, et al. Mobile Technology in Allergic Rhinitis: Evolution in Management or Revolution in Health and Care? J Allergy Clin Immunol Pract 2019;7(8):2511-2523. [CrossRef] [Medline]
    12. Huang J, DeSimone J. Insufficient Effort Responding as a Potential Confound between Survey Measures and Objective Tests. J Bus Psychol 2020 Aug 22;36(5):807-828 [FREE Full text] [CrossRef]
    13. Baer RA, Kroll LS, Rinaldo J, Ballenger J. Detecting and Discriminating Between Random Responding and Overreporting on the MMPI-A. Journal of Personality Assessment 1999 Apr;72(2):308-320. [CrossRef]
    14. Piedmont RL, McCrae RR, Riemann R, Angleitner A. On the invalidity of validity scales: evidence from self-reports and observer ratings in volunteer samples. J Pers Soc Psychol 2000 Mar;78(3):582-593. [CrossRef] [Medline]
    15. Costa P, McCrae R. The revised NEO personality inventory (NEO-PI-R). In: Boyle GJ, Matthews G, Saklofske DH, editors. The SAGE Handbook of Personality Theory and Assessment (Volume 2): Personality Measurement and Testing. California: SAGE Publications Ltd; 2008.
    16. Wise SL, DeMars CE. An Application of Item Response Time: The Effort-Moderated IRT Model. J Educational Measurement 2006 Mar;43(1):19-38. [CrossRef]
    17. Kardas P, Aguilar-Palacio I, Almada M, Cahir C, Costa E, Giardini A, et al. The Need to Develop Standard Measures of Patient Adherence for Big Data: Viewpoint. J Med Internet Res 2020 Aug 27;22(8):e18150 [FREE Full text] [CrossRef] [Medline]
    18. Dima AL, Dediu D. Computation of adherence to medication and visualization of medication histories in R with AdhereR: Towards transparent and reproducible use of electronic healthcare data. PLoS One 2017;12(4):e0174426 [FREE Full text] [CrossRef] [Medline]
    19. Florack J, Brighetti MA, Perna S, Pizzulli A, Pizzulli A, Tripodi S, et al. Comparison of six disease severity scores for allergic rhinitis against pollen counts a prospective analysis at population and individual level. Pediatr Allergy Immunol 2016 Jun;27(4):382-390. [CrossRef] [Medline]
    20. Bastl K, Kmenta M, Pessi A, Prank M, Saarto A, Sofiev M, et al. First comparison of symptom data with allergen content (Bet v 1 and Phl p 5 measurements) and pollen data from four European regions during 2009-2011. Sci Total Environ 2016 Apr 01;548-549:229-235. [CrossRef] [Medline]
    21. Bousquet J, Caimmi DP, Bedbrook A, Bewick M, Hellings PW, Devillier P, et al. Pilot study of mobile phone technology in allergic rhinitis in European countries: the MASK-rhinitis study. Allergy 2017 Jun;72(6):857-865. [CrossRef] [Medline]
    22. Bédard A, Basagaña X, Anto JM, Garcia-Aymerich J, Devillier P, Arnavielhe S, MASK study group. Mobile technology offers novel insights into the control and treatment of allergic rhinitis: The MASK study. J Allergy Clin Immunol 2019 Jul;144(1):135-143.e6. [CrossRef] [Medline]
    23. Bastl K, Bastl M, Bergmann K, Berger M, Berger U. Translating the Burden of Pollen Allergy Into Numbers Using Electronically Generated Symptom Data From the Patient's Hayfever Diary in Austria and Germany: 10-Year Observational Study. J Med Internet Res 2020 Feb 21;22(2):e16767 [FREE Full text] [CrossRef] [Medline]
    24. Caimmi D, Baiz N, Tanno LK, Demoly P, Arnavielhe S, Murray R, MASK Study Group. Validation of the MASK-rhinitis visual analogue scale on smartphone screens to assess allergic rhinitis control. Clin Exp Allergy 2017 Dec;47(12):1526-1533. [CrossRef] [Medline]
    25. Silver JD, Spriggs K, Haberle SG, Katelaris CH, Newbigin EJ, Lampugnani ER. Using crowd-sourced allergic rhinitis symptom data to improve grass pollen forecasts and predict individual symptoms. Sci Total Environ 2020 Jun 10;720:137351. [CrossRef] [Medline]
    26. Arasi S, Castelli S, Di Fraia M, Villalta D, Tripodi S, Perna S, et al. @IT2020: An innovative algorithm for allergen immunotherapy prescription in seasonal allergic rhinitis. Clin Exp Allergy 2021 Jun;51(6):821-828. [CrossRef] [Medline]
    27. Di Fraia M, Tripodi S, Arasi S, Dramburg S, Castelli S, Villalta D, et al. Adherence to Prescribed E-Diary Recording by Patients With Seasonal Allergic Rhinitis: Observational Study. J Med Internet Res 2020 Mar 16;22(3):e16642 [FREE Full text] [CrossRef] [Medline]
    28. Canonica GW, Baena-Cagnani CE, Bousquet J, Bousquet PJ, Lockey RF, Malling H, et al. Recommendations for standardization of clinical trials with Allergen Specific Immunotherapy for respiratory allergy. A statement of a World Allergy Organization (WAO) taskforce. Allergy 2007 Mar;62(3):317-324 [FREE Full text] [CrossRef] [Medline]
    29. Pfaar O, Demoly P, Gerth van Wijk R, Bonini S, Bousquet J, Canonica GW, European Academy of AllergyClinical Immunology. Recommendations for the standardization of clinical outcomes used in allergen immunotherapy trials for allergic rhinoconjunctivitis: an EAACI Position Paper. Allergy 2014 Jul;69(7):854-867. [CrossRef] [Medline]
    30. Bousquet PJ, Combescure C, Neukirch F, Klossek JM, Méchin H, Daures J, et al. Visual analog scales can assess the severity of rhinitis graded according to ARIA guidelines. Allergy 2007 Apr;62(4):367-372. [CrossRef] [Medline]
    31. Pfaar O, Bastl K, Berger U, Buters J, Calderon MA, Clot B, et al. Defining pollen exposure times for clinical trials of allergen immunotherapy for pollen-induced rhinoconjunctivitis - an EAACI position paper. Allergy 2017 May;72(5):713-722. [CrossRef] [Medline]
    32. Sousa-Pinto B, Eklund P, Pfaar O, Klimek L, Zuberbier T, Czarlewski W, et al. Validity, reliability, and responsiveness of daily monitoring visual analog scales in MASK-air®. Clin Transl Allergy 2021 Aug;11(7):e12062 [FREE Full text] [CrossRef] [Medline]

    ‎
    CSMS: Combined Symptom and Medication Score
    RTSS: Rhinoconjunctivitis Total Symptom Score
    VAS: Visual Analogue Scale

    Edited by L Buis; submitted 23.06.21; peer-reviewed by D Antolín-Amérigo, D Ryan; comments to author 08.07.21; revised version received 30.08.21; accepted 07.12.21; published 03.06.22

    Copyright

    ©Stephanie Dramburg, Serena Perna, Marco Di Fraia, Salvatore Tripodi, Stefania Arasi, Sveva Castelli, Danilo Villalta, Francesca Buzzulini, Ifigenia Sfika, Valeria Villella, Ekaterina Potapova, Maria Antonia Brighetti, Alessandro Travaglini, Pierluigi Verardo, Simone Pelosi, Paolo Maria Matricardi. Originally published in JMIR mHealth and uHealth (https://mhealth.jmir.org), 03.06.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on https://mhealth.jmir.org/, as well as this copyright and license information must be included.