Digital and mobile health technologies are creating new perspectives in many areas of research and medical care. One important aspect in both fields is the ability to easily collect patient-generated data via smartphone apps and connected devices such as wearables, diagnostic tools, or environmental sensors [1-5]. Although the use of patient-reported outcomes has become popular over the last decade [6-9], it is not sure how accurately the collected data represent the patient’s state, as recording is done without supervision [10,11]. In particular, daily reporting over a longer time period may be perceived as challenging and cause a certain degree of reporting fatigue. The risk of potentially lower quality owing to disengaged survey respondents has been described with different terms, most recently in the field of psychology as “insufficient effort reporting” [11,12]. However, consensus on methodologies assessing the quantity and quality of entered data is still missing. Proposed methodologies include the (1) response pattern approach [13], (2) response time approach [14], (3) infrequency approach [15], (4) inconsistency approach [16], and (5) the number of unanswered questions. Most of these methodologies, however, refer to single points of data collection and extended questionnaires, which make their application in a setting with daily data recording via smartphone apps difficult to impossible. As digital methods of data collection via openly available mobile apps usually generate very large data sets, new challenges for the analysis and interpretation apply, such as the lack of standard measures [17]. The importance of unified approaches to data recording has recently been underlined in the context of patient adherence, and computational solution approaches were published to support uniform data formats [18]. A representative example for the daily acquisition of patient-generated data are mobile apps related to seasonal allergic rhinitis [19,20]. A variety of apps has been published for patients with pollen allergies, providing exposure forecasts, individualized symptom prediction, symptom and medication diaries, and in some cases, the possibility of exchanging recorded data with the attending physician [3,10,21-23]. Although several studies have shown the potential of mobile technologies for research purposes and clinical disease management, only few address the topic of data quality and validation [24,25]. The purpose of this study is to retrospectively investigate 4 putative validation criteria to assess the quality of data longitudinally collected by patients with seasonal allergic rhinitis and defined as follows: (1) intravariation index, (2) percentage of zero values, (3) coefficient of variation, and (4) percentage of changes in trends. To this end, we have taken advantage of the patient-generated symptom and medication data set, which has been acquired via a mobile app in a cohort of patients with seasonal allergic rhinitis in the context of the @IT.2020 pilot project [26,27].

We retrospectively analyzed patient-generated data recorded during the grass pollen season by patients with seasonal allergic rhinitis to grasses. The collected data contained information on daily symptoms (RTSS), a combination of symptoms and medication intake (CSMS), and the overall impact of pollen allergy on daily life (VAS). Our analysis shows that (1) RTSS, CSMS, and VAS trajectories correlate well over time at population level; (2) VAS average values correlate well with average values of RTSS and CSMS at individual level; (3) the percentage of days with a change in trend during the observation period is higher in patients with high adherence to recording; and (4) other investigated parameters such as the percentage of zero values, the coefficient of variation, and the intravariation index are not significantly different among highly versus poorly adherent patients. Overall, our results suggest that the percentage of days with a change in trend deserves further investigation in a prospective study as a proxy of data quality in patients monitoring their pollen allergy with an electronic diary app.

Electronic diaries are increasingly produced and used in medicine, particularly in allergology. Nevertheless, studies focusing on the accuracy and completeness of the patient-generated information collected via electronic diaries are substantially missing. This is of high priority, as data validation is a prerequisite of any scientific or clinical use of the information collected through mobile apps from patients. A recent study demonstrated that daily monitoring with a VAS score has a high intrarater reliability and medium-high validity, reliability, and responsiveness, suggesting the validity of this simple methodology in monitoring disease impact on the patient’s daily life [32]. Along the same line of evidence, our study demonstrates that VAS correlates well with complex measurements such as RTSS and CSMS, both in terms of trajectory at the population level (Figure 1) and as average values at the individual level (Figure 2).

The percentage of days with changes in trend within the registration period is an interesting parameter, as it can be examined within the context of any trajectory, independently from the structure of the algorithm generating the clinical score. Therefore, this parameter can be applied to VAS, RTSS, CSMS, or any other index that will be generated and validated in the future. We speculate that patients whose personal interest in the recording of their electronic diary is lower, may still be adherent but inaccurate, replicating the same pattern of values every day. The day-to-day variability of pollen counts coupled with daily variability of exposure to pollen as well as the use of preventative medication may impact markedly on day-to-day variability in symptom score. A patient highly adherent to recording is better placed to record the symptom variability and thus more likely to report a higher number of changes in trend. With regard to the use of electronic diaries in clinical practice and research, a tool to predict the quantity and quality of expected data collection would be helpful. Unfortunately, to our knowledge, no such tools exist at the moment. In a previous approach to assess and predict the adherence of patients to symptom recording of patients with pollen allergies, we observed an association between the reporting behavior between the 7th and 21st day of recording compared to the rest of the monitoring period (up to 70 days) [27].

The lack of a previously established methodology also justifies our explorative approach in the attempt to identify new statistical methods or methods generated in other contexts to address a novel research question. We therefore speculate that future studies will adopt similar approaches and generate new and even more precise methodologies to answer the same research questions. Further, an expansion to other chronic diseases for which digital data collection has been well adopted, for example, asthma, will be of great value.

We are aware of the limitations of our analysis. First of all, we retrospectively examined in an opportunistic approach a database already generated with different targets. Consequently, our paper can only generate the hypothesis that the percentage of changes in trend is a valuable parameter measuring the quality of patient-recorded data. Unfortunately, other important parameters such as clinical validity could not be investigated within this data set, as this parameter should be investigated independently from the adherence to reporting. Therefore, before any use of changes in trend as a parameter in clinical practice, our hypothesis must be prospectively proven in studies designed with this specific scope. Second, we limited our investigation period to a maximum of 56 days of recording and in the context of a clinical investigation. The generalizability of our conclusions to a real-life setting and to longer periods of monitoring are also to be proven in real-world studies and longer observation periods. Third, our study population was composed of children, whose electronic diary recording is partially (in general until the age of 14 years) performed with the assistance of parents and whose influence on the reliability of data should also be accounted for. Fourth, we have used adherence to recording as a reference parameter under the assumption that patients more compliant in regularly filling their electronic diaries are also those whose data are more reliable. This assumption also should be proven in a new prospective study by adopting external quality standards not affected by recording patterns. Fifth, there is no possibility of correlating medication usage with adherence to digital symptom recording.

In conclusion, our retrospective analysis identifies the percentage of changes in trend in the trajectory of RTSS, CSMS, and VAS as a parameter, intrinsic to the trajectory itself, thereby representing a valuable candidate as proxy measure of data quality. This hypothesis deserves now to be investigated in prospective studies.