We recruited participants for our prospective longitudinal study from staff of 3 different employers in Germany during fall 2018: Investitionsbank Berlin, PEI, and University Hospital Frankfurt. We included volunteers on 2 respective days. To be eligible, individuals had to be employed at these institutions, vaccinated against influenza by the occupational health physician on these days, own a smartphone, be at least 18 years old, and be proficient in the German language.

We gave participants who agreed to take part in our study an information leaflet comprising information about the study, data transmission, and data storage; a random ID; and instructions on downloading and using the SafeVac app. Once the individuals agreed to participate and signed the declaration of consent, we asked participants to download and log in to SafeVac immediately after receiving the influenza vaccination. In the app, participants provided sociodemographic information and their vaccination history and then entered information about occurrence or nonoccurrence of adverse events. After participants used the app, we distributed usability questionnaires based on the System Usability Scale (SUS) and evaluated them using the adjective rating scale [17,18]. Additionally, participants could take part in a lottery to win a tablet or smartwatch. This was offered to all participants who completed the AEFI reports for at least 1 week.

The study protocol and data protection concept were reviewed and assessed without any concerns by the ethical committee of the Medical Association of Lower Saxony in Germany, the HZI institutional data protection officer, and the Federal Commissioner for Data Protection and Freedom of Information.

In SafeVac, we asked participants to enter information on receipt of influenza and previous vaccinations and sociodemographic variables and describe any AEFI occurrence. We asked these questions at 15 intervals defined over a period of 3 months: 1 hour, 4 hours, and 8 hours after vaccination; daily until the 7th day after vaccination; weekly for 3 weeks after vaccination; and monthly for 2 subsequent months after vaccination. In case of AEFI occurrence, participants answered additional questions in the app based on requirements of the online reporting platform for adverse events hosted by the national responsible authorities PEI and the Federal Institute for Drugs and Medical Devices. Participants could select a specific AEFI (eg, fatigue) from a drop-down list or enter it manually in the comment text field. In addition, participants were asked to provide additional information about chronic medical conditions, medications taken for chronic conditions, and pregnancy status.

SafeVac was available in Android (version 4.4 or higher) and iOS (10 or higher) app stores. For the development and design of SafeVac, we took a user-centered approach for which we gathered data in a previous study on users’ app preferences [19]. Based on these results, we used gamification elements such as the appearance of puzzle pieces and a loading bar to enhance app use adherence (Figure 1). Every app entry was transmitted anonymously and securely to PEI using https/SSL encryption. To evaluate the feasibility of the app, we used the adherence rate as a surrogate parameter and assessed whether AEFIs entered into the app were in accordance with the literature.

We included all information collected through SafeVac from fall 2018 until March 14, 2019, in our study. Before analyzing the data, we excluded test IDs. We coded AEFI entries in the free-text fields according to the preferred terms in the Medical Dictionary of Regulatory Activities and validated them using the four-eyes principle. We also checked for consistency in cases where participants had entered AEFIs manually in the comment text field and additionally selected them from the drop-down list at the same time point. In case of discrepancy, we coded them as 2 different types of AEFIs. If participants entered 2 identical AEFIs for the same time point, we removed one. If the free-text field described the AEFI in more detail than the AEFI selected on the drop-down list, we ignored the selected AEFI and coded the manually entered AEFI systematically according to the corresponding preferred term (eg, if “pain in extremities” was selected in the drop-down list and “pain in the arm (injection site)” was entered manually, we coded the AEFI as “injection site pain”). The BMI was calculated and categorized using the World Health Organization scale approach [20].

By using logistic regression, we estimated the relationship between sociodemographic variables and vaccination uptakes with the binary outcomes app adherence until the end of the study, AEFI occurrence and correct entry of vaccination information, respectively. We defined incorrect entry of vaccination information as any misspellings in the name of the received influenza vaccination or its associated batch number but did not take case sensitivity into account. To find determinants for the outcome AEFI occurrence in 3 months, we used a Cox regression model. Variables were selected by using a backward selection by Akaike information criterion. In all models we set age and sex as a priori confounders. Missing data were not included in the model.

Additionally, we used Cox regression to explore determinants of reporting an influenza-like illness as an AEFI. We used R software version 3.2.5 (R Foundation for Statistical Computing) for data analysis and visualization.

To assess quality and usability of the app, we developed a feedback questionnaire in LimeSurvey, an online survey tool, by including the SUS and using an approach of the User Version of the Mobile Application Rating Scale [21,22]. We distributed the feedback questionnaire through a link 1 year after the start of the study via the internal websites of the 3 participating institutions.

Of the participants who provided informed consent, 72.9% (337/462) logged into the SafeVac (study population) and made a minimum of one app entry; 61.4% (207/337) used the app until the end of the study (Figure 2).

The majority of participants were female (224/337, 66.5%) and 84.0% (283/337) had a general certificate of education (Abitur). Almost half of the participants (166/337, 49.3%) stated that they had also been vaccinated against influenza in an earlier year. We assessed characteristics according to adherence of app use in the study (fully vs partially adherent participants), and they appear mostly similar (eg, 74/207 [35.7%] vs 39/130 [30.0%] male participants; Table 1).

Of the participants who logged in, 81.4% (271/333) reported experiencing one or more AEFIs after vaccination. We found a rise in reported AEFIs at 4 hours after vaccination (156/333, 47.8%); thereafter, the number of participants reporting an AEFI remained steady over 8 hours (160/333, 48.0%) and 1 day (149/333, 44.7%) after vaccination. A decline started from 2 days after vaccination (77/333, 23.1%) onward. Study participant reporting attrition increased slightly over time, with the highest attrition difference (78) between 56 days and 84 days after vaccination (Figure 3).

Regarding adherence to SafeVac use until the end of follow-up, our logistic regression revealed a decrease of 43% (odds ratio [OR] 0.47; CI 0.25-0.91) for females and staff of the banking institution. The latter population was significantly less adherent than staff of the university hospital (OR 0.40; CI 0.17-0.94) after adjusting for age, sex, education, influenza vaccination received in the last year, number of influenza vaccinations in previous 5 years, occurrences of AEFIs, and institutional affiliation (Figure 4).

Overall, 260 participants correctly entered the data for their received vaccination. Having no Abitur (OR 0.31; CI 0.13-0.76) and increasing age (OR 0.96; CI 0.93-0.99) were negatively associated with correctness of entry of vaccine information (Figure 5). Other factors we analyzed (eg, age, gender, vaccination received last year, and affiliated institution) were not significantly associated with correct app entry of vaccination.

The most mentioned AEFIs from all participants were injection site pain (106/337, 31.5%), followed by pain in extremity (103/337, 30.6%) and fatigue/asthenia (73/337, 21.7%). Proportional differences in reporting were found in fully adherent and partially adherent participants (eg, pain in extremity, 60/207 [29.0%], 43/130 [33.1%]). However, there were no statistically significant differences in AEFI reporting between fully adherent and partially adherent app users (Table 2).

For the outcome reported AEFI occurrence, the results of the Cox regression indicated a negative association with increasing age (hazard ratio 0.98; CI 0.97-0.99) and a positive association with female individuals (hazard ratio 1.41; CI 1.01-1.96; Figure 6).

The hazard ratio for reporting an influenza-like illness at any time point within 3 months was 0.26 (CI 0.11-0.60) for persons who were vaccinated against influenza in the last year compared with those who were not vaccinated in the last year (Multimedia Appendix 1).

There were 6 users who responded to the usability questionnaire. The mean SUS score built out of 6 feedback questionnaire responders was 86.67 with no negative rating of the entertainment of the app, quality of information, or overall app suitability for reporting AEFIs.

Our study shows the feasibility of an app-based reporting of AEFIs and suggests an added value of a mobile app with near real-time features to report AEFIs for a period of 3 months. We found that adherence to app use was dependent on gender but independent of age and AEFI occurrence. On the other hand, determinants of a correct app entry were increasing age and higher education. The most observed AEFI types reported in our study were injection site pain, pain in extremity, and fatigue/asthenia.

In previous studies, several methods for active adverse events reporting have been tested, including diary cards and telephone interviews and a mobile app [15,23]. Compared with a study using diary cards and telephone interviews [23], the dropout rate in our study was lower. The attrition rate in our study is consistent with the one reported by Wilson et al [15], who also used a mobile app as a reporting system for reporting AEFIs. The high dropout rate in the study using telephone interviews and diary cards could be related to a lack of active reminders. In our research and in Wilson et al [15], reminder notifications were implemented in the apps. This is of relevance especially for newly introduced vaccines for which the reporting period immediately after vaccination is often a crucial interval. For a 1-week interval, for example, our app showed an attrition rate of less than 2%. Common determinants of attrition in longitudinal studies are sex and age, with males and younger age being more likely to discontinue their participation [24,25]. However, our results showed that female sex was associated with attrition, whereas age had no influence. In our project, participants adapted to a new technology in order to stay adherent in a longitudinal study. The known factors associated with the nonadaption of new technologies are female sex and older age [26,27]. Therefore, it is not surprising that our results showed different determinants for attrition than in common longitudinal studies.

Vaccine-related information, such as the batch number, is often incorrectly entered when done manually [28]. Accordingly, one-third of our study participants reported they struggled with entering those details into the SafeVac app. For future studies or routine recording, the use of a mobile phone camera as barcode scanner could be an alternative to capture such information on vaccinations [29-31].

Older age of vaccinees, child age, and female sex are known triggers for AEFIs [32]. These particular age groups were not included in our study. This could explain why we found a decrease in reported AEFIs with increasing age. We cannot rule out to what extent AEFI occurrence is due to actual occurrence or reporting behavior. However, this influences any kind of voluntary reporting.

In our study, we also asked participants to indicate previous influenza vaccinations. The uptake of the most recent previous vaccination was related to low reporting of an influenza-like illness as an AEFI within 3 months, whereas the overall number of influenza vaccination uptakes within the last 5 years seemed to have no effect.

The most observed types of AEFIs reported in our study (ie, injection site pain, pain in extremity, and fatigue/asthenia) correspond with other safety studies of influenza vaccinations [33]. One clinical trial, however, reported injection site pain, headaches, and myalgia more often (8% to 23% more often) than in our study [34]. The reasons for these differences are unclear. However, one possible reason could be that our study population might not be as healthy as the study population selected for the clinical trial.

When analyzing the kind of AEFI occurrence in relation to adherence level of app users/study participants (ie, fully and partially adherent users), we did not find a difference. Therefore, we conclude that no specific AEFI type was responsible for attrition. Furthermore, most of the attrition was toward the end of the study, leading to a similar likelihood of AEFI type occurrence in both groups.

The main strengths of our feasibility study are its multicentric study nature, long observation time, and use of a custom designed app for reporting AEFIs. However, the study did not aim to provide generalizable results. In our study, we could demonstrate that individuals are willing to report AEFIs for more than the usually implemented follow-up period of 2 weeks to 1 month for active influenza AEFI reporting [14,15]. With the help of additional studies, this could mean that a broader spectrum of AEFIs (ie, unexpected late-onset AEFIs) could be captured with an app. In addition, by using a longitudinal approach, we were able to generate determinants of adherence to app use and information on AEFIs occurring after an annual influenza vaccination. With another study design (eg, use of app in routine vaccination settings and for different vaccines), a more diverse and increased participant group (eg, elderly) could be captured. This would help to assess the causality of AEFIs, their frequencies, seriousness, and course. A general limitation in our findings is the fact that we cannot disentangle how and which determinants influence the reporting and factual occurrence of AEFIs.

Given that the mean SUS score of all respondents was more than 85/100, the app’s usability seemed to be excellent according to the adjective scale rating [18]. However, we distributed the questionnaire to assess the usability and quality of the mobile app 1 year after recruitment due to difficulties in separating the feedback questionnaire from the national databank of PEI. The time delay between vaccination and questionnaire distribution could have led to a low response rate. In our study, we recruited participants face to face and used participation in a lottery as an incentive measure. Therefore, it is unclear if the same response and adherence rate can be expected outside of the study. Nevertheless, in order for participants to qualify for lottery participation, AEFI reports were required until 1 week after vaccination. As we do not see any immediate drop after that time point, we would exclude the incentive as the main reason behind the adherence to the app until the end of the study.

Additionally, our reported adverse event must be interpreted as such and not as an adverse reaction, meaning without any causality assessment. For that purpose, a comparison group with no vaccine would have been required.

We have shown that the use of a mobile app to report AEFIs for 3 months was feasible for more than 60% of participants, and the most reported adverse event after influenza vaccination were similar to those reported in clinical trials. Future studies could use the SafeVac concept for enhanced AEFI reporting, especially by including broader target groups (eg, elderly people and children) and by implementing the app in routine settings for various vaccines (eg, in general practitioners’ offices). In addition, new vaccines like the one against COVID-19 can benefit from using an approach like SafeVac for safety reporting. In fact, SafeVac 2.0 was recently adapted to the COVID-19 vaccination and is used as the national reporting tool for AEFIs in an active surveillance study in Germany [35].