Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 17.10.19 in Vol 7, No 10 (2019): October

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/14276, first published Apr 06, 2019.

This paper is in the following e-collection/theme issue:

    Original Paper

    The Fever Coach Mobile App for Participatory Influenza Surveillance in Children: Usability Study

    1Mobile Doctor, Co, Ltd, Seoul, Republic of Korea

    2Institute of Basic Medical Sciences, Graduate School of Medicine, CHA University, Seongnam-si, Gyeonggi-do, Republic of Korea

    3Department of Biomedical Informatics, Graduate School of Medicine, CHA University, Seongnam-si, Gyeonggi-do, Republic of Korea

    4Healthcare Bigdata Center, Bundang CHA General Hospital, Seongnam-si, Republic of Korea

    *these authors contributed equally

    Corresponding Author:

    Hyun Wook Han, MD, PhD

    Department of Biomedical Informatics

    Graduate School of Medicine, CHA University

    Pangyo-ro 335, Bundang-gu

    Seongnam-si, Gyeonggi-do, 13488

    Republic of Korea

    Phone: 82 31 881 7109

    Email: stepano7@gmail.com


    ABSTRACT

    Background: Effective surveillance of influenza requires a broad network of health care providers actively reporting cases of influenza-like illnesses and positive laboratory results. Not only is this traditional surveillance system costly to establish and maintain but there is also a time lag between a change in influenza activity and its detection. A new surveillance system that is both reliable and timely will help public health officials to effectively control an epidemic and mitigate the burden of the disease.

    Objective: This study aimed to evaluate the use of parent-reported data of febrile illnesses in children submitted through the Fever Coach app in real-time surveillance of influenza activities.

    Methods: Fever Coach is a mobile app designed to help parents and caregivers manage fever in young children, currently mainly serviced in South Korea. The app analyzes data entered by a caregiver and provides tailored information for care of the child based on the child’s age, sex, body weight, body temperature, and accompanying symptoms. Using the data submitted to the app during the 2016-2017 influenza season, we built a regression model that monitors influenza incidence for the 2017-2018 season and validated the model by comparing the predictions with the public influenza surveillance data from the Korea Centers for Disease Control and Prevention (KCDC).

    Results: During the 2-year study period, 70,203 diagnosis data, including 7702 influenza reports, were submitted. There was a significant correlation between the influenza activity predicted by Fever Coach and that reported by KCDC (Spearman ρ=0.878; P<.001). Using this model, the influenza epidemic in the 2017-2018 season was detected 10 days before the epidemic alert announced by KCDC.

    Conclusions: The Fever Coach app successfully collected data from 7.73% (207,699/2,686,580) of the target population by providing care instruction for febrile children. These data were used to develop a model that accurately estimated influenza activity measured by the central government agency using reports from sentinel facilities in the national surveillance network.

    JMIR Mhealth Uhealth 2019;7(10):e14276

    doi:10.2196/14276

    KEYWORDS



    Introduction

    Seasonal influenza affects 5% to 15% of the world population and is estimated to be accountable for 3 to 5 million cases of severe illness and up to 500,000 deaths annually [1]. Children younger than 5 years, adults aged 65 years or older, and pregnant women are at high risk of developing serious influenza-related complications [2]. In 2008, it was estimated that 90 million new cases, 1 to 2 million cases of severe illness, and 28,000 to 111,500 deaths that were attributable to influenza infection occurred in this population [3].

    Monitoring influenza activities is essential to understand the epidemiology, implement appropriate prevention strategies, and adequately allocate public health resources to mitigate the burden of epidemics. Currently, most developed countries have surveillance systems based on the networks of sentinel facilities and primary care practitioners reporting the weekly number of patients visited with influenza-like illness (ILI) or acute respiratory infection. Many countries, including the United States and South Korea, also collect data of virologic information, hospitalization, and mortality. At a global level, the World Health Organization Global Influenza Surveillance and Response System, comprising 143 National Influenza Centers in 111 countries, has provided virologic information used to select strains for influenza vaccine formulations [4,5].

    Although the information provided by influenza surveillance is invaluable, a traditional surveillance system requires substantial amount of resources, including trained staff, laboratory capacities, and information infrastructure. Establishing a sustainable surveillance system in countries with limited resources depends on financial and technical support from external sources as well as active involvement of the national health authority [6]. In addition, although timeliness is a crucial element of infection surveillance, the traditional system is associated with time lags as it takes several days from event to reports, from reports to analysis, and from analysis to dissemination [7].

    To overcome the challenges of traditional surveillance systems, novel approaches to influenza surveillance using the internet and mobile technologies have been developed. The use of big data drawn from Web search queries, social media, and electronic health records has attracted much attention from public health researchers and policy makers. Among these, Google Flu Trends (GFT) [8] was one of the key drivers of the hype around big data in public health. Despite its constant overestimations of flu prevalence that led to the termination of the service, GFT shed light on the use of internet big data in disease surveillance and nowcasting [9,10]. A number of researchers have also evaluated search queries and social media as influenza surveillance tools [11-15]. However, the big data approach faces serious challenges including big data hubris, privacy concerns, and a lack of accurate algorithmic models [9,10].

    Another recently emerged internet-based surveillance method is participatory surveillance [16]. Participatory disease surveillance systems ask the target population to submit disease-related data through various forms of survey tools. Although it is a potentially valuable data source, participatory surveillance has low specificity because many such systems collect subjective symptoms of illness without information on clinical diagnosis [16,17]. Furthermore, as these systems completely rely on volunteers in communities, participation is low and difficult to maintain over time [18].

    In this paper, we describe a new approach that collects participatory data, including symptoms and clinical diagnosis, of febrile diseases and encourages participation by providing fever care instructions. In addition to analyzing individual data and delivering customized advice, the vast amount of aggregate information could generate a population-level data. We examined if such data could be used for influenza surveillance while addressing the aforementioned challenges of participatory disease surveillance systems.


    Methods

    Exploratory Data Analysis

    Participatory Data: Fever Coach

    All self-reported data were obtained from Fever Coach. Fever Coach ( in Korean) is a mobile app created to provide actionable, tailored information for fever management in children younger than 5 years. Upon registration for the service, a user is asked to enter the child’s date of birth, body weight, sex, and history of febrile seizure. A single user can manage multiple accounts if they have more than 1 child. Figure 1 shows screenshots of the app. After registration, the user may enter various fever-related information of the child including body temperature, other symptoms, dose and time of antipyretic administration, vaccination and antibiotic history, and physician-made diagnosis if the child was seen by a physician for the management of current illness. On the basis of these inputs, the app provides instructions for fever control and advises on the timing of seeking medical attention. If a physician-made diagnosis is submitted, detailed information about the disease is also provided. Users can also choose to not submit any data and to only read fever care instructions. The service was designed and reviewed by 2 board-certified family physicians and 1 board-certified pediatrician. Fever Coach was launched in July 2015 in South Korea. The app is currently available for free on Google Play and Apple App Store in South Korea, Japan, and China. As of June 2018, there have been 400,000 downloads, 99% of which were in South Korea.

    Figure 1. Screenshots of Fever Coach app. (a) body temperature page, (b) fever management page, (c) diagnosis page. The service is not provided in English and is translated from Korean for these screenshots. (options list: bronchitis, bronchiolitis, pneumonia, adenovirus infection, laryngitis, pharyngitis/tonsillitis, common cold, influenza, canker sore, herpangina, hand-foot-mouth, enterocolitis, meningitis, otitis media, urinary tract infection, mumps, scarlet fever, chicken pox, measles, roseola, not diagnosed, and others).
    View this figure

    The study data were collected retrospectively only from users who agreed to the use of deidentified information for research purposes from September 2016 to August 2018. The data submitted between September 2016 and August 2017 (development period, 52 weeks) were used to develop a prediction model, and the data submitted from September 2017 to August 2018 (validation period, 52 weeks) were used to validate the model.

    All user-reported data were stored in Fever Coach databases in real time, and metadata (number of submissions and location information) on each day were processed and saved on a daily basis. For comparison with the available reference population, only the data from children aged 1 year (365 days) to 6 years (2554 days) submitted from South Korea were included for this study. To filter out falsely submitted data, we only used body weight values from 5 percentile of newborns (2.5 kg) to 95 percentile of 18 year olds (79.8 kg) based on the Korean child growth curve in 2017 [19]. As the age of a child changed over time, we calculated the mean of the age during each period if data were submitted multiple times. For body weight, we only considered the last submitted data for each child. These data were used to understand user demographics, clinical characteristics, and fever-related behavior. The study protocol was approved by the institutional review board (IRB) of CHA university (IRB number 1044308-201804-HR-022-03).

    The total app user activity was measured by the number of body temperature submissions to the app in each week. As Fever Coach is typically used only when a child is ill, the authors decided that it was more appropriate to measure user activity by the number of data submissions than by daily or weekly active users. The proportion of influenza diagnoses to the total number of any diagnoses submitted to the app was used as the indicator of influenza activity.

    Reference Data: Korea Centers for Disease Control and Prevention Surveillance System

    The KCDC ILI surveillance system provides weekly updates on influenza surveillance data as the proportion of the ILI-related visits to the total outpatient visits [20]. An ILI is defined as a fever equal to or greater than 38.0°C (100.4°F) accompanied by cough or sore throat. The surveillance data are collected from the Korea Influenza Surveillance Scheme that has 36 sentinel primary care facilities. KCDC also reports the proportion of the ILI-related visits of children aged 1 year through 6 years to the total outpatient visits of the same age range in a week, as well as in the entire age range.

    Prediction of Influenza Activities

    Prediction Model

    A linear regression model was developed to predict influenza activity in children from the Fever Coach data:

    P (t)=β0 + β1F (t) + ε

    P (t) is the proportion of the ILI-related visits of children aged 1 year through 6 years to the total outpatient visits of the same-aged population reported by KCDC, F (t) and is the proportion of influenza diagnosis reports to all diagnosis reports through the Fever Coach app. Both P (t) and F (t) include data accrued in 7 days that precedes date t Intercept β0 and slope β1 were obtained from regression, and ε is an error term. We also developed another linear regression model on F (t) to predict the proportion of ILI visits in all age ranges to detect the influenza epidemic alert, which is based on the ILI visits in the total population. The threshold of KCDC for an influenza epidemic alert in the 2017 flu season was set at 6.6 ILI visits per 1000 visits, determined as the average value of ILI plus 2 SD during the nonflu seasons in the past 3 years (2014-2016).

    Statistical Analysis

    To compare user characteristics and behavior between the development period and the validation period, the Pearson chi-square test was used for categorical values, the Student t test was used for continuous values with normal distribution, and the Mann-Whitney U test was used for continuous values with non-normal distribution. The Shapiro-Wilk test was used for determining the normality of data distribution.

    The accuracy of the prediction model was evaluated using the Spearman's rank correlation coefficient (Spearman ρ) and root mean square error (RMSE) for comparison between P (t) and their predicted values. All statistical analyses were performed using Scikit-learn 0.19.1 [21].


    Results

    Exploratory Data Analysis

    During the entire study period, 10,002,512 fever-related health data were collected. The number of total records decreased by 18.5%, whereas the number of children whose data were submitted decreased by 44.2% from the development period to the validation period (Table 1).

    Table 1. User characteristics and number of datapoints submitted during the development and validation periods.
    View this table

    During the development period, 149,329 people used the service at least once, and 115,674 (77.5%) of them used it for more than 2 separate days during the 1-year development period. From the 149,329 users, fever-related information was recorded from 209,270 children, and 49.2% (103,083) of them were female. This was 7.7% of the total population in that age group in South Korea [22]. They submitted 41,578 diagnoses, including 2965 influenza reports. The minimum and maximum values of F (t) were 0 and 0.495, respectively, and the range of P (t) was from 0.0037 to 0.0862 in the same period. The influenza activity during the study period is visualized geographically on a web page titled Fever Coach Flu Visualization [23]. The video record of this visualization can be found in Multimedia Appendix 1.

    During the validation period, fever-related information of 116,804 children was recorded, and 26,848 diagnoses, including 5800 influenza reports, were collected. F (t) and P (t) ranged between 0 to 0.495 and 0.0023 to 0.0928, respectively. We classified all diagnosis records into 3 categories: influenza, pharyngitis/common cold, and herpangina/stomatitis. The change of diagnosis and app user activity over time is shown in Figure 2.

    Figure 2. Change of diagnosis submissions and app user activity during the study period. Y-axis shows the number of submissions in each week. Fever records (left y-axis) was measured by the number of body temperature submissions to the app and is shown as a pink line. Number of each diagnosis categories (right y-axis) is shown with different colors. Pharyngitis or common cold (green line) shows the sum of pharyngitis ortonsillitis and common cold, and herpangina or stomatitis (orange line) shows the sum of canker sore and herpangina.
    View this figure

    Prediction of Influenza Activities

    The prediction model derived from the linear regression between ILI visits [P (t)] for children aged 1 year through 6 years and influenza diagnosis proportion submitted to Fever Coach [F (t)] came out with its slope (β1) as 0.1316, intercept (β0) as 0.0063, and R2 as 0.930 (Figure 3). P (t) is the reference value from KCDC. Spearman ρ was 0.895 (P<.001) and RMSE was 0.0083 between P (t) and predicted value in the validation period (the maximum value of P (t) was 0.0928 and minimum value was 0.0023). Figure 4 shows a graphical plot of P (t) and predicted values over the study period as well as the error of the prediction model. From the all-age model that was developed between F (t) and proportion of ILI visits in all age ranges, the 𝜌 was 0.924 (P<.001) and the RMSE was 0.0135. The predicted value exceeded the epidemic threshold on November 21, 2017, by this model. KCDC announced the influenza epidemic alert on December 1, 2017.

    Figure 3. A log-scaled plot. Linear regression between F(t) and influenza-like illness (ILI) visit proportion of children younger than 7 years during the development period. Black dots show actual values, and red line shows the result of linear regression.
    View this figure
    Figure 4. A comparison between Korea Centers for Disease Control and Prevention (KCDC) influenza-like illness (ILI) data and Fever Coach data: blue line shows ILI visits per 1000 outpatient visits reported by KCDC, red line shows ILI prediction from Fever Coach data, and pink line shows the total number of fever-related records in a week. Errors between P (t) and predicted values were displayed at the bottom plot as a black line. The development period was illustrated with a white background color, and green-colored background was used for the validation period.
    View this figure

    Discussion

    Principal Findings

    The Fever Coach app successfully collected data from 7.7% of the target population over a 1-year period, which is remarkably higher than participation rates of previously reported Web- or mobile-based participatory surveillance systems that reported participation rates between 0.02% [24] and 0.13% [25]. The influenza activity monitoring model based on voluntary data submitted through the app showed a close association with the large-scale government-reported influenza activity. The participatory data are updated daily, rather than weekly, and the prediction model based on these data led to more sensitive detection of an influenza epidemic compared with the epidemic alert from the KCDC.

    The result of this study sheds light on the potential of mobile apps in collecting data for public health surveillance and highlights the importance of user benefits for voluntary data submission. Previous participatory systems for influenza surveillance relied on the social responsibility of individuals for recruiting and retention of participants, which resulted in low participation rate and biased representation of target populations.

    Comparison With Previous Work—Recruitment and Retention

    There are several other Web- or mobile phone-based survey platforms for participatory surveillance of influenza and other infectious diseases [17,26,24,27]. Some only collect information regarding symptoms [24] and health-related behavior, whereas others also ask outcomes of medical services [17]. These platforms rely on volunteers by advertising the benefits of self-reported data in disease surveillance and do not incentivize individual participants. Therefore, recruitment and retention of users have been identified as the main challenges for them [24]. They used social media, national television, institutional networks, professional societies, dissemination events, and word of mouth to recruit participants. Using this strategy, Flu Near You (FNY) in the United States recruited 61,000 participants during its second influenza season (2012-2013), which was about 24 per 100,000 population of age 13 years and older. Influenzanet, which was established in 5 European countries in 2009, reported 36,192 participants in 10 countries in Europe during the 2015-2016 influenza season, with a participation rate of 13 individuals per 100,000 population.

    In South Korea, online communities are very active and Koreans seek information and communicate with their peers about studying, career, hobbies, and parenting through such communities as well as blogs [28]. Fever Coach used these communities to promote the app by providing free coffee coupons for randomly selected users who downloaded the app and shared their experience and by providing free online health consultation. After reaching 100,000 downloads in May 2016, there have not been active marketing activities, but parents who used the app have voluntarily spread the word on the internet and in their local communities. In February 2017, Fever Coach had been downloaded 300,000 times and over 200,000 users submitted their data at least once. Using this strategy, the participation rate was about 7700 per 100,000 population aged 1 year to 6 years, one of the high-risk groups for influenza infection. The Fever Coach app was primarily designed to enable parents to take care of their kids when they have any kind of fever, but it was able to collect data that can be used specifically for influenza surveillance. We believe that by providing immediate benefit and targeting the vulnerable population, recruiting and retention of participatory surveillance can be much more effective than by relying on volunteers. This strategy can be applied to target other high-risk groups such as pregnant women, elderly people, or immunocompromised individuals.

    Comparison With Previous Work—User Characteristics

    As users of Fever Coach submit data on behalf of their children, the only information we collected about the users themselves was email address and device location. However, as the service is designed for caregivers of children younger than 5 years, we can assume that the users’ age range is approximately between late 20s and early 40s, which is younger than the user age ranges reported by other systems. As the app is only accessible via a smartphone, every user is assumed to be a smartphone user. The rate of smartphone ownership in South Korea is reported to be as high as 89.5% [29,30], and the smartphone usage rate is above 99% in the age group between 20 and 50 years [29]. Given the very high rate of smartphone penetration, smartphone users represent the general population of this age group in South Korea. Although there are no data about the gender of the users, it is expected that more females than males use the app based on the higher dependency on mothers for childcare in South Korea. This aligns with other studies that reported female participants being the majority. In a study that analyzed data from FNY, participants reporting for household members showed higher participation rate. This finding can partially explain the successful user recruitment and retention of Fever Coach, which is mostly used by parents for their children. As the service itself is targeted for children, the data collected from the app can only represent the pediatric population. However, our model from these data was able to predict influenza activity in the total population as well as in the pediatric population. This was because influenza activity in children is very closely related to that in adolescents and adults, but it may not be applicable for monitoring other diseases. Another noteworthy finding was the close prediction of influenza activity using the prediction model developed from the previous year despite several changes in the user population. Compared with the development period, the number of users decreased and the median number of data submission per child increased. This finding suggests that retention of active users can make up for the loss of inactive users for participatory surveillance.

    Limitations

    There are many limitations of this study that need to be addressed. First, most of the data submitted to Fever Coach came from children with febrile illness, and users generally do not use the app when their children are well. Therefore, increased app usage can either indicate an outbreak of an infectious disease or a surge of interest in the app itself following national media coverage. To overcome this limitation, we only used diagnosis reports to estimate ILI visits, contrary to other participatory surveillance systems that ask symptoms of users and use the data for syndromic surveillance. Although this strategy led to a good approximation of influenza activity, it introduces limited generalization. In most regions of South Korea, with physician density of 2.3 per 1000 population [31], primary care is considered affordable and readily accessible. However, in areas where health care is not as accessible, use of physician-made diagnosis may result in a bias toward a small group of people who have better access to health care than the general population. To overcome this second limitation, an additional study to develop more complex prediction models that include symptom and demographic data is in progress.

    Conclusions

    The Fever Coach app successfully collected data from 7.73% (207,699/2,686,580) of the target population by providing care instruction for febrile children. These data were used to develop a model that accurately estimated influenza activity measured by the central government agency using reports from sentinel facilities in the national surveillance network.

    Acknowledgments

    Namsoo Oh, CEO of Mobile Doctor, and Jaewon Shin, CEO of Mobile Doctor, provided the data used for this study. This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, Information and Communication Technology, and Future Planning (NRF-2017R1E1A1A03070934). This study used data from Mobile Doctor as well as Department of infectious disease surveillance, KCDC.

    Authors' Contributions

    HWH led the study and the discussion and review for the entire study. MK developed and designed the study, analyzed the data, and wrote the manuscript. SY reviewed the literature, designed the study, and wrote the manuscript. YJ and SOS managed the project and reviewed the manuscript. SC designed the study and analyzed the data. The final manuscript was approved by all the authors. HWH is the guarantor.

    Conflicts of Interest

    MK is a CMIO of Mobile Doctor which makes the service Fever Coach ( in Korean) and he has equity in Mobile Doctor less than 5%. SC is a CTO of Mobile Doctor and he has equity in Mobile Doctor less than 5%. No other author has reported any other conflict of interest relevant to this study.

    Multimedia Appendix 1

    A video of visualization. Visualization shows diagnosis report submission on the map geographically as time passes.

    MP4 File (MP4 Video), 25544 KB

    References

    1. Thompson WW, Weintraub E, Dhankhar P, Cheng P, Brammer L, Meltzer MI, et al. Estimates of US influenza-associated deaths made using four different methods. Influenza Other Respir Viruses 2009 Jan;3(1):37-49 [FREE Full text] [CrossRef] [Medline]
    2. World Health Organization. 2018. Influenza (Seasonal)   URL: https://www.who.int/en/news-room/fact-sheets/detail/influenza-(seasonal) [accessed 2019-01-30]
    3. Nair H, Brooks WA, Katz M, Roca A, Berkley JA, Madhi SA, et al. Global burden of respiratory infections due to seasonal influenza in young children: a systematic review and meta-analysis. Lancet 2011 Dec 3;378(9807):1917-1930. [CrossRef] [Medline]
    4. World Health Organization. 2017. Global Influenza Surveillance and Response System (GISRS)   URL: https://www.who.int/influenza/gisrs_laboratory/en/ [accessed 2019-01-30]
    5. World Health Organization. 2017. National Influenza Centres   URL: https:/​/www.​who.int/​influenza/​gisrs_laboratory/​national_influenza_centres/​national_influenza_centres.​pdf?ua=1 [accessed 2019-01-30]
    6. Steffen C, Debellut F, Gessner BD, Kasolo FC, Yahaya AA, Ayebazibwe N, et al. Improving influenza surveillance in sub-Saharan Africa. Bull World Health Organ 2012 Apr 1;90(4):301-305 [FREE Full text] [CrossRef] [Medline]
    7. Ortiz JR, Sotomayor V, Uez OC, Oliva O, Bettels D, McCarron M, et al. Strategy to enhance influenza surveillance worldwide. Emerg Infect Dis 2009 Aug;15(8):1271-1278 [FREE Full text] [CrossRef] [Medline]
    8. Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, Brilliant L. Detecting influenza epidemics using search engine query data. Nature 2009 Feb 19;457(7232):1012-1014. [CrossRef] [Medline]
    9. Choi J, Cho Y, Shim E, Woo H. Web-based infectious disease surveillance systems and public health perspectives: a systematic review. BMC Public Health 2016 Dec 8;16(1):1238 [FREE Full text] [CrossRef] [Medline]
    10. Lazer D, Kennedy R, King G, Vespignani A. Big data. The parable of Google Flu: traps in big data analysis. Science 2014 Mar 14;343(6176):1203-1205. [CrossRef] [Medline]
    11. Santillana M, Nsoesie EO, Mekaru SR, Scales D, Brownstein JS. Using clinicians' search query data to monitor influenza epidemics. Clin Infect Dis 2014 Nov 15;59(10):1446-1450 [FREE Full text] [CrossRef] [Medline]
    12. Sharpe JD, Hopkins RS, Cook RL, Striley CW. Evaluating Google, Twitter, and Wikipedia as tools for influenza surveillance using Bayesian change point analysis: a comparative analysis. JMIR Public Health Surveill 2016 Oct 20;2(2):e161 [FREE Full text] [CrossRef] [Medline]
    13. Woo H, Cho Y, Shim E, Lee J, Lee C, Kim SH. Estimating influenza outbreaks using both search engine query data and social media data in south Korea. J Med Internet Res 2016 Jul 4;18(7):e177 [FREE Full text] [CrossRef] [Medline]
    14. Kandula S, Hsu D, Shaman J. Subregional nowcasts of seasonal influenza using search trends. J Med Internet Res 2017 Nov 6;19(11):e370 [FREE Full text] [CrossRef] [Medline]
    15. Yang S, Santillana M, Brownstein JS, Gray J, Richardson S, Kou SC. Using electronic health records and internet search information for accurate influenza forecasting. BMC Infect Dis 2017 May 8;17(1):332 [FREE Full text] [CrossRef] [Medline]
    16. Smolinski MS, Crawley AW, Olsen JM, Jayaraman T, Libel M. Participatory disease surveillance: engaging communities directly in reporting, monitoring, and responding to health threats. JMIR Public Health Surveill 2017 Oct 11;3(4):e62 [FREE Full text] [CrossRef] [Medline]
    17. Koppeschaar CE, Colizza V, Guerrisi C, Turbelin C, Duggan J, Edmunds WJ, et al. Influenzanet: citizens among 10 countries collaborating to monitor influenza in Europe. JMIR Public Health Surveill 2017 Sep 19;3(3):e66 [FREE Full text] [CrossRef] [Medline]
    18. Bajardi P, Paolotti D, Vespignani A, Eames K, Funk S, Edmunds WJ, et al. Association between recruitment methods and attrition in internet-based studies. PLoS One 2014;9(12):e114925 [FREE Full text] [CrossRef] [Medline]
    19. Kim JH, Yun S, Hwang SS, Shim JO, Chae HW, Lee YJ, Committee for the Development of Growth Standards for Korean Children and Adolescents, Committee for School Health and Public Health Statistics‚ the Korean Pediatric Society, Division of Health and Nutrition Survey‚ Korea Centers for Disease Control and Prevention. The 2017 Korean National Growth Charts for children and adolescents: development, improvement, and prospects. Korean J Pediatr 2018 May;61(5):135-149 [FREE Full text] [CrossRef] [Medline]
    20. Centers for Disease Control and Prevention Korea. 2019. Stastical Information of Sentinel Surveillance of Influenza, 2019   URL: http://www.cdc.go.kr/npt/biz/npp/iss/influenzaStatisticsMain.do [accessed 2019-01-30]
    21. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in python. J Mach Learn Res 2015;12(2011):2825-2830 [FREE Full text]
    22. National Statistics Portal - KOSIS. 2019. Summary of Census Population (By Administrative District/Sex/Age), 2017   URL: http://kosis.kr/statHtml/statHtml.do?orgId=101&tblId=DT_1IN1503&conn_path=I2 [accessed 2019-01-30]
    23. Fever Coach. 2019. Fever Coach Flu Visualization   URL: http://ec2-3-83-42-212.compute-1.amazonaws.com/ [accessed 2019-01-30]
    24. Smolinski MS, Crawley AW, Baltrusaitis K, Chunara R, Olsen JM, Wójcik O, et al. Flu near you: crowdsourced symptom reporting spanning 2 influenza seasons. Am J Public Health 2015 Oct;105(10):2124-2130. [CrossRef] [Medline]
    25. Dalton C, Carlson S, Butler M, Cassano D, Clarke S, Fejsa J, et al. Insights from flutracking: thirteen tips to growing a web-based participatory surveillance system. JMIR Public Health Surveill 2017 Aug 17;3(3):e48 [FREE Full text] [CrossRef] [Medline]
    26. Fujibayashi K, Takahashi H, Tanei M, Uehara Y, Yokokawa H, Naito T. A new influenza-tracking smartphone app (flu-report) based on a self-administered questionnaire: cross-sectional study. JMIR Mhealth Uhealth 2018 Jun 6;6(6):e136 [FREE Full text] [CrossRef] [Medline]
    27. Chan TC, Hu TH, Hwang JS. Estimating the risk of influenza-like illness transmission through social contacts: web-based participatory cohort study. JMIR Public Health Surveill 2018 Apr 9;4(2):e40 [FREE Full text] [CrossRef] [Medline]
    28. Nguyen T. Medium. 2017. Online Communities in South Korea: A Case Study   URL: https://medium.com/@thinguyens/online-communities-in-south-korea-a-case-study-5815dd589630 [accessed 2019-01-30]
    29. Korea Internet Information Center (KRNIC). 2018. Survey on the Internet Usage, 2017   URL: https://isis.kisa.or.kr/board/index.jsp?pageId=060100&bbsId=7&itemId=821&pageIndex=1 [accessed 2019-01-30]
    30. Poushter J. Pew Research Center. 2016. Smartphone Ownership and Internet Usage Continues to Climb in Emerging Economies   URL: http:/​/www.​pewglobal.org/​2016/​02/​22/​smartphone-ownership-and-internet-usage-continues-to-climb-in-emerging-economies/​ [accessed 2019-01-15]
    31. World Health Organization. 2018. Global Health Observatory Data Repository   URL: http://apps.who.int/gho/data/node.main.A1444 [accessed 2019-01-15]


    Abbreviations

    FNY: Flu Near You
    GFT: Google Flu Trends
    ILI: influenza-like illness
    IRB: institutional review board
    KCDC: Korea Centers for Disease Control and Prevention
    NRF: National Research Foundation of Korea
    RMSE: root mean square error


    Edited by G Eysenbach; submitted 06.04.19; peer-reviewed by D Paolotti, L Sheets; comments to author 28.05.19; revised version received 14.07.19; accepted 09.08.19; published 17.10.19

    ©Myeongchan Kim, Sehyo Yune, Seyun Chang, Yuseob Jung, Soon Ok Sa, Hyun Wook Han. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 17.10.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.