The Growing healthy program utilized a quasi-experimental design aimed to support parents of young infants with healthy infant feeding behaviors. To enhance intervention effectiveness, the behavior change wheel model [24], as well as the mode of delivery, content, and quality of the program were considered during the development phase.

Eligible participants were offered to use the Growing healthy app and could choose to receive 3 tailored push notifications through the app each week of the intervention (9 months of the baby’s age). Although, midway through the intervention implementation period, participants were also sent a weekly email due to identifying technological issues with receiving and opening push notifications. The weekly emails included the same messages as the push notifications sent each week. Participants who did not own a phone that was compatible with the app were offered access to the Growing healthy website and were sent 3 text messages. Details of the study has been published previously [5]. The focus of recruitment was parents from socioeconomically disadvantaged regions and resulted in 300 participants. Recruitment was conducted via health practitioners, face to face, or Web-based methods. Eligibility criteria included: (1) expectant parents (30+ weeks gestation) or parents with an infant less than 3 months of age, (2) literate in English, (3) living in Australia, (4) 18 years or older, and (5) ownership of any type of mobile phone or Internet access. Further details of the recruitment process and outcomes have been published elsewhere [25]. As Growing healthy was a feasibility study, the sample size was tailored to logistical limitations of the time and funds available to support recruitment. The EI scoring was only performed for participants’ data when the participant registered for the Growing healthy app, activated and accessed the app at least once, and used the app and opened push notifications or weekly emails of the Growing healthy program. The focus of this paper was to report the EI for the intervention group.

Study participants completed 3 quantitative surveys: (1) baseline (T1) (infant age ≤3 months), (2) infant aged 6 months (T2), and (3) infant aged 9 months (T3). The surveys included demographic and infant feeding behavior questions. Participants’ use of the app was captured and the data was used to develop the EI and evaluate the Growing healthy program.

The Web Analytics Demystified visitor EI [16] was adapted to develop a composite measure of engagement for Growing healthy app users. This index was chosen because a detailed description on how to develop and apply it was available. The original index comprised 7 subindices which measured: (1) click depth, (2) loyalty, (3) recency, (4) interaction, (5) feedback, (6) brand, and (7) duration index. Figure 1 presents the adapted term-definitions of subindices included in this study (see Multimedia Appendix 1 for questions). All but 2 subindices (brand index and duration index) from the Web Analytics Demystified visitor EI were available from the app database collected in this study. Although measuring all indices is ideal, the Web Analytics Demystified visitor EI protocol emphasized that the calculation can be adapted to suit the program based on data collected [16]. The developed EI provided a score for each participant that measured their overall engagement with the app against a predetermined criteria. The time frame under consideration was from date of registration to 39 weeks (9 months) from the participants’ infant’s date of birth.

Metrics needed to calculate the subindices (outlined in Figure 1) were identified and extracted from the Growing healthy app database. The key metrics collected included “session duration,” “page views per session,” and “number of push notifications opened.” Furthermore, subjective markers such as feedback and satisfaction captured at the T3 survey (9 month of the baby’s age) was also used to calculate the EI score.

Equal weight for each of the subindices was assigned to the overall EI score so that each element was equally important in contributing to the measurement of engagement. Four of the subindices were calculated using app data. The feedback index was informed using responses to the 9-month survey (T3) feedback questions. The final formula used to calculate the EI incorporated click depth, loyalty, recency, interaction, and feedback subindices (see equation 1). The EI was then converted to a value between 0 and 100.

Equation 1: Engagement index formula

where EI is engagement index, C_i is click depth index, L_i is loyalty index, I_i is interaction index, R_i is recency index, and F_i is feedback index.

The calculation for each subindex except the feedback index (data were only collected at the end of the program) was done for three time periods, including initial (0-3 months), interim (3-6 months), and final (6-9 months) and were then averaged. This grouping of time periods was chosen because there was an initial intense use of the app followed by infrequent participant use toward the end of the 9-month program. A detailed explanation for the calculation of each subindex follows.

Basic descriptive data analysis was performed on the metrics and components of the EI as well as the final EI score. To analyze the EI scores, cut-off points were developed based on the distribution of the total samples’ EI scores using quartiles. Participants were then categorized as either poorly, moderately, or highly engaged. This method was chosen as there were no existing mHealth interventions that utilized an EI and categorized participants’ engagement based on app use.

Group comparisons between poorly, moderately, or highly engaged participants were then conducted using t-tests, analysis of variance (ANOVA), Mann-Whitney, Kruskall-Wallis, and Spearman correlation tests were used as appropriate. Consideration of independent variables associated with the EI score were modeled using linear regression models.

The following variables were dichotomized for analysis including:

Other independent variables considered included mother’s age, country of birth, as well as infant’s age at the start of the program, their birth weight, and feeding status at baseline. All analyses were performed using used IBM SPSS Version 23.0.

This is one of the first studies to develop and implement an mHealth program supporting parents with healthy infant feeding practices through a mobile phone app. To our knowledge, this is the first study to utilize an EI to quantify and categorize participants’ engagement level using the app. We found that engagement level was positively correlated with primiparous status, use of both the app and email, exposure to the program for a longer period, and recruitment through health practitioners. Negative correlation was found with age of child at start of program and engagement level.

The identification of the correlates of participant engagement is not only beneficial to inform future enhancements of the Growing healthy program, but more broadly to evaluate mHealth programs. The EI has its origins in measurement of consumer engagement with Web-based products. Adjusting the index to measure engagement with a mHealth program was possible as the metrics measured are the same; only the measurement of the content and behavior will be different [16].

A criterion to categorize participants as poor, moderate, or highly engaged with the Growing healthy program based on their overall EI score was developed. Previously, program engagement has arbitrarily been labeled as high [27] or low [22] based on the frequency participants accessed websites or apps. Few studies have considered participant engagement on the basis of their interaction with multiple intervention elements. In addition, there is not a standardized approach to measuring engagement. For example, a point system to gauge individual user activity with the program features was used in a study targeting reduction of high-risk sexual behaviors. The measures included were frequency of access, profile modification, message views, article views, completion of quizzes, number of pages viewed, and updates of personal goals [23]. Whereas modeling participants’ engagement with frequency of access, average daily steps, and the number of days since participants last accessed the program was performed for a physical activity focused mHealth intervention [20].

Participant app use over the 9-month period in this study varied such that engagement was high after initially joining the program but decreased from the 3- to 6-month period. Previous mHealth programs targeting long and short term behavior change have identified similar patterns of use [27-29]. Attrition with mHealth programs is negatively affected by factors such as lack of commitment or motivation to change health behaviors [28], confidence in knowledge about managing the targeted behavior [27], and programs that are perceived as overly burdensome by participants [23]. The Growing healthy program was a “just in time” resource [15] developed to provide infant feeding information up to 9 months of the infants age. Feeding milestones targeted included breastfeeding, best practice formula feeding, timing of the introduction of solids and optimizing dietary exposure to fruit and vegetables, and minimizing exposure to noncore foods. Once that knowledge is obtained, it is likely that participant app use will drop off [30]. When targeting long term behavior change, mHealth developers need to consider ongoing novel strategies that will keep participants engaged. Qualitative findings suggest users prefer to engage with apps periodically [28]. Such findings highlight that we must seek to understand app users’ behavior to inform the most appropriate time to engage them to join the program, the factors that lead to disengagement, and to consider strategies that will maintain their engagement.

Study participants who accessed both the website and the app attained a significantly higher EI score compared with participants who had just used the app. This supports the notion that delivering the intervention using various modes enhances engagement and to the intervention exposure [11,23]. Primiparous participants had significantly higher EI scores than multiparous women. This is congruent with qualitative analysis conducted as part of the development phase for the Growing healthy program (unpublished) where most of the primiparous participants expressed an interest in the program, while multiparous participants suggested the resource would have been more useful as a first time parent. Despite multiparous participants being less engaged, more than one third of those classified as highly engaged were indeed multiparous.

While initial engagement is the initial hurdle for any intervention, sustaining engagement remains the most difficult part of intervention implementation, it is more difficult to achieve [11,23,31]. It has been found that novelty and relevance are main contributors to sustained app user engagement [11,32,33]. The downward trend in engagement subindices scores from 3 to 6 months reported in this study may be a reflection of a lack of perceived novelty in the Growing healthy app throughout the intervention period. This may also explain the lower engagement of multiparous participants (who have developed their thinking around infant feeding already).

The infant’s age at baseline (ie, when the app was downloaded) was also strongly associated with higher EI scores. Participants who joined the program when their infant was younger had a higher EI score compared with those who joined when their infant’s age was closer to 3 months. Similar to traditional interventions that targeted childhood obesity prevention [34], early recruitment was necessary to increase participant engagement. Early recruitment is likely to increase intervention exposure, which is associated with an increased likelihood of influencing the uptake of the desired behaviors [27]. This is important to target as infant development clearly occurs rapidly within the first year of life. The app was likely to be most useful and provided novel information to mothers if they were recruited from early postpartum or during pregnancy.

Participants who were recruited from their health practitioner were more likely to have higher EI scores compared with those who were recruited on the Web. This may be attributed to mothers’ perception that health practitioners are a trustworthy source of information [35]. The involvement of health practitioners such as maternal and child health nurses and practice nurses who do routine infant health checks during the first few years of life [36], are important as a key “referral pathway” to evidence based apps and in turn, to the most effective utilization of apps.

Several studies describing mHealth interventions encouraging healthy infant feeding behaviors have recently been published. Delivery modes used in these studies included app [37,38], websites [39,40], and social media [41]. Due to the different delivery modes, the findings of this study cannot be compared with other programs. However, as mHealth interventions are novel modes of delivering health behavior change interventions across health disciplines, similar patterns of engagement have been reported by several researchers albeit using different measures [23,42,43].

This study has several limitations. First, a number of technological issues were experienced by participants in receiving and opening push notifications. Adaptation were therefore made midway during the program and all participants were sent weekly emails. Second, app quality is an important influencer on participant engagement [44-46]. The participants’ responses to the satisfaction survey (feedback index) demonstrated low satisfaction with respect to the push notifications, emphasizing the impact technological difficulties have on participant engagement. Third, the weekly emails contained links to the Growing healthy website rather than the app. Finally, participant behavior on the website, such as the number of pages viewed, was not accessible at an individual level. This explains the increase in loyalty index scores at around 3 to 6 months, as participant access to email links was included. Click depth index scores decreased at that time point because the number of pages viewed on the website could not be measured. Overall, the EI score calculated for these participants is most likely an underestimate of their engagement with the program.

Some features of the Growing healthy program were not measured using the EI because there were difficulties in obtaining individual participants’ information such as, participant use of the Growing healthy Facebook group and sharing the app with another carer or sharing information from the app with others (interconnectivity). Although participant interaction with these features was not measured, satisfaction and use of these features was included in the 9-month survey that made up the feedback index.

Some studies have shown that mothers from a disadvantaged background were less likely to use the Internet as a source of information for infant feeding [47]. A strength of our study was that approximately equal number of participants of both high and low educational background were recruited unlike other mHealth programs targeted at addressing infant feeding [48].

To our knowledge, the utilization of an index to measure participant engagement has not yet been implemented in mHealth interventions. The EI provided detailed analysis regarding the frequency participants accessed the app and push notifications, how many pages they accessed per session, and their satisfaction with the program which was measured over 3 time points across the 9 months of the program.

The EI provided a comprehensive understanding of participant behavior with the app over the 9-month period of the Growing healthy program. The participants’ engagement with the Growing healthy app was determined by various factors including participant characteristics, novelty, intervention exposure time, and the quality of the app including technological aspects. Primiparous participants, those who accessed both the emails and the app, those who were exposed to the program for a longer period, and those who were recruited from their health practitioner all had higher EI scores. The use of the EI in this study demonstrates that rich and useful data can be collected and used to assess the strengths and weaknesses of mHealth interventions and in turn inform improvements in their design and delivery.