This paper is in the following e-collection/theme issue:

Evaluation and Research Methodology for mHealth (193) Usage Data and Process Outcomes in eHealth Trials (56) Use and User Demographics of mHealth (311) Engagement with and Adherence to Digital Health Interventions, Law of Attrition (457)

Published on in Vol 5, No 6 (2017): June

Assessing User Engagement of an mHealth Intervention: Development and Implementation of the Growing Healthy App Engagement Index

Assessing User Engagement of an mHealth Intervention: Development and Implementation of the Growing Healthy App Engagement Index

Assessing User Engagement of an mHealth Intervention: Development and Implementation of the Growing Healthy App Engagement Index

Authors of this article:

Sarah Taki1, 2 Author Orcid Image ;   Sharyn Lymer2, 3 Author Orcid Image ;   Catherine Georgina Russell2, 4 Author Orcid Image ;   Karen Campbell2, 4 Author Orcid Image ;   Rachel Laws2, 4 Author Orcid Image ;   Kok-Leong Ong5 Author Orcid Image ;   Rosalind Elliott1 Author Orcid Image ;   Elizabeth Denney-Wilson1, 2 Author Orcid Image
Article Authors Cited by (75) Tweetations (32) Metrics

    Original Paper

    1University of Technology Sydney, Sydney, Australia

    2Centre for Obesity Management and Prevention Research Excellence in Primary Health Care, Sydney, Australia

    3University of Sydney, Sydney, Australia

    4Deakin University, Victoria, Australia

    5Latrobe University, Melbourne, Australia

    Corresponding Author:

    Sarah Taki, PhD

    University of Technology Sydney

    Bldg 10, 8th Fl

    235 Jones St, Ultimo NSW

    Sydney, 2007

    Australia

    Phone: 61 0424393630

    Fax:61 95142000

    Email: sarah.b.taki@student.uts.edu.au


    Background: Childhood obesity is an ongoing problem in developed countries that needs targeted prevention in the youngest age groups. Children in socioeconomically disadvantaged families are most at risk. Mobile health (mHealth) interventions offer a potential route to target these families because of its relatively low cost and high reach. The Growing healthy program was developed to provide evidence-based information on infant feeding from birth to 9 months via app or website. Understanding user engagement with these media is vital to developing successful interventions. Engagement is a complex, multifactorial concept that needs to move beyond simple metrics.

    Objective: The aim of our study was to describe the development of an engagement index (EI) to monitor participant interaction with the Growing healthy app. The index included a number of subindices and cut-points to categorize engagement.

    Methods: The Growing program was a feasibility study in which 300 mother-infant dyads were provided with an app which included 3 push notifications that was sent each week. Growing healthy participants completed surveys at 3 time points: baseline (T1) (infant age ≤3 months), infant aged 6 months (T2), and infant aged 9 months (T3). In addition, app usage data were captured from the app. The EI was adapted from the Web Analytics Demystified visitor EI. Our EI included 5 subindices: (1) click depth, (2) loyalty, (3) interaction, (4) recency, and (5) feedback. The overall EI summarized the subindices from date of registration through to 39 weeks (9 months) from the infant’s date of birth.
    Basic descriptive data analysis was performed on the metrics and components of the EI as well as the final EI score. Group comparisons used t tests, analysis of variance (ANOVA), Mann-Whitney, Kruskal-Wallis, and Spearman correlation tests as appropriate. Consideration of independent variables associated with the EI score were modeled using linear regression models.

    Results: The overall EI mean score was 30.0% (SD 11.5%) with a range of 1.8% - 57.6%. The cut-points used for high engagement were scores greater than 37.1% and for poor engagement were scores less than 21.1%. Significant explanatory variables of the EI score included: parity (P=.005), system type including “app only” users or “both” app and email users (P<.001), recruitment method (P=.02), and baby age at recruitment (P=.005).

    Conclusions: The EI provided a comprehensive understanding of participant behavior with the app over the 9-month period of the Growing healthy program. The use of the EI in this study demonstrates that rich and useful data can be collected and used to inform assessments of the strengths and weaknesses of the app and in turn inform future interventions.

    JMIR Mhealth Uhealth 2017;5(6):e89

    doi:10.2196/mhealth.7236

    Keywords

    mHealth; social medium; infant obesity; infant development; children; infants; practitioners; primary healthcare 


    Mobile phone ownership is widespread in Australia and internationally [1,2] and many people use their phone to gain information, browse websites, and use apps [1,2]. Ownership of mobile phones is high across all socioeconomic groups and the mobile phone is a promising tool for delivery of behavior change interventions [3,4]. A mobile phone app was used to provide information and support to parents regarding infant feeding for the Growing healthy program in Australia [5].

    Capturing the attention of an app user is clearly paramount to the app’s potential effectiveness for behavior change. To be successful, apps must continuously and actively engage the user. User engagement refers to the quality of the user experience, the positive aspects of their interaction, and their desire to use the app over longer periods of time or repeatedly [6]. A recently published review on digital behavior change interventions identified that content and delivery, the setting in which the intervention is used, the demographic, and the targeted behavior influences engagement [7]. Furthermore, the Medical and Research Council (MRC) framework emphasized the importance to utilize theoretical models for the development of effective interventions [8]. This was further supported by the findings in a review which explored the effectiveness of mobile phone apps targeting health behaviors [9]. They identified that interventions which utilized health behavior models were more likely to have an impact.

    Engagement with technology is inherently complex and multifaceted in its nature and it may be mediated by factors such as family, community, culture, and context [10]. O’Brien and Toms [11] posit that engagement is not static, but a process with four distinct stages: (1) point of engagement, (2) period of engagement, (3) disengagement, and (4) reengagement. Thus, a user’s engagement is considered to be operating over a continuum and this may vary within a session and over long time periods [11]. User engagement and its measurement can be either short or long term, with long term engagement reflecting the degree of involvement a user has with the system (eg, an app) over time [12]. There have been multiple approaches to the measurement of user engagement, reflecting the many elements considered to comprise engagement. These include users’ physical participation in a specific target behavior and behavior in virtual spaces (eg, frequency of access), although it is the user’s psychological state and perceived experience that is most relevant to engagement [10].

    Large scale quantitative measures of engagement rely on Web analytics which provide the opportunity to measure behavioral aspects of engagement. Some examples of data that can be collected, but is not exhaustive to, includes frequency of access to the app, page views, push notifications opened, and average time spent on a page [13]. These metrics unlike other engagement measures based on subjective questionnaires and psychological testing can be applied to the study population with no respondent burden. Web analytics provide insight via these proxy measures about the dynamics of participant engagement and its relationship with app effectiveness. They also provide insights regarding areas for app improvements, lower participant attrition, and in turn increased intervention exposure [14,15].

    It has been suggested that Web analytics measures can be classed into three main dimensions of engagement: popularity, activity, and loyalty [6]. To achieve a more in-depth understanding of consumer behaviors and their influencers, “engagement indices,” accounting for these three dimensions of engagement, have been used to calculate the users’ overall interaction with Web-based technologies [16,17]. Engagement indices provide quantitative evidence regarding the strengths and weaknesses of website and app features to optimize participant engagement and sustainable long-term use of the app.

    Little work has been done in the mHealth arena with respect to the conceptualization and measurement of user engagement [7,18,19]. The work done has been mainly around apps for patient engagement of those with chronic disease or around public health and behavior change such as increased physical activity and weight loss [20-22]. Few mHealth programs comprehensively use the available data to analyze participant engagement or to consider its associations with primary outcomes [18,20,21,23]. This paper describes the development of a fit-for-purpose engagement index (EI) based on Web metrics that allows large scale implementation. The EI reported in this paper was developed for the Growing healthy program which used a mobile phone app [5] to provide information and support to parents regarding infant feeding. We provide a rationale and description of the development of an EI to measure participants’ behavior utilizing the Growing healthy app; describe the assignment of cut-points for poorly, moderately, or highly engaged users; and investigate determinants effecting participants’ engagement with the app.


    Growing Healthy Feasibility Study

    The Growing healthy program utilized a quasi-experimental design aimed to support parents of young infants with healthy infant feeding behaviors. To enhance intervention effectiveness, the behavior change wheel model [24], as well as the mode of delivery, content, and quality of the program were considered during the development phase.

    Eligible participants were offered to use the Growing healthy app and could choose to receive 3 tailored push notifications through the app each week of the intervention (9 months of the baby’s age). Although, midway through the intervention implementation period, participants were also sent a weekly email due to identifying technological issues with receiving and opening push notifications. The weekly emails included the same messages as the push notifications sent each week. Participants who did not own a phone that was compatible with the app were offered access to the Growing healthy website and were sent 3 text messages. Details of the study has been published previously [5]. The focus of recruitment was parents from socioeconomically disadvantaged regions and resulted in 300 participants. Recruitment was conducted via health practitioners, face to face, or Web-based methods. Eligibility criteria included: (1) expectant parents (30+ weeks gestation) or parents with an infant less than 3 months of age, (2) literate in English, (3) living in Australia, (4) 18 years or older, and (5) ownership of any type of mobile phone or Internet access. Further details of the recruitment process and outcomes have been published elsewhere [25]. As Growing healthy was a feasibility study, the sample size was tailored to logistical limitations of the time and funds available to support recruitment. The EI scoring was only performed for participants’ data when the participant registered for the Growing healthy app, activated and accessed the app at least once, and used the app and opened push notifications or weekly emails of the Growing healthy program. The focus of this paper was to report the EI for the intervention group.

    Study participants completed 3 quantitative surveys: (1) baseline (T1) (infant age ≤3 months), (2) infant aged 6 months (T2), and (3) infant aged 9 months (T3). The surveys included demographic and infant feeding behavior questions. Participants’ use of the app was captured and the data was used to develop the EI and evaluate the Growing healthy program.

    Engagement Index

    The Web Analytics Demystified visitor EI [16] was adapted to develop a composite measure of engagement for Growing healthy app users. This index was chosen because a detailed description on how to develop and apply it was available. The original index comprised 7 subindices which measured: (1) click depth, (2) loyalty, (3) recency, (4) interaction, (5) feedback, (6) brand, and (7) duration index. Figure 1 presents the adapted term-definitions of subindices included in this study (see Multimedia Appendix 1 for questions). All but 2 subindices (brand index and duration index) from the Web Analytics Demystified visitor EI were available from the app database collected in this study. Although measuring all indices is ideal, the Web Analytics Demystified visitor EI protocol emphasized that the calculation can be adapted to suit the program based on data collected [16]. The developed EI provided a score for each participant that measured their overall engagement with the app against a predetermined criteria. The time frame under consideration was from date of registration to 39 weeks (9 months) from the participants’ infant’s date of birth.

    Metrics needed to calculate the subindices (outlined in Figure 1) were identified and extracted from the Growing healthy app database. The key metrics collected included “session duration,” “page views per session,” and “number of push notifications opened.” Furthermore, subjective markers such as feedback and satisfaction captured at the T3 survey (9 month of the baby’s age) was also used to calculate the EI score.

    ‎
    Figure 1. The definitions of the subindices for the engagement index designed for the Growing healthy program where i=ith person and j=jth time period and n=3 for Ci, Li, Ii, and Ri (sum of calculation period) and n=37 for Fi.
    View this figure

    Equal weight for each of the subindices was assigned to the overall EI score so that each element was equally important in contributing to the measurement of engagement. Four of the subindices were calculated using app data. The feedback index was informed using responses to the 9-month survey (T3) feedback questions. The final formula used to calculate the EI incorporated click depth, loyalty, recency, interaction, and feedback subindices (see equation 1). The EI was then converted to a value between 0 and 100.

    Equation 1: Engagement index formula

    EI=∑(C i+L i+I i+R i+F i)×100

    where EI is engagement index, Ci is click depth index, Li is loyalty index, Ii is interaction index, Ri is recency index, and Fi is feedback index.

    The calculation for each subindex except the feedback index (data were only collected at the end of the program) was done for three time periods, including initial (0-3 months), interim (3-6 months), and final (6-9 months) and were then averaged. This grouping of time periods was chosen because there was an initial intense use of the app followed by infrequent participant use toward the end of the 9-month program. A detailed explanation for the calculation of each subindex follows.

    Click Depth Index (Ci)

    The number of pages a participant viewed the app in each access session over the total number of sessions in each time period formed the basis of this subindex. Two metrics were used in the calculation of Ci: the number of sessions in the time period and number of pages viewed per session. A threshold of the number of pages viewed per session was applied. There is no benchmark of an effective click depth, that is, “dose” of the interaction in the mHealth environ. Based on the data collected, the median value of 2 pages per session was used as the threshold. The overall score of Ci was the average of each time period calculation: Ci1, Ci2, and Ci3.

    Loyalty Index (Li)

    This subindex was based on the frequency of app access throughout the 9-month program. Li was the reciprocal of the number of sessions in each time period. The total score was dependent on when participants activated the app. The overall score of Li was the average of each time period calculation: Li1, Li2, and Li3.

    Interaction Index (Ii)

    The number of push notifications opened versus total sent throughout the 9-month program formed the basis of this subindex. Interaction Index was the total number of push notifications opened divided by the number sent in the time period. This was calculated for 3 month time intervals of the infant’s age according to when the participant activated the app until the infant reached 9 months of age. The overall score of Ii is the average of each time period calculation: Ii1, Ii2, and Ii3.

    Recency Index (Ri)

    The number of days between each session was the basis of the recency index. The Ri was calculated for three different time points: (1) the number of days elapsed from registration to when the participant first accessed the app (Ri1), (2) the average number of days between sessions when the participant accessed the app between 3 to 6 months (Ri2), and (3) 6 and 9 months (Ri3). The data were transformed by taking the reciprocal of each Ri1 to Ri3. The overall score of Ri was the average of each time period calculation: Ri1, Ri2, and Ri3.

    Feedback Index (Fi)

    This subindex was a self-reported measure of participant satisfaction with the app, which was captured in the 9-month survey (T3). Constructive feedback was scored positively as 1 and negatively as 0. The 9-month survey included 37 questions which formed the basis of Fi. Each question (Multimedia Appendix 1) used a 5-point Likert scale response ( “strongly agree” to “strongly disagree” and “didn’t use”). The responses were dichotomized as either 1 or 0 according to whether they answered an extreme positive response or not; for example: strongly agree=1, agree=0, neither here nor there=0, disagree=0, strongly disagree=0, and didn’t use=0. Extreme positive scoring was reversed on the Likert scale for questions worded negatively. Although only app users were eligible for this study, some app users reported using the website rather than the app in the T3 survey (n=15) and thus were not asked the feedback questions. The EI total score for these participants were averaged across the 4 subindices that data were available. In addition, a number of participants (n=102) did not complete the T3 survey. For these participants Fi was zero and the EI was averaged across the 5 subindices.

    Statistical Analysis

    Basic descriptive data analysis was performed on the metrics and components of the EI as well as the final EI score. To analyze the EI scores, cut-off points were developed based on the distribution of the total samples’ EI scores using quartiles. Participants were then categorized as either poorly, moderately, or highly engaged. This method was chosen as there were no existing mHealth interventions that utilized an EI and categorized participants’ engagement based on app use.

    Group comparisons between poorly, moderately, or highly engaged participants were then conducted using t-tests, analysis of variance (ANOVA), Mann-Whitney, Kruskall-Wallis, and Spearman correlation tests were used as appropriate. Consideration of independent variables associated with the EI score were modeled using linear regression models.

    The following variables were dichotomized for analysis including:

    • Education level: university degree (“degree” or “higher degree”) or no university (“high school education or less,” “trade certificate,” or “diploma”)
    • Employment status: working or studying (“full or part-time,” “casual paid work,” and “full or part-time studying”) or not in labor force (“keeping house and/or raising children full-time” and “unemployed or laid off”)
    • Gross household income: below average (“Aus $1-$119 per week,” “Aus $120-$299 per week,” “Aus $300-$599 per week,” “Aus $600-$799 per week,” “Aus $800-$999 per week”) average (“Aus $1000-$1499 per week”), above average (“Aus $1500-1999 per week”), or higher income (“Aus $2000 or more per week”)
    • Marital status: relationship (“married,” “living in a defacto relationship”) or single (“separated,” “divorced,” “widowed,” “never married”)
    • Recruitment method: practitioner, Web-based, or family or friends
    • Device type: android or iOS
    • System type: app only or both app and email

    Other independent variables considered included mother’s age, country of birth, as well as infant’s age at the start of the program, their birth weight, and feeding status at baseline. All analyses were performed using used IBM SPSS Version 23.0.


    Of the 300 Growing healthy participants who completed the baseline survey, 75.0% (225/300) met the inclusion criteria for this study. The average age of participants was 30 years, with 62.2% (186/300) being first time parents, 97.0 % (291/300) living with their partner, and 84.0% (252/300) being full-time carers of the infant. The infants’ were on average 6.9 weeks old when registration occurred and 56.4 %( 169/300) were breastfed.

    The EI score had a distribution that was not statistically significant as evidenced by nonsignificant Kolomogorov Smirnov (KS) test at P value of .05 and a standard error of skewness (SES) between >−1.96 and <1.96 (SES=0.58). The mean EI score was 30.0% (SD 11.5%) and ranged between 1.8% to 57.6% (see Figure 2). The interquartile ranges were used for categorization, where: (1) poor engagement for scores less than or equal to 21.1% (≤Q1), (2) moderate engagement if scores were between 21.1% and 37.1% (Q1-Q3), and (3) high engagement if score were greater than or equal to 37.1%(≥Q3).

    Three variables were significantly associated with high engagement in univariate analysis (see Table 1 for details). Participants most likely to be classed as having high engagement were first time parents (primiparous), who used both the app and opened weekly emails, and had joined the program with a younger infant and were part of the program for longer. Approximately 64% of participants with higher level education (university degree) were classed as having high engagement compared with 55.3% of those with lower levels of education (no university). Additional demographic descriptors for the different engagement levels are shown in Table 1. Engagement index cut-points for scores: poor engagement ≤21.1%, moderate engagement=21.1%-37.1%, and high engagement ≥37.1%. Variables are based on data provided at baseline or T1 (age ≤3 months).

    ‎
    Figure 2. Overall engagement index scores distribution.
    View this figure
    Table 1. Characteristics of growing healthy participants based on engagement index level (n=255).
    VariablesPoor engagement (n=56)
    15.1 (SD 4.6)    		Moderate engagement (n=113)
    30.0 (SD 4.3)    		High engagement (n=56)
    45.0 (SD 5.5)    		P value
    Participant characteristics




    		Age (years)a, mean (SD)30.3 (4.4)30.5 (4.4)30.6 (4.5).61

    		Education (no university)b, n (%)31 (55)61 (54)20 (36).11

    		Income (higher income)c, n (%)14 (25)36 (31.8)13 (23.2).70

    		Marital status (relationship)b, n (%)54 (96)110 (97.3)53 (95).59

    		Employment status (not in labor force)b, n (%)51 (91)97 (85.8)45 (80).14

    		Parity (Primiparous)b, n (%)29 (52)70 (61.9)41 (73).004d

    		Recruitment method (Practitioner)c, n (%)24 (48)52 (47.9)30 (48).06

    		Device type (iOS)b, n (%)38 (68)87 (72.5)36 (64).73

    		System type (both app & email users)b, n (%)26 (46)82 (72.5)52 (93)<.001d
    Infant characteristics




    		Age at registration (weeks)a, mean (SD)7.3 (3.6)7.4 (3.6)5.6 (3.4).02d

    		Birth weight (kg)a, mean (SD)3.46 (0.591)3.47 (0.593)3.47 (0.592).20

    		Gender (male)b, n (%)31 (55.3)53 (46.9)23 (41.0).34
    Baseline feeding statusc, n (%)




    		Breastfeeding35 (63)64 (56.6)28 (50).13

    		Formula feeding14 (25)25 (22.1)23 (41)

    		Mixed feeding7 (12.5)24 (21.2)5 (8.9)

    aPearson correlation; mean, standard deviation (SD) reported.

    bt test; % within group (count) reported.

    cBased on ANOVA; % within group (count) reported.

    dStatistically significant engagement level and independent variable <.05.

    Of the 14 variables assessed in this study, 8 met the including criterion of P ≤.25 in the univariate analysis and were included in the multivariate linear model (full model) [26] presented in Table 2. Similar results were found for 4 variables which were significantly associated with EI scores as presented in the reduced model (see Table 2 for details). Higher EI scores were found among those mothers who were primiparous, using both the app and accessing the email, recruited to the program by their health practitioner and those who registered when their infant was younger.

    To better understand the drivers of engagement descriptive analysis of the subindices that made up the overall EI score was performed (Table 3). The click depth index (Ci) median score was 30.8% (IQR: 21.0%-37.2%). Of the 303 pages that were available to view, the mean number of pages viewed was 30 (range: 1-156) and a median of 24. Although, throughout the program, participants viewed a mean of 44.2 pages (range: 1-316) and a median of 29. Figure 3 illustrates the most commonly viewed pages on the app including the number of times each page was visited and the number of participants that visited each page. The solids section was viewed the most and mixed feeding was viewed the least.

    Table 2. Linear regression to explore the predictors of infant and participant characteristics with the engagement index scores.
    VariableUnivariate model (B)P valueFull model (B)P valueReduced model (B)P value
    R2

    		0.154
    		0.164
    Parity
    		.004
    		.006
    		.005

    		Multiparous1.00
    		1.00
    		1.00

    		Primiparous4.532
    		4.147
    		4.209
    Recruitment method
    		.06
    		.07
    		.02

    		Family or friends1.00
    		1.00
    		1.00

    		Practitioner5.346
    		6.423
    		4.221

    		Web-based2.795
    		4.267
    		0.989
    System type
    		<.001
    		<.001
    		<.001

    		App only1.00
    		1.00
    		1.00

    		Both (app and email)7.977
    		−6.426
    		−6.937

    		Infant age at T1 (weeks)−0.477.02-0.522.02−0.459.005
    Income
    		.70




    		No response1.00





    		Below Average−0.033





    		Average2.921





    		Above Average0.061





    		Higher income1.181




    Marital status
    		.59




    		Relationship1.00





    		Single2.208




    Employment status
    		.14
    		.08


    		Working or studying1.00
    		1.00



    		Not in labor force−3.189
    		−2.927


    Country of birth
    		.31




    		Other1.00





    		Australia−2.389





    		New Zealand−0.074





    		United Kingdom6.9.41




    Device type
    		.73




    		iOS1.00





    		Android0.580




    Birth weight (grams)0.002.200.001.42

    Gender
    		.34




    		Male1.00
    		1.00



    		Female−1.462
    		−0.440.77

    Baseline feeding status
    		.13
    		.17


    		Mixed feeding1.00
    		1.00



    		Breastfeeding−0.401
    		0.524



    		Formula feeding3.124
    		3.941


    The loyalty index (Li) average score was 50.8% (IQR: 26.7%-75.7%). The average number of sessions participants visited the app was 11.6 times (range 1-64) and a median of 9. The recency index (Ri) median score was 34.4% (IQR: 10.7%-37.3%). On average participants took 14 days to activate the app (range 0-184 days). The interaction index (Ii) median score was 8.9% (IQR: 1.9%-18.1%). On average, 91.8 (range: 16-216) push notifications were sent and an average of 11.1 (range: 0-70) were opened with a median of 6. Participants who used both the app (including access to push notifications) and opened weekly emails scored lower on the Ii compared with participants who only used the app and only accessed push notifications.

    The feedback index (Fi) was calculated for 154 participants as 71 participants either did not complete the 9-month survey, or reported using the website (n=15) and were not asked for feedback about the app. The median score for Fi was 2.7 (IQR: 0-16.2). As presented in Table 4 the app features participants were most satisfied with included the language used, usefulness in sharing the app with another carer, and the quantity of Internet data required to use the app. Participants were least satisfied with the push notifications, including the number of push notifications sent (too few or too many), and many participants experienced technical problems using them. There was a low satisfaction with respect to the videos available on the app which they felt did not cover sufficient information to answer their queries about infant feeding.

    Over the duration of the program, there was a decrease in the mean index score for each subindex. The Ci and Li scores shared similar scores during the initial (0-3 months) and final (6-9 months) period, whereas for the interim period (3-6 months) the mean score was lower for Ci (43.7%) compared with Li (54.6%). The recency index dropped dramatically after the initial period by 55.4% and continued to track down, whereas the interaction index attained the lowest mean compared with the other subindices at the initial period (21.4%) and trended down over time (See Figure 4).

    Table 3. Descriptive statistics of each subindex (N %).
    SubindexMeanMedianInterquartile rangeRange
    Click depth index46.745.533.3-63.30-100
    Loyalty index50.850.826.7-75.70-93.4
    Recency index26.034.410.7-37.30.6-53.7
    Interaction index12.78.91.9-18.10-64.3
    Feedback index13.32.70-16.20-94.6
    ‎
    Figure 3. Number of participants and total number of times participants visited each section of the Growing healthy app. BF=breastfeeding, FF=formula feeding, MF=mixed feeding.
    View this figure
    Table 4. Participants’ reported satisfaction with aspects of the Growing healthy program (feedback index, Fi; n=154).
    Satisfaction questionnaireScores (N)a
    I found the Growing healthy app easy to use46
    I liked the layout or “look” of the app34
    I found it hard to navigate through the appb23
    The Growing healthy app didn’t take long to load information45
    The Growing healthy app failed to work at timesb28
    The different sections of the app worked well together20
    The language used in the app was easy to understand57
    The app did everything I expected it to do31
    I couldn’t find all of the answers I needed in the appb11
    I had to use the search feature to find what I was looking for14
    Using the app was an enjoyable experience22
    I found the app complicatedb43
    I can trust the information on the Growing healthy app39
    I felt confident using this app40
    I found the information for mums useful31
    I found the information on feed and sleep patterns useful29
    I found the information about breastfeeding useful20
    I found the information about formula feeding useful17
    I found the information on mixed feeding useful15
    I found the information on solid feeding useful27
    I found the videos on the app useful12
    I found the recipe section of the app useful22
    I shared the information from the app with other friends and family16
    I was concerned about the Internet data usage on my phone when using the appb47
    I found the information provided easy to understand36
    Overall, I liked the Growing healthy program36
    I would recommend the Growing healthy program to a friend45
    I found it helpful to share the app with my partner or another carer48
    The Growing healthy program covered all of the things about infant feeding that I wanted it to25
    I received push notifications on my phone, from the Growing healthy programc122
    The push notification messages often disappeared before I had a chance to tap on themb12
    I didn’t know how to retrieve push notification messages once they disappeared from screenb12
    I would prefer to receive text messages rather than push notifications from the app19
    I was happy with the number of notifications or messages received each week6
    I was happy with the time that the notification was sent to me during the day18
    I found the notifications or messages helpful16
    I found the notifications or messages suited my baby’s age and stage of development23

    aTotal scores only include the extreme positive responses based on scoring criteria .

    bLikert scale scoring reversed for these questions: strongly disagree (1), disagree (0), no strong feelings either way (0), agree (0), strongly agree (1), and didn’t use (0).

    cResponse option and scoring: Yes, I received weekly push notifications (1), no, I received text messages instead of push notifications (1), and no, I disabled my push notifications so I didn\'t receive any weekly messages (0).

    ‎
    Figure 4. The frequency of scores for click-depth index (Ci), loyalty index (Li), interaction index (Ii), and recency index (Ri) at each time point (initial, interim, and final).
    View this figure

    Principal Findings

    This is one of the first studies to develop and implement an mHealth program supporting parents with healthy infant feeding practices through a mobile phone app. To our knowledge, this is the first study to utilize an EI to quantify and categorize participants’ engagement level using the app. We found that engagement level was positively correlated with primiparous status, use of both the app and email, exposure to the program for a longer period, and recruitment through health practitioners. Negative correlation was found with age of child at start of program and engagement level.

    The identification of the correlates of participant engagement is not only beneficial to inform future enhancements of the Growing healthy program, but more broadly to evaluate mHealth programs. The EI has its origins in measurement of consumer engagement with Web-based products. Adjusting the index to measure engagement with a mHealth program was possible as the metrics measured are the same; only the measurement of the content and behavior will be different [16].

    A criterion to categorize participants as poor, moderate, or highly engaged with the Growing healthy program based on their overall EI score was developed. Previously, program engagement has arbitrarily been labeled as high [27] or low [22] based on the frequency participants accessed websites or apps. Few studies have considered participant engagement on the basis of their interaction with multiple intervention elements. In addition, there is not a standardized approach to measuring engagement. For example, a point system to gauge individual user activity with the program features was used in a study targeting reduction of high-risk sexual behaviors. The measures included were frequency of access, profile modification, message views, article views, completion of quizzes, number of pages viewed, and updates of personal goals [23]. Whereas modeling participants’ engagement with frequency of access, average daily steps, and the number of days since participants last accessed the program was performed for a physical activity focused mHealth intervention [20].

    Participant app use over the 9-month period in this study varied such that engagement was high after initially joining the program but decreased from the 3- to 6-month period. Previous mHealth programs targeting long and short term behavior change have identified similar patterns of use [27-29]. Attrition with mHealth programs is negatively affected by factors such as lack of commitment or motivation to change health behaviors [28], confidence in knowledge about managing the targeted behavior [27], and programs that are perceived as overly burdensome by participants [23]. The Growing healthy program was a “just in time” resource [15] developed to provide infant feeding information up to 9 months of the infants age. Feeding milestones targeted included breastfeeding, best practice formula feeding, timing of the introduction of solids and optimizing dietary exposure to fruit and vegetables, and minimizing exposure to noncore foods. Once that knowledge is obtained, it is likely that participant app use will drop off [30]. When targeting long term behavior change, mHealth developers need to consider ongoing novel strategies that will keep participants engaged. Qualitative findings suggest users prefer to engage with apps periodically [28]. Such findings highlight that we must seek to understand app users’ behavior to inform the most appropriate time to engage them to join the program, the factors that lead to disengagement, and to consider strategies that will maintain their engagement.

    Study participants who accessed both the website and the app attained a significantly higher EI score compared with participants who had just used the app. This supports the notion that delivering the intervention using various modes enhances engagement and to the intervention exposure [11,23]. Primiparous participants had significantly higher EI scores than multiparous women. This is congruent with qualitative analysis conducted as part of the development phase for the Growing healthy program (unpublished) where most of the primiparous participants expressed an interest in the program, while multiparous participants suggested the resource would have been more useful as a first time parent. Despite multiparous participants being less engaged, more than one third of those classified as highly engaged were indeed multiparous.

    While initial engagement is the initial hurdle for any intervention, sustaining engagement remains the most difficult part of intervention implementation, it is more difficult to achieve [11,23,31]. It has been found that novelty and relevance are main contributors to sustained app user engagement [11,32,33]. The downward trend in engagement subindices scores from 3 to 6 months reported in this study may be a reflection of a lack of perceived novelty in the Growing healthy app throughout the intervention period. This may also explain the lower engagement of multiparous participants (who have developed their thinking around infant feeding already).

    The infant’s age at baseline (ie, when the app was downloaded) was also strongly associated with higher EI scores. Participants who joined the program when their infant was younger had a higher EI score compared with those who joined when their infant’s age was closer to 3 months. Similar to traditional interventions that targeted childhood obesity prevention [34], early recruitment was necessary to increase participant engagement. Early recruitment is likely to increase intervention exposure, which is associated with an increased likelihood of influencing the uptake of the desired behaviors [27]. This is important to target as infant development clearly occurs rapidly within the first year of life. The app was likely to be most useful and provided novel information to mothers if they were recruited from early postpartum or during pregnancy.

    Participants who were recruited from their health practitioner were more likely to have higher EI scores compared with those who were recruited on the Web. This may be attributed to mothers’ perception that health practitioners are a trustworthy source of information [35]. The involvement of health practitioners such as maternal and child health nurses and practice nurses who do routine infant health checks during the first few years of life [36], are important as a key “referral pathway” to evidence based apps and in turn, to the most effective utilization of apps.

    Comparison With Prior Work

    Several studies describing mHealth interventions encouraging healthy infant feeding behaviors have recently been published. Delivery modes used in these studies included app [37,38], websites [39,40], and social media [41]. Due to the different delivery modes, the findings of this study cannot be compared with other programs. However, as mHealth interventions are novel modes of delivering health behavior change interventions across health disciplines, similar patterns of engagement have been reported by several researchers albeit using different measures [23,42,43].

    Limitations and Strengths

    This study has several limitations. First, a number of technological issues were experienced by participants in receiving and opening push notifications. Adaptation were therefore made midway during the program and all participants were sent weekly emails. Second, app quality is an important influencer on participant engagement [44-46]. The participants’ responses to the satisfaction survey (feedback index) demonstrated low satisfaction with respect to the push notifications, emphasizing the impact technological difficulties have on participant engagement. Third, the weekly emails contained links to the Growing healthy website rather than the app. Finally, participant behavior on the website, such as the number of pages viewed, was not accessible at an individual level. This explains the increase in loyalty index scores at around 3 to 6 months, as participant access to email links was included. Click depth index scores decreased at that time point because the number of pages viewed on the website could not be measured. Overall, the EI score calculated for these participants is most likely an underestimate of their engagement with the program.

    Some features of the Growing healthy program were not measured using the EI because there were difficulties in obtaining individual participants’ information such as, participant use of the Growing healthy Facebook group and sharing the app with another carer or sharing information from the app with others (interconnectivity). Although participant interaction with these features was not measured, satisfaction and use of these features was included in the 9-month survey that made up the feedback index.

    Some studies have shown that mothers from a disadvantaged background were less likely to use the Internet as a source of information for infant feeding [47]. A strength of our study was that approximately equal number of participants of both high and low educational background were recruited unlike other mHealth programs targeted at addressing infant feeding [48].

    To our knowledge, the utilization of an index to measure participant engagement has not yet been implemented in mHealth interventions. The EI provided detailed analysis regarding the frequency participants accessed the app and push notifications, how many pages they accessed per session, and their satisfaction with the program which was measured over 3 time points across the 9 months of the program.

    Conclusions

    The EI provided a comprehensive understanding of participant behavior with the app over the 9-month period of the Growing healthy program. The participants’ engagement with the Growing healthy app was determined by various factors including participant characteristics, novelty, intervention exposure time, and the quality of the app including technological aspects. Primiparous participants, those who accessed both the emails and the app, those who were exposed to the program for a longer period, and those who were recruited from their health practitioner all had higher EI scores. The use of the EI in this study demonstrates that rich and useful data can be collected and used to assess the strengths and weaknesses of mHealth interventions and in turn inform improvements in their design and delivery.

    Acknowledgments

    The research reported in this paper is a project of the Australian Primary Health Care Research Institute, which is supported by a grant from the Australian government Department of Health and Ageing. The information and opinions contained in it do not necessarily reflect the views or policies of the Australian Primary Health Care Research Institute or the Australian government Department of Health and Ageing.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    Participant satisfaction survey.

    JPG File, 376KB
    1. Australian Communication and Media Authority. 2015. Australians get mobile   URL: http://www.acma.gov.au/theACMA/engage-blogs/engage-blogs/Research-snapshots/Australians-get-mobile [accessed 2016-12-27] [WebCite Cache]
    2. International Data Corporation. 2015. Worldwide mobile phone semiconductor 2015-2019 forecast   URL: https://www.idc.com/getdoc.jsp?containerId=256341 [accessed 2016-12-27] [WebCite Cache]
    3. Li Y, Wang W, van Velthoven MH, Chen L, Car J, Rudan I, et al. Text messaging data collection for monitoring an infant feeding intervention program in rural China: feasibility study. J Med Internet Res 2013 Dec 04;15(12):e269 [FREE Full text] [CrossRef] [Medline]
    4. Turner T. Prevention and treatment of pediatric obesity using mobile and wireless technologies: a systematic review. Pediatr Obes 2015;10(6):403-409. [CrossRef]
    5. Denney-Wilson E, Laws R, Russell CG, Ong K, Taki S, Elliot R, et al. Preventing obesity in infants: the growing healthy feasibility trial protocol. BMJ Open 2015 Nov 30;5(11):e009258 [FREE Full text] [CrossRef] [Medline]
    6. Lalmas M, O'Brien H, Yom-Tov E. In: Marchionini G, editor. Measuring User Engagement: Synthesis Lectures on Information Concepts, Retrieval, and Services. North Carolina: Morgan & Claypool Publishers; Nov 2014:1-132.
    7. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2016 Dec 13:- Epub ahead of print. [CrossRef] [Medline]
    8. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008 Sep 29;337:a1655 [FREE Full text] [Medline]
    9. Zhao J, Freeman B, Li M. Can mobile phone apps influence people's health behavior change? an evidence review. J Med Internet Res 2016 Oct 31;18(11):e287 [FREE Full text] [CrossRef] [Medline]
    10. Wiebe E, Lamb A, Hardy M, Sharek D. Measuring engagement in video game-based environments: investigation of the user engagement scale. Comput Human Behav 2014 Mar;32:123-132. [CrossRef]
    11. O'Brien H, Toms EG. What is user engagement? a conceptual framework for defining user engagement with technology. J Am Soc Inf Sci 2008 Apr;59(6):938-955. [CrossRef]
    12. Bickmore T, Schulman D, Yin L. Maintaining engagement in long-term interventions with relational agents. Appl Artif Intell 2010 Jul 01;24(6):648-666 [FREE Full text] [CrossRef] [Medline]
    13. Burby J. Digital Analytics Association. Washington DC: Web Analytics Association; 2007. Web analytics definitions   URL: https://www.digitalanalyticsassociation.org/Files/PDF_standards/WebAnalyticsDefinitionsVol1.pdf [accessed 2017-05-30] [WebCite Cache]
    14. Crutzen R, de NJ, Brouwer W, Oenema A, Brug J, de Vries Nanne K. A conceptual framework for understanding and improving adolescents' exposure to internet-delivered interventions. Health Promot Int 2009 Sep;24(3):277-284. [CrossRef] [Medline]
    15. Danaher BG, Brendryen H, Seeley JR, Tyler MS, Woolley T. From black box to toolbox: outlining device functionality, engagement activities, and the pervasive information architecture of mHealth interventions. Internet Interv 2015 Mar 01;2(1):91-101 [FREE Full text] [CrossRef] [Medline]
    16. Peterson E, Carrabis J. Verticalstudio. Camas, WA: Web Analytics Demystified; 2008. Measuring the immeasurable: visitor engagement   URL: http:/​/www.​verticalstudio.com/​hs-fs/​hub/​74398/​file-15425996-pdf/​docs/​web_analytics_demystified_and_nextstage_global_-_measuring_the_immeasurable_-_visitor_engagement.​pdf [accessed 2017-05-30] [WebCite Cache]
    17. Schadler T, McCarthy J. Forrester. 2012. Mobile is the new face of engagement   URL: https://www.forrester.com/report/Mobile+Is+The+New+Face+Of+Engagement/-/E-RES60544 [accessed 2016-12-28] [WebCite Cache]
    18. Couper MP, Alexander GL, Zhang N, Little RJ, Maddy N, Nowak MA, et al. Engagement and retention: measuring breadth and depth of participant use of an online intervention. J Med Internet Res 2010 Nov 18;12(4):e52 [FREE Full text] [CrossRef] [Medline]
    19. Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al. Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med 2016 Nov;51(5):833-842. [CrossRef] [Medline]
    20. Davies C, Corry K, Van IA, Vandelanotte C, Caperchione C, Mummery WK. Prospective associations between intervention components and website engagement in a publicly available physical activity website: the case of 10,000 Steps Australia. J Med Internet Res 2012 Jan 11;14(1):e4 [FREE Full text] [CrossRef] [Medline]
    21. Arden-Close E, Smith E, Bradbury K, Morrison L, Dennison L, Michaelides D, et al. A visualization tool to analyse usage of web-based interventions: the example of positive online weight reduction (POWeR). JMIR Hum Factors 2015 May 19;2(1):e8 [FREE Full text] [CrossRef] [Medline]
    22. Partridge S, McGeechan K, Hebden L, Balestracci K, Wong AT, Denney-Wilson E, et al. Effectiveness of a mhealth lifestyle program with telephone support (TXT2BFiT) to prevent unhealthy weight gain in young adults: randomized controlled trial. JMIR Mhealth Uhealth 2015 Jun 15;3(2):e66 [FREE Full text] [CrossRef] [Medline]
    23. Baltierra NB, Muessig KE, Pike EC, LeGrand S, Bull SS, Hightow-Weidman LB. More than just tracking time: Complex measures of user engagement with an internet-based health promotion intervention. J Biomed Inform 2016 Feb;59:299-307 [FREE Full text] [CrossRef] [Medline]
    24. Michie S, Atkins L, West R. The Behaviour Change Wheel: A Guide to Designing Interventions. Great Britain: Silverback Publishing; 2014.
    25. Laws R. A comparison of recruitment methods for an mhealth intervention targeting mothers: lessons from the growing healthy program. J Med Internet Res 2016;18(9):e248. [CrossRef]
    26. Hosmer JD, Lemeshow S. Applied Logistic Regression. Canada: John Wiley & Sons; 2004.
    27. Guertler D, Vandelanotte C, Kirwan M, Duncan MJ. Engagement and nonusage attrition with a free physical activity promotion program: the case of 10,000 steps Australia. J Med Internet Res 2015 Jul 15;17(7):e176 [FREE Full text] [CrossRef] [Medline]
    28. Dennison L, Morrison L, Conway G, Yardley L. Opportunities and challenges for smartphone applications in supporting health behavior change: qualitative study. J Med Internet Res 2013 Apr 18;15(4):e86 [FREE Full text] [CrossRef] [Medline]
    29. Neve MJ, Collins CE, Morgan PJ. Dropout, nonusage attrition, and pretreatment predictors of nonusage attrition in a commercial web-based weight loss program. J Med Internet Res 2010 Dec 14;12(4):e69 [FREE Full text] [CrossRef] [Medline]
    30. National Health and Medical Research Council. Infant Feeding Guidelines. Canberra: National Health and Medical Research Council; 2012.
    31. Morgan P, Young MD, Smith JJ, Lubans DR. Targeted health behavior interventions promoting physical activity: a conceptual model. Exerc Sport Sci Rev 2016 Apr;44(2):71-80. [CrossRef] [Medline]
    32. Aboulafia A, Bannon LJ. Understanding affect in design: an outline conceptual framework. TIES 2004;5(1):4-15.
    33. Webb T, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4 [FREE Full text] [CrossRef] [Medline]
    34. Askie LM, Baur LA, Campbell K, Daniels LA, Hesketh K, Magarey A, et al. The early prevention of obesity in children (EPOCH) collaboration--an individual patient data prospective meta-analysis. BMC Public Health 2010 Nov 25;10:728 [FREE Full text] [CrossRef] [Medline]
    35. Gildea A, Sloan S, Stewart M. Sources of feeding advice in the first year of life: who do parents value? Community Pract 2009 Mar;82(3):27-31. [Medline]
    36. Goldfeld SR, Wright M, Oberklaid F. Parents, infants and health care: utilization of health services in the first 12 months of life. J Paediatr Child Health 2003;39(4):249-253. [Medline]
    37. White B, Martin A, White JA, Burns SK, Maycock BR, Giglia RC, et al. Theory-based design and development of a socially connected, gamified mobile app for men about breastfeeding (milk man). JMIR Mhealth Uhealth 2016 Jun 27;4(2):e81 [FREE Full text] [CrossRef] [Medline]
    38. Wen L. Obesity in young children: what can we do about? J Obes Weight Loss Ther 2015;05(06):286. [CrossRef]
    39. Uesugi K, Dattilo AM, Black MM, Saavedra JM. Design of a digital-based, multicomponent nutrition guidance system for prevention of early childhood obesity. J Obes 2016;2016:5067421 [FREE Full text] [CrossRef] [Medline]
    40. Campbell K, Hesketh KD, McNaughton SA, Ball K, McCallum Z, Lynch J, et al. The extended infant feeding, activity and nutrition trial (InFANT Extend) program: a cluster-randomized controlled trial of an early intervention to prevent childhood obesity. BMC Public Health 2016 Feb 18;16:166 [FREE Full text] [CrossRef] [Medline]
    41. Horodynski M, Silk K, Hsieh G, Hoffman A, Robson M. Tools for teen moms to reduce infant obesity: a randomized clinical trial. BMC Public Health 2015 Jan 21;15:22 [FREE Full text] [CrossRef] [Medline]
    42. O'Malley G, Dowdall G, Burls A, Perry IJ, Curran N. Exploring the usability of a mobile app for adolescent obesity management. JMIR Mhealth Uhealth 2014 Jun 27;2(2):e29 [FREE Full text] [CrossRef] [Medline]
    43. Allison S, Bauermeister JA, Bull S, Lightfoot M, Mustanski B, Shegog R, et al. The intersection of youth, technology, and new media with sexual health: moving the research agenda forward. J Adolesc Health 2012 Sep;51(3):207-212 [FREE Full text] [CrossRef] [Medline]
    44. Eysenbach G, Powell J, Kuss O, Sa E. Empirical studies assessing the quality of health information for consumers on the world wide web. JAMA 2002 May 22;287(20):2691-2700. [CrossRef]
    45. Taki S, Campbell KJ, Russell CG, Elliott R, Laws R, Denney-Wilson E. Infant feeding websites and apps: a systematic assessment of quality and content. Interact J Med Res 2015 Sep 29;4(3):e18 [FREE Full text] [CrossRef] [Medline]
    46. QUT. 2014. Mobile application rating scale (MARS): a new tool for assessing the quality of health mobile applications   URL: http://eprints.qut.edu.au/71733/5/71733.pdf [accessed 2017-05-30] [WebCite Cache]
    47. Gage H, Williams P, Von Rosen-Von H, Laitinen K, Jakobik V, Martin-Bautista E, et al. Influences on infant feeding decisions of first-time mothers in five European countries. Eur J Clin Nutr 2012 Aug;66(8):914-919. [CrossRef] [Medline]
    48. Gazmararian JA, Yang B, Elon L, Graham M, Parker R. Successful enrollment in Text4Baby more likely with higher health literacy. J Health Commun 2012;17(Suppl 3):303-311. [CrossRef] [Medline]

    ‎
    Ci: Click depth index
    EI: engagement index
    Fi: feedback index
    Ii: interaction index
    Li: loyalty index
    Ri: recency index
    IQR: interquartile range

    Edited by G Eysenbach; submitted 27.12.16; peer-reviewed by J Scott, S Baptista, M Ferguson; comments to author 26.01.17; revised version received 05.03.17; accepted 23.03.17; published 29.06.17

    Copyright

    ©Sarah Taki, Sharyn Lymer, Catherine Georgina Russell, Karen Campbell, Rachel Laws, Kok-Leong Ong, Rosalind Elliott, Elizabeth Denney-Wilson. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 29.06.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.