This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Childhood obesity is an ongoing problem in developed countries that needs targeted prevention in the youngest age groups. Children in socioeconomically disadvantaged families are most at risk. Mobile health (mHealth) interventions offer a potential route to target these families because of its relatively low cost and high reach. The Growing healthy program was developed to provide evidence-based information on infant feeding from birth to 9 months via app or website. Understanding user engagement with these media is vital to developing successful interventions. Engagement is a complex, multifactorial concept that needs to move beyond simple metrics.
The aim of our study was to describe the development of an engagement index (EI) to monitor participant interaction with the Growing healthy app. The index included a number of subindices and cut-points to categorize engagement.
The Growing program was a feasibility study in which 300 mother-infant dyads were provided with an app which included 3 push notifications that was sent each week. Growing healthy participants completed surveys at 3 time points: baseline (T1) (infant age ≤3 months), infant aged 6 months (T2), and infant aged 9 months (T3). In addition, app usage data were captured from the app. The EI was adapted from the Web Analytics Demystified visitor EI. Our EI included 5 subindices: (1) click depth, (2) loyalty, (3) interaction, (4) recency, and (5) feedback. The overall EI summarized the subindices from date of registration through to 39 weeks (9 months) from the infant’s date of birth.
Basic descriptive data analysis was performed on the metrics and components of the EI as well as the final EI score. Group comparisons used
The overall EI mean score was 30.0% (SD 11.5%) with a range of 1.8% - 57.6%. The cut-points used for high engagement were scores greater than 37.1% and for poor engagement were scores less than 21.1%. Significant explanatory variables of the EI score included: parity (
The EI provided a comprehensive understanding of participant behavior with the app over the 9-month period of the Growing healthy program. The use of the EI in this study demonstrates that rich and useful data can be collected and used to inform assessments of the strengths and weaknesses of the app and in turn inform future interventions.
Mobile phone ownership is widespread in Australia and internationally [
Capturing the attention of an app user is clearly paramount to the app’s potential effectiveness for behavior change. To be successful, apps must continuously and actively engage the user. User engagement refers to the quality of the user experience, the positive aspects of their interaction, and their desire to use the app over longer periods of time or repeatedly [
Engagement with technology is inherently complex and multifaceted in its nature and it may be mediated by factors such as family, community, culture, and context [
Large scale quantitative measures of engagement rely on Web analytics which provide the opportunity to measure behavioral aspects of engagement. Some examples of data that can be collected, but is not exhaustive to, includes frequency of access to the app, page views, push notifications opened, and average time spent on a page [
It has been suggested that Web analytics measures can be classed into three main dimensions of engagement: popularity, activity, and loyalty [
Little work has been done in the mHealth arena with respect to the conceptualization and measurement of user engagement [
The Growing healthy program utilized a quasi-experimental design aimed to support parents of young infants with healthy infant feeding behaviors. To enhance intervention effectiveness, the behavior change wheel model [
Eligible participants were offered to use the Growing healthy app and could choose to receive 3 tailored push notifications through the app each week of the intervention (9 months of the baby’s age). Although, midway through the intervention implementation period, participants were also sent a weekly email due to identifying technological issues with receiving and opening push notifications. The weekly emails included the same messages as the push notifications sent each week. Participants who did not own a phone that was compatible with the app were offered access to the Growing healthy website and were sent 3 text messages. Details of the study has been published previously [
Study participants completed 3 quantitative surveys: (1) baseline (T1) (infant age ≤3 months), (2) infant aged 6 months (T2), and (3) infant aged 9 months (T3). The surveys included demographic and infant feeding behavior questions. Participants’ use of the app was captured and the data was used to develop the EI and evaluate the Growing healthy program.
The Web Analytics Demystified visitor EI [
Metrics needed to calculate the subindices (outlined in
The definitions of the subindices for the engagement index designed for the Growing healthy program where i=ith person and j=jth time period and n=3 for Ci, Li, Ii, and Ri (sum of calculation period) and n=37 for Fi.
Equal weight for each of the subindices was assigned to the overall EI score so that each element was equally important in contributing to the measurement of engagement. Four of the subindices were calculated using app data. The feedback index was informed using responses to the 9-month survey (T3) feedback questions. The final formula used to calculate the EI incorporated click depth, loyalty, recency, interaction, and feedback subindices (see equation 1). The EI was then converted to a value between 0 and 100.
Equation 1: Engagement index formula
where EI is engagement index, Ci is click depth index, Li is loyalty index, Ii is interaction index, Ri is recency index, and Fi is feedback index.
The calculation for each subindex except the feedback index (data were only collected at the end of the program) was done for three time periods, including initial (0-3 months), interim (3-6 months), and final (6-9 months) and were then averaged. This grouping of time periods was chosen because there was an initial intense use of the app followed by infrequent participant use toward the end of the 9-month program. A detailed explanation for the calculation of each subindex follows.
The number of pages a participant viewed the app in each access session over the total number of sessions in each time period formed the basis of this subindex. Two metrics were used in the calculation of Ci: the number of sessions in the time period and number of pages viewed per session. A threshold of the number of pages viewed per session was applied. There is no benchmark of an effective click depth, that is, “dose” of the interaction in the mHealth environ. Based on the data collected, the median value of 2 pages per session was used as the threshold. The overall score of Ci was the average of each time period calculation: Ci1, Ci2, and Ci3.
This subindex was based on the frequency of app access throughout the 9-month program. Li was the reciprocal of the number of sessions in each time period. The total score was dependent on when participants activated the app. The overall score of Li was the average of each time period calculation: Li1, Li2, and Li3.
The number of push notifications opened versus total sent throughout the 9-month program formed the basis of this subindex. Interaction Index was the total number of push notifications opened divided by the number sent in the time period. This was calculated for 3 month time intervals of the infant’s age according to when the participant activated the app until the infant reached 9 months of age. The overall score of Ii is the average of each time period calculation: Ii1, Ii2, and Ii3.
The number of days between each session was the basis of the recency index. The Ri was calculated for three different time points: (1) the number of days elapsed from registration to when the participant first accessed the app (Ri1), (2) the average number of days between sessions when the participant accessed the app between 3 to 6 months (Ri2), and (3) 6 and 9 months (Ri3). The data were transformed by taking the reciprocal of each Ri1 to Ri3. The overall score of Ri was the average of each time period calculation: Ri1, Ri2, and Ri3.
This subindex was a self-reported measure of participant satisfaction with the app, which was captured in the 9-month survey (T3). Constructive feedback was scored positively as 1 and negatively as 0. The 9-month survey included 37 questions which formed the basis of Fi. Each question (
Basic descriptive data analysis was performed on the metrics and components of the EI as well as the final EI score. To analyze the EI scores, cut-off points were developed based on the distribution of the total samples’ EI scores using quartiles. Participants were then categorized as either poorly, moderately, or highly engaged. This method was chosen as there were no existing mHealth interventions that utilized an EI and categorized participants’ engagement based on app use.
Group comparisons between poorly, moderately, or highly engaged participants were then conducted using
The following variables were dichotomized for analysis including:
Education level: university degree (“degree” or “higher degree”) or no university (“high school education or less,” “trade certificate,” or “diploma”)
Employment status: working or studying (“full or part-time,” “casual paid work,” and “full or part-time studying”) or not in labor force (“keeping house and/or raising children full-time” and “unemployed or laid off”)
Gross household income: below average (“Aus $1-$119 per week,” “Aus $120-$299 per week,” “Aus $300-$599 per week,” “Aus $600-$799 per week,” “Aus $800-$999 per week”) average (“Aus $1000-$1499 per week”), above average (“Aus $1500-1999 per week”), or higher income (“Aus $2000 or more per week”)
Marital status: relationship (“married,” “living in a defacto relationship”) or single (“separated,” “divorced,” “widowed,” “never married”)
Recruitment method: practitioner, Web-based, or family or friends
Device type: android or iOS
System type: app only or both app and email
Other independent variables considered included mother’s age, country of birth, as well as infant’s age at the start of the program, their birth weight, and feeding status at baseline. All analyses were performed using used IBM SPSS Version 23.0.
Of the 300 Growing healthy participants who completed the baseline survey, 75.0% (225/300) met the inclusion criteria for this study. The average age of participants was 30 years, with 62.2% (186/300) being first time parents, 97.0 % (291/300) living with their partner, and 84.0% (252/300) being full-time carers of the infant. The infants’ were on average 6.9 weeks old when registration occurred and 56.4 %( 169/300) were breastfed.
The EI score had a distribution that was not statistically significant as evidenced by nonsignificant Kolomogorov Smirnov (KS) test at
Three variables were significantly associated with high engagement in univariate analysis (see
Overall engagement index scores distribution.
Characteristics of growing healthy participants based on engagement index level (n=255).
Variables | Poor engagement (n=56) |
Moderate engagement (n=113) |
High engagement (n=56) |
||
Age (years)a, mean (SD) | 30.3 (4.4) | 30.5 (4.4) | 30.6 (4.5) | .61 | |
Education (no university)b, n (%) | 31 (55) | 61 (54) | 20 (36) | .11 | |
Income (higher income)c, n (%) | 14 (25) | 36 (31.8) | 13 (23.2) | .70 | |
Marital status (relationship)b, n (%) | 54 (96) | 110 (97.3) | 53 (95) | .59 | |
Employment status (not in labor force)b, n (%) | 51 (91) | 97 (85.8) | 45 (80) | .14 | |
Parity (Primiparous)b, n (%) | 29 (52) | 70 (61.9) | 41 (73) | .004d | |
Recruitment method (Practitioner)c, n (%) | 24 (48) | 52 (47.9) | 30 (48) | .06 | |
Device type (iOS)b, n (%) | 38 (68) | 87 (72.5) | 36 (64) | .73 | |
System type (both app & email users)b, n (%) | 26 (46) | 82 (72.5) | 52 (93) | <.001d | |
Age at registration (weeks)a, mean (SD) | 7.3 (3.6) | 7.4 (3.6) | 5.6 (3.4) | .02d | |
Birth weight (kg)a, mean (SD) | 3.46 (0.591) | 3.47 (0.593) | 3.47 (0.592) | .20 | |
Gender (male)b, n (%) | 31 (55.3) | 53 (46.9) | 23 (41.0) | .34 | |
Breastfeeding | 35 (63) | 64 (56.6) | 28 (50) | .13 | |
Formula feeding | 14 (25) | 25 (22.1) | 23 (41) | ||
Mixed feeding | 7 (12.5) | 24 (21.2) | 5 (8.9) |
aPearson correlation; mean, standard deviation (SD) reported.
b
cBased on ANOVA; % within group (count) reported.
dStatistically significant engagement level and independent variable <.05.
Of the 14 variables assessed in this study, 8 met the including criterion of
To better understand the drivers of engagement descriptive analysis of the subindices that made up the overall EI score was performed (
Linear regression to explore the predictors of infant and participant characteristics with the engagement index scores.
Variable | Univariate model (B) | Full model (B) | Reduced model (B) | ||||
R2 | 0.154 | 0.164 | |||||
.004 | .006 | .005 | |||||
Multiparous | 1.00 | 1.00 | 1.00 | ||||
Primiparous | 4.532 | 4.147 | 4.209 | ||||
.06 | .07 | .02 | |||||
Family or friends | 1.00 | 1.00 | 1.00 | ||||
Practitioner | 5.346 | 6.423 | 4.221 | ||||
Web-based | 2.795 | 4.267 | 0.989 | ||||
<.001 | <.001 | <.001 | |||||
App only | 1.00 | 1.00 | 1.00 | ||||
Both (app and email) | 7.977 | −6.426 | −6.937 | ||||
Infant age at T1 (weeks) | −0.477 | .02 | -0.522 | .02 | −0.459 | .005 | |
.70 | |||||||
No response | 1.00 | ||||||
Below Average | −0.033 | ||||||
Average | 2.921 | ||||||
Above Average | 0.061 | ||||||
Higher income | 1.181 | ||||||
.59 | |||||||
Relationship | 1.00 | ||||||
Single | 2.208 | ||||||
.14 | .08 | ||||||
Working or studying | 1.00 | 1.00 | |||||
Not in labor force | −3.189 | −2.927 | |||||
.31 | |||||||
Other | 1.00 | ||||||
Australia | −2.389 | ||||||
New Zealand | −0.074 | ||||||
United Kingdom | 6.9.41 | ||||||
.73 | |||||||
iOS | 1.00 | ||||||
Android | 0.580 | ||||||
Birth weight (grams) | 0.002 | .20 | 0.001 | .42 | |||
.34 | |||||||
Male | 1.00 | 1.00 | |||||
Female | −1.462 | −0.440 | .77 | ||||
.13 | .17 | ||||||
Mixed feeding | 1.00 | 1.00 | |||||
Breastfeeding | −0.401 | 0.524 | |||||
Formula feeding | 3.124 | 3.941 |
The loyalty index (Li) average score was 50.8% (IQR: 26.7%-75.7%). The average number of sessions participants visited the app was 11.6 times (range 1-64) and a median of 9. The recency index (Ri) median score was 34.4% (IQR: 10.7%-37.3%). On average participants took 14 days to activate the app (range 0-184 days). The interaction index (Ii) median score was 8.9% (IQR: 1.9%-18.1%). On average, 91.8 (range: 16-216) push notifications were sent and an average of 11.1 (range: 0-70) were opened with a median of 6. Participants who used both the app (including access to push notifications) and opened weekly emails scored lower on the Ii compared with participants who only used the app and only accessed push notifications.
The feedback index (Fi) was calculated for 154 participants as 71 participants either did not complete the 9-month survey, or reported using the website (n=15) and were not asked for feedback about the app. The median score for Fi was 2.7 (IQR: 0-16.2). As presented in
Over the duration of the program, there was a decrease in the mean index score for each subindex. The Ci and Li scores shared similar scores during the initial (0-3 months) and final (6-9 months) period, whereas for the interim period (3-6 months) the mean score was lower for Ci (43.7%) compared with Li (54.6%). The recency index dropped dramatically after the initial period by 55.4% and continued to track down, whereas the interaction index attained the lowest mean compared with the other subindices at the initial period (21.4%) and trended down over time (See
Descriptive statistics of each subindex (N %).
Subindex | Mean | Median | Interquartile range | Range |
Click depth index | 46.7 | 45.5 | 33.3-63.3 | 0-100 |
Loyalty index | 50.8 | 50.8 | 26.7-75.7 | 0-93.4 |
Recency index | 26.0 | 34.4 | 10.7-37.3 | 0.6-53.7 |
Interaction index | 12.7 | 8.9 | 1.9-18.1 | 0-64.3 |
Feedback index | 13.3 | 2.7 | 0-16.2 | 0-94.6 |
Number of participants and total number of times participants visited each section of the Growing healthy app. BF=breastfeeding, FF=formula feeding, MF=mixed feeding.
Participants’ reported satisfaction with aspects of the Growing healthy program (feedback index, Fi; n=154).
Satisfaction questionnaire | Scores (N)a |
I found the Growing healthy app easy to use | 46 |
I liked the layout or “look” of the app | 34 |
I found it hard to navigate through the appb | 23 |
The Growing healthy app didn’t take long to load information | 45 |
The Growing healthy app failed to work at timesb | 28 |
The different sections of the app worked well together | 20 |
The language used in the app was easy to understand | 57 |
The app did everything I expected it to do | 31 |
I couldn’t find all of the answers I needed in the appb | 11 |
I had to use the search feature to find what I was looking for | 14 |
Using the app was an enjoyable experience | 22 |
I found the app complicatedb | 43 |
I can trust the information on the Growing healthy app | 39 |
I felt confident using this app | 40 |
I found the information for mums useful | 31 |
I found the information on feed and sleep patterns useful | 29 |
I found the information about breastfeeding useful | 20 |
I found the information about formula feeding useful | 17 |
I found the information on mixed feeding useful | 15 |
I found the information on solid feeding useful | 27 |
I found the videos on the app useful | 12 |
I found the recipe section of the app useful | 22 |
I shared the information from the app with other friends and family | 16 |
I was concerned about the Internet data usage on my phone when using the appb | 47 |
I found the information provided easy to understand | 36 |
Overall, I liked the Growing healthy program | 36 |
I would recommend the Growing healthy program to a friend | 45 |
I found it helpful to share the app with my partner or another carer | 48 |
The Growing healthy program covered all of the things about infant feeding that I wanted it to | 25 |
I received push notifications on my phone, from the Growing healthy programc | 122 |
The push notification messages often disappeared before I had a chance to tap on themb | 12 |
I didn’t know how to retrieve push notification messages once they disappeared from screenb | 12 |
I would prefer to receive text messages rather than push notifications from the app | 19 |
I was happy with the number of notifications or messages received each week | 6 |
I was happy with the time that the notification was sent to me during the day | 18 |
I found the notifications or messages helpful | 16 |
I found the notifications or messages suited my baby’s age and stage of development | 23 |
aTotal scores only include the extreme positive responses based on scoring criteria
bLikert scale scoring reversed for these questions: strongly disagree (1), disagree (0), no strong feelings either way (0), agree (0), strongly agree (1), and didn’t use (0).
cResponse option and scoring: Yes, I received weekly push notifications (1), no, I received text messages instead of push notifications (1), and no, I disabled my push notifications so I didn't receive any weekly messages (0).
The frequency of scores for click-depth index (Ci), loyalty index (Li), interaction index (Ii), and recency index (Ri) at each time point (initial, interim, and final).
This is one of the first studies to develop and implement an mHealth program supporting parents with healthy infant feeding practices through a mobile phone app. To our knowledge, this is the first study to utilize an EI to quantify and categorize participants’ engagement level using the app. We found that engagement level was positively correlated with primiparous status, use of both the app and email, exposure to the program for a longer period, and recruitment through health practitioners. Negative correlation was found with age of child at start of program and engagement level.
The identification of the correlates of participant engagement is not only beneficial to inform future enhancements of the Growing healthy program, but more broadly to evaluate mHealth programs. The EI has its origins in measurement of consumer engagement with Web-based products. Adjusting the index to measure engagement with a mHealth program was possible as the metrics measured are the same; only the measurement of the content and behavior will be different [
A criterion to categorize participants as poor, moderate, or highly engaged with the Growing healthy program based on their overall EI score was developed. Previously, program engagement has arbitrarily been labeled as high [
Participant app use over the 9-month period in this study varied such that engagement was high after initially joining the program but decreased from the 3- to 6-month period. Previous mHealth programs targeting long and short term behavior change have identified similar patterns of use [
Study participants who accessed both the website and the app attained a significantly higher EI score compared with participants who had just used the app. This supports the notion that delivering the intervention using various modes enhances engagement and to the intervention exposure [
While initial engagement is the initial hurdle for any intervention, sustaining engagement remains the most difficult part of intervention implementation, it is more difficult to achieve [
The infant’s age at baseline (ie, when the app was downloaded) was also strongly associated with higher EI scores. Participants who joined the program when their infant was younger had a higher EI score compared with those who joined when their infant’s age was closer to 3 months. Similar to traditional interventions that targeted childhood obesity prevention [
Participants who were recruited from their health practitioner were more likely to have higher EI scores compared with those who were recruited on the Web. This may be attributed to mothers’ perception that health practitioners are a trustworthy source of information [
Several studies describing mHealth interventions encouraging healthy infant feeding behaviors have recently been published. Delivery modes used in these studies included app [
This study has several limitations. First, a number of technological issues were experienced by participants in receiving and opening push notifications. Adaptation were therefore made midway during the program and all participants were sent weekly emails. Second, app quality is an important influencer on participant engagement [
Some features of the Growing healthy program were not measured using the EI because there were difficulties in obtaining individual participants’ information such as, participant use of the Growing healthy Facebook group and sharing the app with another carer or sharing information from the app with others (interconnectivity). Although participant interaction with these features was not measured, satisfaction and use of these features was included in the 9-month survey that made up the feedback index.
Some studies have shown that mothers from a disadvantaged background were less likely to use the Internet as a source of information for infant feeding [
To our knowledge, the utilization of an index to measure participant engagement has not yet been implemented in mHealth interventions. The EI provided detailed analysis regarding the frequency participants accessed the app and push notifications, how many pages they accessed per session, and their satisfaction with the program which was measured over 3 time points across the 9 months of the program.
The EI provided a comprehensive understanding of participant behavior with the app over the 9-month period of the Growing healthy program. The participants’ engagement with the Growing healthy app was determined by various factors including participant characteristics, novelty, intervention exposure time, and the quality of the app including technological aspects. Primiparous participants, those who accessed both the emails and the app, those who were exposed to the program for a longer period, and those who were recruited from their health practitioner all had higher EI scores. The use of the EI in this study demonstrates that rich and useful data can be collected and used to assess the strengths and weaknesses of mHealth interventions and in turn inform improvements in their design and delivery.
Participant satisfaction survey.
Click depth index
engagement index
feedback index
interaction index
loyalty index
recency index
interquartile range
The research reported in this paper is a project of the Australian Primary Health Care Research Institute, which is supported by a grant from the Australian government Department of Health and Ageing. The information and opinions contained in it do not necessarily reflect the views or policies of the Australian Primary Health Care Research Institute or the Australian government Department of Health and Ageing.
None declared.