This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Background
Digital interventions to reduce excessive alcohol consumption have the potential to have a broader reach and be more cost-effective than traditional brief interventions. However, there is not yet strong evidence for their ability to engage users or their effectiveness.
Objective
This study aimed to identify the behavior change techniques (BCTs) and engagement strategies most worthy of further study by inclusion in a smartphone app to reduce alcohol consumption, using formal expert consensus methods.
Methods
The first phase of the study consisted of a Delphi exercise with three rounds. It was conducted with 7 international experts in the field of alcohol and/or behavior change. In the first round, experts identified BCTs most likely to be effective at reducing alcohol consumption and strategies most likely to engage users with an app; these were rated in the second round; and those rated as effective by at least four out of seven participants were ranked in the third round. The rankings were analyzed using Kendall’s W coefficient of concordance, which indicates consensus between participants. The second phase consisted of a new, independent group of experts (n=43) ranking the BCTs that were identified in the first phase. The correlation between the rankings of the two groups was assessed using Spearman’s rank correlation coefficient.
Results
Twelve BCTs were identified as likely to be effective. There was moderate agreement among the experts over their ranking (W=.465, χ211=35.8, P<.001) and the BCTs receiving the highest mean rankings were self-monitoring, goal-setting, action planning, and feedback in relation to goals. There was a significant correlation between the ranking of the BCTs by the group of experts who identified them and a second independent group of experts (Spearman’s rho=.690, P=.01). Seventeen responses were generated for strategies likely to engage users. There was moderate agreement among experts on the ranking of these engagement strategies (W=.563, χ215=59.2, P<.001) and those with the highest mean rankings were ease of use, design – aesthetic, feedback, function, design – ability to change design to suit own preferences, tailored information, and unique smartphone features.
Conclusions
The BCTs with greatest potential to include in a smartphone app to reduce alcohol consumption were judged by experts to be self-monitoring, goal-setting, action planning, and feedback in relation to goals. The strategies most likely to engage users were ease of use, design, tailoring of design and information, and unique smartphone features.
Excessive alcohol consumption is a serious problem for population health [1,2]. Brief interventions to address this are time limited interventions delivered by health care workers targeting heavier drinkers and can be effective at reducing alcohol consumption [3]. There are substantial barriers to their delivery such as lack of time, training, and financial resources. These barriers can perhaps be avoided by delivering an intervention via a digital platform. While digital interventions have not been found to be as effective as face-to-face brief interventions [4], they may be more effective than no intervention [4-13], and have the advantage of being cost effective, avoid the stigma associated with help-seeking in person [10], and have greater reach than traditional health services. Smartphone applications or ‘apps’ have the additional advantage of being with the individual almost all of the time, which offers the potential to engage users in real time and in their everyday situations. Apps also have the ability to sense and report locations and events (in conjunction with calendar function) to provide moment-to-moment support when it is needed unlike traditional interventions. Despite a large number of apps to reduce excessive alcohol consumption in the general population, none, to our knowledge, have been rigorously evaluated. There has been a recent trial of an app on the related issue of recovery from alcoholism [14] that showed a reduction in the number of risky drinking days and therefore of probable benefit to patients in continuing care for alcohol dependence.
Reviews of digital interventions (not apps) suggest they can be effective, but there is substantial heterogeneity between different interventions [4,7,8,11,12]. Moreover, interventions have many components and their evaluations have rarely specified content in a way that would allow identification of the components responsible for the variation (e.g. [4,8,11,12]). A reliable method for specifying content and evaluating the effectiveness of complex behavior change interventions is to identify behavior change techniques (BCTs) [15]. BCTs are defined as the smallest, observable, replicable components with the potential to bring about change in behavior [16].
In order for an alcohol reduction app to be effective, it must be engaging for users, thus allowing them to be exposed to its active components. It is well established that a large proportion of users of digital interventions in health trials do not maintain engagement [17]. This degree of attrition undermines the potential of apps to be effective, and generalizable evaluation is made difficult when a large proportion of users cannot be recontacted due to disengagement with the intervention [18]. Engagement in Web-based interventions is increased by use of prompts [19-21], peer support [19], counselor support [19], and the combination of tailored communication with the use of reminders and incentives [22]. However, these have only been examined in the context of websites and there is a need to identify the most effective strategies for engagement with apps.
In sum, there is not yet an established evidence base to draw on to inform the selection of BCTs or engagement strategies in developing apps aimed at reducing alcohol consumption amongst the general population. In areas of research where there is a lack of, inconsistent, or contradictory scientific evidence, formal consensus methods have been used to guide action [23,24]. This study used a formal consensus building methodology with a small group of world-class experts in the field of alcohol and/or behavior change to identify intervention components judged to be the ‘best bets’ to reduce alcohol consumption (in general and in the context of an app) and to maintain engagement with an app, and then compared the original expert group’s ranking of intervention components with a new, broader expert review.
This study addressed the following research questions:
What BCTs do experts in the field of alcohol research agree are most likely to be effective in general and when delivered by an app?
What engagement strategies do experts believe are most likely to be effective initially and over time?
A Delphi-style methodology was used to generate consensus among experts about what intervention components are likely to be the most effective at reducing alcohol consumption, and what strategies are most likely to improve engagement with an app. Experts were asked to generate a list of ‘best bet’ intervention components and engagement strategies which were subsequently rated and ranked.
The Delphi method of generating consensus was selected as a formal, systematic and reproducible method of arriving at a consensus. It was conducted anonymously to avoid biases produced by perceived authority, persuasion or bandwagon effects [23,25].
Participants
Seven international academic experts (six male) were purposively identified from a range of scientific networks and backgrounds (health psychology, biological psychology, developmental psychopathology and addiction research) on the basis of their knowledge of the alcohol literature, and/or experience of designing or delivering behavior change interventions. Seven participants are considered sufficient for reliable group judgment [24,25]. None of the experts were identified based on any user experience expertise. The authorship team used their experience to judge the suitability of invited experts. Once the experts were identified, each was formally approached by an email invitation. All the experts who were approached agreed to take part. Experts were from the UK (n=6) and the Netherlands (n=1). Six were professors and one was a senior research fellow.
MeasuresRound 1:
Participants were asked to provide between three and five responses to each of three questions:
What intervention components do you believe would be the best bets for helping people reduce their alcohol consumption?
What intervention components do you believe would be the best bets for helping people to reduce their alcohol consumption when delivered by a smartphone app?
What do you think are the best strategies or techniques for maintaining engagement with an app aiming to help people reduce their alcohol consumption?
Each question was preceded by the statement: “Please answer the following questions based on your knowledge of the research literature, relevant theory and your clinical experience. Please also provide the reason behind your choice.” For question 2, participants were given the option to indicate that their answers were the same as for question 1.
Round 2:
Participants were provided with an alphabetical list of the responses generated in the first round for each of the questions. They were instructed “Please rate your agreement with each of these techniques for the three different questions on the five-point Likert scales provided”. The scale ranged from 1 (strongly disagree), 2 (disagree), 3 (neither agree nor disagree), 4 (agree) to 5 (strongly agree). Participants were given the option to make comments on their rating.
Round 3:
The n responses were listed alphabetically with the mean agreement rating and rationale provided for each response. Participants were asked to rank the n responses from 1 (most likely to be a best bet) to n (least likely to be a best bet) for each of the questions. At this stage, participants were only asked to rank responses about which there had been broad agreement in the previous round, defined as a minimum of four out of seven of the participants agreeing (i.e., rating of 4 or above) that the technique was likely to be either effective or engaging (depending upon the question) [23]. The reason for removing responses about which there was little agreement was to improve responding by minimizing the time required to complete the survey [23]. There was the option to make any final comments at this point.
Procedure
This study was conducted using the online survey tool Qualtrics. A link to the survey for each of the three rounds was emailed to the participants and they were given between one and two weeks to complete it. Non-responders were sent reminders until all participants had completed each round. Participants provided informed consent.
AnalysisRound 1:
For each question, similar responses were summarized and combined. For question 1, a BCT was selected from one of two taxonomies [15,26] to describe each response for the intervention components, where appropriate. The summarizing, combining and coding of responses was conducted by CG & SM.
Round 2:
The mean, standard deviation (SD), and mode of the agreement ratings for each response to each of the three questions were calculated.
Round 3:
The final rankings were analyzed by calculating Kendall’s W coefficient of concordance [27], which measures the extent to which judges agree on their rankings of items. The value of W ranges from 0 (indicating no consensus) to 1 (indicating perfect consensus) between participants. A value of .1 corresponds to very weak agreement, .3 to weak agreement, .5 to moderate agreement, .7 to strong agreement and .9 to unusually strong agreement [28]. The Kendall’s W statistic uses the χ2test to test the independence of the ranking of the components.
Second Phase: External ValidationStudy design
The intervention components generated and ranked in the first phase of the study were also ranked by a second group of experts in the field of alcohol.
Participants
Assistant and Senior Editors (n=179) from the journal Addiction were invited to take part in the study if they believed they had a sufficiently informed ‘opinion on interventions that might help people who drink more alcohol than is good for them to reduce or quit’. This invitation yielded 43 participants.
Measures
Participants were asked to rank from 1 (highest) to 12 (lowest), the value of 12 responses generated in the first phase of the study by the original group of experts, in response to the question “What intervention components do you believe would be the best bets for helping people reduce their alcohol consumption?”
Procedure
An email was circulated to all the assistant and senior editors at the journal of Addiction with an alphabetical list of the “best bet” intervention component responses. If they wished to take part in the study, they were asked to reply (via email) with a ranking for each of the intervention components. Participants were given one week to reply before the study closed.
Analysis
The correlation between the rankings of the original and the new independent group of experts was assessed using Spearman’s rank correlation coefficient. The new rankings were also analyzed using Kendall’s W coefficient of concordance [27] to assess the extent to which this second group agreed with each other.
ResultsFirst Phase: 3-Round Consensus Exercise
In response to the question of what intervention components are likely to be the most effective at reducing alcohol consumption, 24 responses were recorded in round 1. Eighteen of these responses were similar to at least one other, resulting in 12 components (see Multimedia Appendix 1), of which 11 corresponded directly with a BCT (see Table 1). Six of the 7 participants thought that intervention components likely to be effective in general would be the same as in an app. The other participant generated one suggestion to do with the intervention modality itself and how to present the intervention in a unique way. The response was therefore included with the responses to the question regarding engagement strategies.
Four of the 12 components (self monitoring, goal setting, action planning, and feedback in relation to goals) had a mean ranking score greater than the average rank (6 out of 12) and the lowest mean agreement rating for these four BCTs was 4.3 (see Table 1). Overall the original group of experts displayed moderate agreement (Kendall’s W=.465) in their ranking of intervention components (χ2(11)=35.77, P<.001).
Responses generated by the expert group on effective behaviour change techniques to reduce alcohol consumption.a
Responses generated
Equivalent BCTs
Agreement ratingb
Ranking scorec
Mean (SD)
Mode
Agree :Disagreed
Mean (SD)
Mode
Self monitoring
Self monitoring of behaviore
4.6 (.54)
5
7:0
2.4 (1.81)
1
Goal setting
Goal setting (behavior)e
4.7 (.049)
5
7:0
2.6 (1.51)
1, 2
Action planning
Action planninge
4.3 (.49)
4
7:0
4.3 (.95)
4
Feedback in relation to goals
Provide feedback on performancef
4.6 (.54)
5
7:0
4.43 (2.70)
3
Behavior substitution
Behavior substitutionf
4.1 (.38)
4
7:0
6.3 (2.06)
5, 7
Environmental triggers and drivers
Advise on environmental restructuringf
3.9 (.69)
4
5:2
7.3 (4.07)
2, 9
Provide information
Provide information on consequences of excessive alcohol consumption & reducing excessive alcohol consumptionf
4.0 (.58)
4
6:1
7.4 (4.47)
12
Feedback in relation to people
Provide normative information about others’ behavior and experiencesf
4.0 (.58)
4
6:1
8.4 (1.90)
7
Motivational interviewing
Conduct motivational interviewingf
3.9 (1.07)
4
5:2
8.4 (3.41)
12
Inhibition training
3.6 (.54)
4
4:3
8.4 (3.51)
10
Reward
Provide rewards contingent on successfully reducing excessive alcohol consumptionf
3.9 (.69)
4
5:2
8.9 (2.12)
11
Habit reversal
Habit reversalf
3.4 (.79)
4
4:3
9.1 (1.68)
10
aResponses ordered in terms of mean ranking score (from round 3).
dAgree:Disagree (ratio of (agree/strongly agree): (neither/disagree/strongly disagree) used as inclusion criteria for round 3.
eBCTs as referred to in the 93-item BCT Taxonomy v1 [15]
fBCTs as referred to in the 42-item excessive alcohol reduction specific taxonomy [26]
Of the 20 engagement strategies generated, six were similar to at least one other and thus were combined, which resulted in 17 unique strategies (see Multimedia Appendix 2 for the rationale for each of the 17 responses). Seven strategies (ease of use, design aesthetic, feedback, function, ability to change design to suit own preferences, tailored information and unique smartphone features) had a mean ranking score greater than average rank (8 out of 16) and the lowest mean agreement rating for these strategies was 3.6 (see Table 2). Overall the experts showed a moderate degree of consensus in their ranking of the strategies (Kendall’s W=.563, χ215=59.2, P<.001).
Responses generated by the expert group on engagement strategies.a
Responses
Agreement ratingb
Ranking scorec
Mean (SD)
Mode
Agree:Disagreed
Mean (SD)
Mode
Ease of use
4.9 (.38)
5
7:0
1.4 (.79)
1
Design – aesthetic
4.6 (.54)
5
7:0
3.1 (1.57)
2, 5
Feedback
4.6 (.54)
5
7:0
3.9 (1.68)
4
Function
4.0 (.82)
4
5:2
6.6 (3.60)
11
Design – ability to change design to suit own preferences
3.6 (.79)
4
5:2
6.9 (4.74)
3
Tailored information
4.3 (.76)
4, 5
6:1
7.9 (3.39)
6, 7
Unique smartphone features
4.4 (.54)
4
7:0
7.9 (5.79)
6
Prompts
4.1 (.38)
4
7:0
8.4 (2.44)
8
Graded tasks
4.0 (.82)
4
5:2
8.7 (3.50)
12
Gamification
4.1 (.69)
4
6:1
8.9 (5.30)
10
Social comparison
3.9 (.69)
4
5:2
10.4 (3.36)
9
Reward type Novelty
4.0 (.82)
4
5:2
11.6 (2.23)
12
Reward type Games
3.7 (.49)
4
5:2
11.9 (2.97)
11, 15
Reward type Positive messages
4.0 (.58)
4
6:1
12.1 (2.79)
8, 10, 11, 12, 13, 15, 16
Reward type Financial
3.6 (.98)
4
4:3
12.3 (1.98)
13
Social connectivity
4.0 (.58)
4
6:1
14.1 (1.95)
15, 16
Reward type- cue signaling rewarde
3.4 (.98)
3
3:4
-
aResponses ordered in terms of mean ranking score (from round 3).
dAgree:Disagree (ratio of (agree/strongly agree): (neither/disagree/strongly disagree) used as inclusion criteria for round 3.
eThis response was not included in round 3 because there was not substantive agreement that it would be an effective engagement strategy in round 2 (defined as a minimum of 4 out of 7 of the participants agreeing (i.e., rating of 4 or above) that the technique was likely to be engaging).
Second Phase: External Validation
The ranking of the BCTs by the original group was validated by an independent group of experts: there was a significant correlation between their two rankings (see Table 3; ρ=.69, P=.01). Table 3 shows the ranking by the independent group of experts of the intervention components generated and agreed by the original group. There was modest but significant agreement amongst the broader group of experts (Kendall’s W=.320, χ211=151.52, P<.001).
Comparison between rankings of phase 1 expert group and larger expert group of effective behavior change techniques for alcohol use reduction.a
Responses
Phase 1 experts
Phase 2 experts
N=7
N=43
Mean Rank (SD)
Mean Rank (SD)
Self monitoring
2.4 (1.81)
3.4 (2.88)
Goal setting
2.6 (1.51)
3.8 (3.00)
Action planning
4.3 (.95)
6.4 (2.72)
Feedback in relation to goals
4.4 (2.70)
4.1 (2.28)
Behavior substitution
6.3 (2.06)
7.6 (2.51)
Environmental triggers and drivers
7.3 (4.07)
5.1 (2.72)
Provide information
7.4 (4.47)
9.5 (2.87)
Feedback in relation to people
8.4 (1.90)
7.4 (3.27)
Motivational interviewing
8.4 (3.41)
7.2 (2.82)
Inhibition training
8.4 (3.51)
8.8 (2.15)
Reward
8.9 (2.12)
6.8 (3.44)
Habit reversal
9.1 (1.68)
7.9 (2.69)
aResponses ordered in terms of mean ranking score for the original experts (from round 3)
Discussion
BCTs of self monitoring, goal setting, action planning, and feedback in relation to goals were ranked most likely to be effective for reducing alcohol use by a group of international experts in the field of alcohol or behavior change or both. This finding was validated by a larger independent group of alcohol experts. None of the experts thought that the BCTs likely to be effective in general would differ from those in an app, though one participant suggested presenting information in a way that was unique to an app. The most highly ranked engagement strategies were ease of use, design-aesthetic, feedback, function, design-ability to change design to suit own preferences, tailored information and unique smartphone features.
There is empirical evidence for the effectiveness of some of the BCTs identified in this study for reducing excessive alcohol consumption. Self monitoring has been found to be effective in brief interventions [26], and is also used in a number of apps to reduce alcohol consumption [29] though none of these have been evaluated. The BCT ‘feedback in relation to people’ is often referred to as normative feedback in the alcohol behavior change literature. There is evidence to suggest that this BCT may have a small effect by several different modes of delivery: face-to-face [30], via phone [31], mailed [32,33] and via digital platforms [30,34,35]. However, this research is often limited to college and university students [30,32,34,35]. The highest priority engagement strategies of prompts, social connectivity and tailored information have all been shown to result in increased use of Web-based interventions [19-22].
The use of a Delphi approach to selecting intervention components is clearly not guaranteed to result in the best choices, but on a priori grounds it seems preferable to the more usual practice of drawing on expertise and interest within a single research team. It may have been that no consensus would be achieved so, while the level of agreement within each group of experts was modest, the fact that the aggregate rankings of the two expert groups showed a high level of concordance was reassuring that the study tapped into a shared perspective on the existing evidence.
It is possible that the results of the Delphi exercise could have been biased by choosing an expert group with similar backgrounds to those of the research team. Therefore, the use of a second group of experts to validate the rankings provided important support for this not being the case. The journal Addiction has a very large pool of international experts on its editorial team and arguably includes most of the leading researchers in the field covering a wide range of expertise. The question regarding user engagement was included for exploratory purposes. As shown in this study, experts in the academic field of research did not identify any BCTs as being effective for an app compared with a traditional intervention. This may be because they are not aware of the additional functions an app can provide in terms of a behavior change intervention. Future research is planned to compare the views of experts in the relevant academic field with that of user experience experts to see if there are any discrepancies between these groups and if so, how their opinions differ.
The results of this study will be used to inform the building of a prototype app that will be evaluated in a field experiment. Following the principle of optimization [36] each component will be included in a full form or minimal form using a factorial design so that its effect can be assessed. The findings should also be useful to other research teams considering developing and evaluating apps in this area.
Multimedia Appendix 1
Intervention components generated by the experts in the first round.
Multimedia Appendix 2
Engagement strategies generated by the experts in the first round.
AbbreviationsBCTs
behavior change techniques
NIHR SPHR
National Institute of Health Research, School for Public Health Research
SSA
UK Society for the Study of Addiction
UKCTAS
UK Centre for Tobacco and Alcohol Studies
CG is funded by the UK Centre for Tobacco and Alcohol Studies (UKCTAS). DC is funded by the National Institute of Health Research, School for Public Health Research (NIHR SPHR). JB’s post is funded by a fellowship from the UK Society for the Study of Addiction (SSA). RW is funded by Cancer Research UK. The research team is part of the UKCTAS and we are grateful to UKCTAS, NIHR SPHR and SSA for funding the study.
JB has received an unrestricted research grant from Pfizer related to the surveillance of smoking cessation trends. RW has received research funding and undertaken consultancy for companies that manufacture smoking cessation medications. CG, DC and SM have no declared conflicts of interest.
World Health OrganizationWorld Health Organization20112015-06-07Global status report on alcohol and healthhttp://www.who.int/substance_abuse/publications/global_alcohol_report/msbgsruprofiles.pdf6Z76jjvHtBalakrishnanRAllenderSScarboroughPWebsterPRaynerMThe burden of alcohol-related ill health in the United KingdomJ Public Health (Oxf)20090931336637310.1093/pubmed/fdp05119493915fdp051KanerEFSDickinsonHOBeyerFPienaarESchlesingerCCampbellFSaundersJBBurnandBHeatherNThe effectiveness of brief alcohol interventions in primary care settings: a systematic reviewDrug Alcohol Rev20090528330132310.1111/j.1465-3362.2009.00071.x19489992DAR071ElliottJCCareyKBBollesJRComputer-based interventions for college drinking: a qualitative reviewAddict Behav200808338994100510.1016/j.addbeh.2008.03.00618538484S0306-4603(08)00083-XPMC2441945CunninghamJAHumphreysKKoski-JännesACordingleyJInternet and paper self-help materials for problem drinking: is there an additive effect?Addict Behav2005093081517152310.1016/j.addbeh.2005.03.00315893433S0306-4603(05)00057-2BewickBMTruslerKBarkhamMHillAJCahillJMulhernBThe effectiveness of web-based interventions designed to decrease alcohol consumption--a systematic reviewPrev Med200807471172610.1016/j.ypmed.2008.01.00518302970S0091-7435(08)00043-1CareyKBScott-SheldonLAJElliottJCBollesJRCareyMPComputer-delivered interventions to reduce college student drinking: a meta-analysisAddiction200911104111807181910.1111/j.1360-0443.2009.02691.x19744139ADD2691PMC2763045CareyKBScott-SheldonLAJElliottJCGareyLCareyMPFace-to-face versus computer-delivered alcohol interventions for college drinkers: a meta-analytic review, 1998 to 2010Clin Psychol Rev20121232869070310.1016/j.cpr.2012.08.00123022767S0272-7358(12)00114-6PMC3511828KhadjesariZMurrayEHewittCHartleySGodfreyCCan stand-alone computer-based interventions reduce alcohol consumption? A systematic reviewAddiction201102106226728210.1111/j.1360-0443.2010.03214.x21083832RiperHSpekVBoonBConijnBKramerJMartin-AbelloKSmitFEffectiveness of E-self-help interventions for curbing adult problem drinking: a meta-analysisJ Med Internet Res2011132e4210.2196/jmir.169121719411v13i2e42PMC3221381RookeSThorsteinssonEKarpinACopelandJAllsopDComputer-delivered interventions for alcohol and tobacco use: a meta-analysisAddiction20100810581381139010.1111/j.1360-0443.2010.02975.x20528806ADD2975VernonMLA review of computer-based alcohol problem services designed for the general publicJ Subst Abuse Treat20100438320321110.1016/j.jsat.2009.11.00120015607S0740-5472(09)00184-6PMC2835799DonoghueKPattonRPhillipsTDelucaPDrummondCThe effectiveness of electronic screening and brief intervention for reducing levels of alcohol consumption: a systematic review and meta-analysisJ Med Internet Res2014166e14210.2196/jmir.319324892426v16i6e142PMC4060043GustafsonDHMcTavishFMChihMAtwoodAKJohnsonRABoyleMGLevyMSDriscollHChisholmSMDillenburgLIshamAShahDA smartphone application to support recovery from alcoholism: a randomized clinical trialJAMA Psychiatry20140571556657210.1001/jamapsychiatry.2013.4642246711651847578PMC4016167MichieSRichardsonMJohnstonMAbrahamCFrancisJHardemanWEcclesMPCaneJWoodCEThe behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventionsAnn Behav Med201308461819510.1007/s12160-013-9486-623512568MichieSAbrahamCEcclesMPFrancisJJHardemanWJohnstonMStrengthening evaluation and implementation by specifying components of behaviour change interventions: a study protocolImplement Sci201101611010.1186/1748-5908-6-10212998601748-5908-6-10PMC3041694EysenbachGThe law of attritionJ Med Internet Res20050171e1110.2196/jmir.7.1.e1115829473v7e11PMC1550631EysenbachGIssues in evaluating health websites in an Internet-based randomized controlled trialJ Med Internet Res20021243E1710.2196/jmir.4.3.e1712554548PMC1761938BrouwerWKroezeWCrutzenRdeNJde VriesNKBrugJOenemaAWhich intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic reviewJ Med Internet Res2011131e210.2196/jmir.163921212045v13i1e2PMC3221341SchneiderFvan OschLSchulzDNKremersSPde VriesHThe influence of user characteristics and a periodic email prompt on exposure to an internet-delivered computer-tailored lifestyle programJ Med Internet Res2012142e4010.2196/jmir.193922382037v14i2e40PMC3376522SvenssonMSvenssonTHansenAWLagerrosYTThe effect of reminders in a web-based intervention studyEur J Epidemiol20120527533334010.1007/s10654-012-9687-522531973CrutzenRde NooijerJBrouwerWOenemaABrugJde VriesNKStrategies to facilitate exposure to internet-delivered health behavior change interventions aimed at adolescents or young adults: a systematic reviewHealth Educ Behav201102381496210.1177/1090198110372878211894221090198110372878OsborneJCollinsSRatcliffeMMillarRDuschlRWhat "ideas-about-science" should be taught in school science? A Delphi study of the expert communityJ Res Sci Teach2003082540769272010.1002/tea.10105BlackNMurphyMLampingDMcKeeMSandersonCAskhamJMarteauTConsensus development methods: a review of best practice in creating clinical guidelinesJ Health Serv Res Policy1999104423624810623041MurphyMKBlackNALampingDLMcKeeCMSandersonCFAskhamJMarteauTConsensus development methods, and their use in clinical guideline developmentHealth Technol Assess1998231889561895MichieSWhittingtonCHamoudiZZarnaniFToberGWestRIdentification of behaviour change techniques to reduce excessive alcohol consumptionAddiction20120810781431144010.1111/j.1360-0443.2012.03845.x22340523OkoliCPawlowskiSDThe Delphi method as a research tool: an example, design considerations and applicationsInformation & Management200412421152910.1016/j.im.2003.11.002SchmidtRCManaging Delphi Surveys Using Nonparametric Statistical TechniquesDecision Sciences19970728376377410.1111/j.1540-5915.1997.tb01330.xCraneDGarnettCBrownJWestRMichieSBehavior change techniques in popular alcohol reduction apps: content analysisJ Med Internet Res2015175e11810.2196/jmir.406025977135v17i5e118FoxcroftDRMoreiraMTSantimanoNMLASmithLASocial norms information for alcohol misuse in university and college studentsCochrane Database Syst Rev20151CD00674810.1002/14651858.CD006748.pub325622306CunninghamJANeighborsCWildTCHumphreysKNormative misperceptions about alcohol use in a general population sample of problem drinkers from a large metropolitan cityAlcohol Alcohol2012471636610.1093/alcalc/agr12522028458agr125PMC3243438CollinsSECareyKBSliwinskiMJMailed personalized normative feedback as a brief intervention for at-risk college drinkersJ Stud Alcohol20020963555956712380852CunninghamJAWildTCBondySJLinEImpact of normative feedback on problem drinkers: a small-area population studyJ Stud Alcohol20010362222823311327189KypriKLangleyJDPerceived social norms and their relation to university student drinkingJ Stud Alcohol20031164682983414743946NeighborsCLarimerMELewisMATargeting misperceptions of descriptive drinking norms: efficacy of a computer-delivered personalized normative feedback interventionJ Consult Clin Psychol20040672343444710.1037/0022-006X.72.3.434152795272004-95166-007CollinsLMMurphySAStrecherVThe multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventionsAm J Prev Med200705325 SupplS112S11810.1016/j.amepre.2007.01.02217466815S0749-3797(07)00051-7PMC2062525