For the purposes of assessing the acceptability of a smoking intervention and determining points for potential program improvement, we performed a qualitative analysis of in-depth interviews conducted with participants of an RCT pilot study that compared the effectiveness of behavioral counseling and the use of e-cigarettes on smoking outcomes to that of behavioral counseling and nicotine replacement therapy (NRT).

The interviews and analysis procedures were reported in accordance with the COREQ (Consolidated Criteria for Reporting Qualitative Research) checklist (Checklist 1), as applicable [17]. The study protocol was approved by the New York University Langone Health Institutional Review Board (approval number i20-00839), and written documentation of informed consent was received prior to starting data collection. Participants were provided with a US $20 incentive for their participation. Participant data were maintained on a secure server. After transcription, all participant data were deidentified prior to data analysis.

Interview participants were recruited upon completion of the intervention phase of the RCT at the 12-week follow-up study visit. All RCT participants were invited to participate in an in-depth interview to discuss their experiences with the intervention and other aspects of the trial. Interview recruitment ended once thematic saturation was reached. Interviews lasted approximately 30 minutes and were performed between April 2021 and November 2022.

The pilot RCT was performed to determine the feasibility and acceptability of an e-cigarette–based smoking intervention and to compare the effectiveness of counseling and e-cigarette use on smoking outcomes to that of counseling and NRT [16]. Text message–based EMA data collection was used to record smoking patterns. Patients from the electronic health record system of New York University Langone Health—a private hospital system serving New York, New Jersey, and Connecticut—were recruited as RCT participants. The RCT participant sample was initially restricted to patients with a diagnosis of chronic obstructive pulmonary disease, but the scope was later expanded to include patients with a diagnosis of coronary artery disease, peripheral artery disease, or asthma. In addition, to be eligible, RCT participants were required to smoke ≥4 days per week, with at least 5 cigarettes smoked on the days that participants did smoke; be motivated to quit smoking; and possess a phone with SMS text messaging capabilities. A total of 121 participants were recruited into the pilot RCT.

In the pilot RCT, combustible cigarette use and e-cigarette use, as well as smoking-related cravings, were measured as part of an EMA protocol, in which SMS text messaging served as a smoking diary. The SMS text messaging was intended for data collection only and not designed to serve as part of the intervention. After a baseline assessment, participants were asked to record their daily nicotine use for 12 weeks by responding to text message prompts that they received 4 times per day. The coverage design prompted participants to provide brief check-in reports via text message over the course of each day, wherein they were asked to report combustible cigarette and e-cigarette use based on their study arm (Figure 1). Responses to 1-item measures of cigarette craving and satisfaction were also collected from each report.

Semistructured, in-depth telephone interviews, which lasted approximately 30 minutes, were conducted by research staff (OW and MV). Interviews were audio-recorded, transcribed, and imported into Dedoose software (SocioCultural Research Consultants LLC) for qualitative data analyses.

The interview guide covered topics that were designed for the following goal: gaining a deeper understanding of the participants’ experiences, intervention satisfaction, attitudes toward e-cigarettes, and intentions to quit. The interviews were intended to assist with the further adaptation of the e-cigarette smoking intervention and behavioral counseling manual used in the RCT. The interview guide was developed based on the pilot RCT procedures to provide additional insights about the barriers and facilitators of e-cigarette use and how to refine the current approach to enhance program retention and outcomes. Interviews covered topics such as program aspects that the participants liked or disliked, features of the intervention that should be modified, participants’ experiences with using e-cigarettes or NRT, intentions of using e-cigarettes after the intervention, and whether participants’ health symptoms interfered with their ability to engage in the intervention. The participants were prompted to share their opinions on the EMA texting component of the trial but were not directly asked about the influence of the EMA on their behaviors. Participants were prompted to discuss the texting in the interview, as follows: “During the program, you answered questions over text on daily basis—What was that experience like for you?” This was followed by probes, including “How may texting have affected your overall experience with the program?” A copy of the interview script is available in Multimedia Appendix 1.

Interview transcripts were coded by using procedures that were designed to ensure thoroughness and reliability. We used Dedoose software to manage the data and coded the data according to the principles of the framework for applied research [18], which consists of the following 5-stage process: familiarization, identifying themes, indexing, charting, and interpretation. Codes were primarily developed a priori based on intervention components and the quality improvement goals of this study. Additional codes were developed by reviewing a random sample of interviews and via discussion with the coding team. The general development of themes arose from the data, using the principles of grounded theory [19]. To enhance reliability, 2 researchers took part in the coding and analysis process for each interview. Prior to full coding, a random sample of interviews was double coded, and intercoder reliability was assessed based on percent agreement (range 79.5%-87.9%). Afterward, all disagreements were discussed as a group to improve concordance among coders. All interviews were then independently coded by 5 coders (GX, KO, RL, RW, and SR) who worked in various pair combinations; each pair met with the other coders and a coauthor (ERS) to resolve discrepancies. When coding was completed, the quotations for each code were examined, summarized, and grouped together into themes.

Interviews were performed with a total of 27 participants. Due to an audio malfunction, 1 interview was not included in the analysis. As such, 14 participants were in the e-cigarette study arm and 12 were in the NRT study arm. The average age of the participants included in the analysis was 57.1 (range 28-74) years; 54% (14/26) of participants identified as female, 46% (12/26) were White, 31% (8/26) were Black, 15% (4/26) were Hispanic, and 4% (1/26) were Asian. The majority (17/26, 65%) of participants had completed at least some college; 35% had a high school education or less. Around two-thirds (17/26, 65%) of participants had a diagnosis of chronic obstructive pulmonary disease, 15% (4/26) of participants were diagnosed with coronary artery disease or peripheral artery disease, and 19% (5/26) of participants were diagnosed with asthma.

This study indicates that smoking intervention participants perceive the collection of EMA smoking behavior data via SMS text messaging as a potential influence on their smoking behaviors. The themes developed from the analysis revealed that EMA, in the form of an SMS text messaging smoking diary, may be perceived as a source of accountability for smoking reduction but may also be a trigger for cigarette use among some people. The results of this study emphasize the need to examine the potential influence of EMA data collection techniques on participants’ behaviors within smoking interventions, as well as in other behavioral research.

The perceived impacts of EMA on smoking behaviors are consistent with previously made observations that the act of receiving EMA prompts can increase behavioral awareness and act as a trigger for craving [6-8], in addition to altering participants’ moods [2,12]. This suggests that EMA for data collection purposes has the potential to unintentionally create an assessment and intervention effects in itself. Although a lack of EMA impact on behavior has been reported in some studies [11,14,15], these studies may be limited by the choice of measures used. As seen in suicide research, EMA prompts have been observed to have an effect on some measures, such as mood, but a minimal effect on other measures, such as suicidal ideation [2]. Therefore, when designing a study, it is important for researchers to reflect on the various factors that may influence the behaviors of interest and consider the potential effect that EMA may have on these factors, in addition to the primary outcomes of interest. Due to its potential effect on participants, investigators should consider and discuss the potential for a behavioral influence to be introduced into a study through the use of EMA data collection.

There is a need for further investigation into the ways that and the degree to which EMA affects participants. Within the EMA literature, there is a general lack of discussion around the effects of EMA on participants’ behaviors. When seeking to improve EMA methods, focus is often placed on participant retention and the validity of the data collection method [20-23], with little to no discussion on the potential behavioral impact of EMA. Indeed, when discussing strengths and limitations of EMA, a large portion of the EMA study literature discusses and reports measures of adherence to and reliability of EMA data collection [20-23], with few studies exploring the potential limitation of EMA in which the data collection itself may affect behaviors of interest [2,5-8,11,12,14,15]. A potential effect from EMA may influence the interpretation of the results; therefore, investigators ought to be encouraged to report considerations related to EMA when designing and publishing a study. Future research may benefit from randomizing a subset of participants to receive one EMA modality (eg, texting) while observing behaviors among all participants with another measurement modality (eg, Bluetooth e-cigarette monitor).

When the potential for EMA to influence study outcomes is identified, less obtrusive EMA methods could be considered, when available. The participants in this study expressed the omnipresent awareness and anticipation of the SMS text messaging–based EMA. This awareness altered participants’ behaviors and resulted in negative emotions that likely would not have been emphasized had the EMA not been used or had been subtler. There are numerous types of EMA strategies used in smoking research [4]. Future intervention research studies could consider less frequent SMS text messaging or EMA data collection methods outside of SMS text messaging that may have a more minor impact on smoking behaviors, such as the use of biosensors [24], Bluetooth-enabled devices [25], or puff counters [26].

This study had a few limitations. First, the interview guide was not designed to investigate the impact of EMA on participants’ smoking behaviors. Therefore, further details on the effects of EMA were not deeply explored, limiting the scope of this analysis. The unprompted nature of the participants’ observations of behavioral impact, however, strengthens the conclusion that the SMS text messaging–based EMA had a meaningful impact on the trial participants. Second, the sample of interview participants was not randomized, and the interview was not required; rather, it was offered to all participants sequentially as an optional component. This potentially introduced selection bias, as those with stronger opinions on the program may have been more likely to participate. Third, as EMA data were collected as part of a smoking reduction trial, it is difficult to completely disentangle the effects of the intervention on changes in behavior from the effects of EMA. Finally, the impact of EMA on behavior change was based on the self-reported perceptions of interview participants, and behavior changes were not directly observed. Therefore, this study cannot be used as conclusive evidence that the EMA had a significant impact on smoking behaviors, and further research is needed.

The collection of EMA smoking behavior data via SMS text messaging may influence the behaviors and perceptions of participants in smoking interventions. More research is needed to determine the magnitude of impact and mechanisms, to account for the potential effects of EMA on behavior change. Furthermore, a broader discussion of the behavioral influence introduced by the use of EMA may be warranted among the EMA research community.