Concordance of Text Message Ecological Momentary Assessment and Retrospective Survey Data Among Substance-Using Men Who Have Sex With Men: A Secondary Analysis of a Randomized Controlled Trial

Introduction

Use of alcohol and illicit drugs (ie, substance use) is a major contributor to the global burden of disease [1,2] and is more prevalent among men who have sex with men (MSM) than among the general population [3,4]. In a 2011 national sample of MSM across 20 US cities, 85% reported drinking alcohol and 50% reported binge drinking (having 5 or more alcoholic beverages on a single occasion) in the past 30 days and 49% reported using noninjection drugs in the past 12 months [3]. Heavy alcohol use and the use of noninjection drugs, particularly methamphetamine, have been linked to the transmission and acquisition of HIV among MSM, who continue to be more affected by HIV than any other demographic group in the United States [5]. Understanding alcohol and substance use behaviors among MSM at high risk for HIV is essential for effectively informing efforts to reduce HIV transmission and improve overall health in this population.

Research assessing individual patterns and the frequency of alcohol and substance use often utilizes questionnaires or interviews that rely on retrospective self-reported information. A major limitation of these methods is that information recalled from memory is prone to distortion, which can affect the reliability of collected data [6,7]. Ecological momentary assessment (EMA) is a set of methods developed to mitigate recall bias by collecting data about subjects’ mental states and behaviors on a near real-time basis [7,8]. EMA methods involve repeated assessments within subjects’ natural environments and routines and vary widely in design [9]. Studies using EMA methods, for example, may combine participant-initiated data reporting with randomly administered assessments throughout the day, or administer assessments at regular intervals to achieve abbreviated recall windows ranging from hours to an entire day [9-13]. Moreover, these methods can employ written or electronic diaries, physiological sensors, or mobile devices, such as mobile phones [7]. The popularity and near-universal prevalence of mobile phones makes them a powerful conduit for data collection, especially with the simplicity of text messaging (short message service, SMS) [14]. Furthermore, EMA may be particularly effective for substance use research due to the episodic and discrete nature of substance use behavior, which makes it better suited for real-time data collection than behaviors or states that are more ongoing [9]. Despite concerns regarding the ability and willingness of substance-using populations to adhere to the high-engagement requirements of EMA methods, a growing body of literature has demonstrated the feasibility, acceptability, and reliability of EMA among those who engage in substance use [15-21].

Although it has shown to be successful, EMA remains underutilized in substance use and HIV research. To date, relatively few studies have employed EMA methods to assess substance use patterns among MSM, and none have assessed the concordance of information reported using EMA with retrospective questionnaires among this population. To better understand the utility of different data collection methods among substance-using MSM, we aimed to assess the concordance between daily reports of substance use collected by EMA text messages and retrospective questionnaires, and to identify predictors of daily concordance.

Methods

Study Sample

We conducted a secondary analysis of EMA text responses and audio computer-assisted self-interview (ACASI) survey data from June 2013 to September 2014 as part of a randomized controlled trial assessing the feasibility, acceptability, and tolerability of a pharmacologic intervention to reduce methamphetamine and alcohol use among nondependent MSM in San Francisco, California [22]. Active recruitment efforts included outreach at municipal sexually transmitted infection (STI) and HIV clinics, MSM community-based organizations, gay bars and events, and syringe access programs. Passive recruitment efforts included flyers at active recruitment sites and advertisements in local newspapers, gay print media, and on social media. Eligibility criteria included: self-reported methamphetamine use at least 2 times per month, binge alcohol use at least once per week, methamphetamine or alcohol use concurrent with anal intercourse in the past 3 months, and a desire to reduce or discontinue methamphetamine and alcohol use. Additionally, participants were required to have a mobile phone that could send and receive text messages. All study participants provided informed consent and study procedures were approved by the Committee on Human Research, University of California, San Francisco.

Data Collection

EMA data were collected via daily text messages. Beginning the day after a baseline visit, participants were sent daily text messages asking about their behaviors on the previous day. They were asked whether they had taken the study drug, used methamphetamine, or used alcohol (and if so, the number of alcoholic drinks consumed). Participants received this text message series every day until the conclusion of the follow-up period (approximately 2 months). As reimbursement for their time, participants received a $1 stipend for each day that they completed the text message series.

ACASI surveys were administered at baseline and approximately every 2 weeks for a total of 5 visits. Each ACASI survey asked participants whether or not they had engaged in the following behaviors on each of the preceding 14 days: methamphetamine use, alcohol use, and binge alcohol use. To enhance recall, participants were provided with the exact calendar dates of the preceding 14 days as part of the assessments for these 3 behaviors, using a modified timeline follow-back approach.

Additional data collected at baseline included: demographic characteristics, including race, age, and highest level of education achieved; number of days in the past 4 weeks during which the participant engaged in binge alcohol use; and the frequency of methamphetamine use in the past 4 weeks as either less than one day per month or a selection of 1 through 4 days per month and 2 through 7 days per week. The frequency of methamphetamine use was then converted into a continuous variable to correspond to the number of days in the past 4 weeks during which methamphetamine was used. Due to our small sample size and the limited number of participants who identified as a race other than white or African American (Latino/Hispanic n=5, Asian American or Pacific Islander n=2, mixed or multi-racial n=2), participants who were neither white nor African American were combined into an “other” race category.

Reporting and Concordance Measures

In the analysis for each substance, days were only included for which both an ACASI recall response and an EMA text response were present. Such that, if a participant did not complete the text series on a given day, did not complete an ACASI, or if study visits were scheduled more than 14 days apart (resulting in gaps in ACASI recall), concordance could not be assessed due to missing data. These days were therefore excluded in the concordance analysis.

Pooled for the entire sample, the total number and proportion of days on which methamphetamine use, alcohol use, and binge alcohol use were reported were calculated for both ACASI and EMA. McNemar’s tests were used to assess differences in reported frequencies of methamphetamine, alcohol, and binge alcohol use between EMA text and ACASI modified timeline follow-back data.

The total number and proportion of days for which ACASI and EMA responses matched for each substance (ie, concordant responses) were calculated. This included both concordant positive responses (ie, days that methamphetamine use, alcohol use, or binge alcohol use were reported through both ACASI and EMA) and concordant negative responses (ie, days that no methamphetamine use, no alcohol use, or no binge alcohol use was reported through both ACASI and EMA). We also calculated the phi coefficient, which is equivalent to Pearson’s correlation coefficient when applied to binary variables, between ACASI and EMA responses.

Multivariable Analysis

For our primary multivariable analysis, we used multivariable logistic regression models, fit with generalized estimating equations that accounted for clustering within-subject, to assess the likelihood of having concordant responses on a given day for methamphetamine use, alcohol use, and binge alcohol use. All models included demographic and behavioral characteristics as time-invariant covariates and the number of days elapsed since the baseline visit as a time-varying covariate. Overall, 3 models (one for each substance-specific concordance outcome) included a time-varying binary covariate indicating whether or not the relevant substance was reported by EMA text on the day being assessed. Each substance-specific model included only the time-varying EMA text response covariate for the substance being assessed in that model (ie, methamphetamine, alcohol, or binge alcohol).

In sensitivity analyses, 3 alternative models (one for each substance-specific concordance outcome) included a covariate indicating the number of alcohol beverages reported by EMA text on the day being assessed. In the model assessing concordance of methamphetamine use reporting, both the binary covariate indicating any methamphetamine use and the continuous covariate indicating the number of drinks consumed were included. In the models assessing concordance of alcohol and binge alcohol use reporting, only the covariate indicating the number of drinks consumed was included. All other covariates in these 3 models were the same as in the primary models.

Results

Study Sample Characteristics

As presented in Table 1, our study sample of 30 participants was racially/ethnically diverse: 12 participants (40%) were white, 9 (30%) were African American, and 9 (30%) identified as any other race/ethnicity. The mean age was 43 (SD 9.3) and 26 (87%) had completed at least some college education, with 9 (30%) having a Bachelor’s degree or higher. The average number of days of substance use in the 4 weeks prior to baseline was 6 (SD 4.6) for methamphetamine, 13 (SD 7.6) for alcohol, and 7 (SD 7.2) for binge alcohol.

ACASI and EMA Text Compliance

All compliance measures are reported in Table 2. In total, 30 participants completed 143 (95%) biweekly ACASI surveys out of the intended 150. Twenty-four participants (80%) completed all 5 intended surveys, 5 participants (17%) completed 4 surveys, and 1 participant (3%) completed 3 surveys. Because initiation of daily text messages did not begin until the day after the participants’ first ACASI (with the exception of 1 participant who initiated daily text messages early and provided 4 days of EMA recall prior to their first ACASI), all but these 4 recall responses from participants’ first ACASI surveys were not relevant to this analysis. Across the 30 participants and including the aforementioned 4 additional days, a total of 1469 (mean per participant 49, SD 7.7) days of methamphetamine use reporting and 1462 (mean per participant 49, SD 7.9) days of alcohol and binge alcohol use reporting were collected via participants’ noninitial ACASI surveys.

Of the 1469 and 1462 days of ACASI reporting for methamphetamine and alcohol/binge alcohol use, respectively, EMA text responses were collected on 994 days (68%) for methamphetamine use, 987 (68%) for alcohol use, and 981 (67%) for binge alcohol use. For each substance, EMA text reporting was not available for 156 (11% for each substance) days due to delayed initiation or technical issues and 319 (22% for each substance) days due to participant nonresponse. Of days for which ACASI reporting was available, the average number of days of EMA text data available was 33.1 for methamphetamine use (range 10-55, SD 13.8), 32.9 for alcohol use (range 10-55, SD 13.8), and 32.7 for binge alcohol use (range 10-55, SD 13.9). As a percentage of the days of ACASI data per participant, these numbers correspond to 67.2% for methamphetamine use (range 19-98, SD 24.2), 67.1% for alcohol use (range 19-98, SD 24.1), and 66.6% for binge alcohol use (range 19-98, SD 24.2).

Reporting and Concordance Measures

A total of 994 days were included in the analysis for methamphetamine use, 987 for alcohol use, and 981 for binge alcohol use (see Figure 1). There were significant differences in the proportion of days on which methamphetamine and alcohol use were reported by text versus ACASI (P<.05). The frequencies reported by EMA text and ACASI, respectively, were 20% and 11% for methamphetamine use (P<.001) and 39% and 35% for alcohol use (P=.005). There was no significant difference in the proportion of days on which binge alcohol use was reported by text versus ACASI (P=.58). Of the concordant responses for each substance, the majority were reports of no substance use (92% for methamphetamine use, 67% for alcohol use, and 87% for binge alcohol use). The phi coefficients were 0.35 for methamphetamine use, 0.49 for alcohol use, and 0.47 for binge alcohol use.

‎View this figure

Multivariable Analysis

The results of the generalized estimating equation fitted multivariable logistic regression models are presented in Tables 3-6. In the primary model assessing methamphetamine reporting (Table 3), text reporting of any methamphetamine use was negatively associated with daily concordance in methamphetamine reporting (adjusted odds ratio 0.06, 95% CI 0.04-0.10). African American (0.15, 0.05-0.43) and other participants of color (ie, Hispanic/Latino, Asian American and Pacific Islander, mixed or multi-racial individuals) (0.34, 0.12-1.00) were also less likely to have daily concordance in methamphetamine reporting compared to white participants. Participants who had graduated from college had a greater odds of methamphetamine reporting concordance (6.79, 1.84-25.04) compared to those with no college experience. In the sensitivity analysis that included both the text reports of any methamphetamine use and number of alcoholic drinks, number of drinks was not independently associated with methamphetamine reporting concordance and the coefficient estimates of all other covariates were not significantly different than those in the primary model (P=0.69, data not shown).

In the primary model assessing alcohol reporting (Table 4), text reporting of any alcohol use was negatively associated with daily concordance in reporting of alcohol use (0.50, 0.34-0.73). African American participants were less likely to have daily concordance in alcohol use reporting compared to white participants (0.18, 0.06-0.57). In the sensitivity analysis that included the number of alcoholic drinks reported by text, number of drinks was not independently associated with alcohol reporting concordance and coefficient estimates of the other covariates were not significantly different than those in the primary model (P=0.11, data not shown).

In the primary model assessing binge alcohol reporting (Table 5), text reporting of any binge alcohol use was negatively associated with daily concordance in reporting of binge alcohol use (0.27, 0.17-0.42). In the sensitivity analysis that included the number of alcoholic drinks reported by text (Table 6), number of drinks was negatively associated with concordance in binge alcohol use reporting (0.89, 0.84-0.94). Also, in this model, African American participants were less likely to have daily concordance in binge alcohol use reporting compared to white participants (0.25, 0.07-0.87).

Discussion

Principal Findings

We found that substance use was reported more frequently using daily text messaging compared to retrospective questionnaires, with the greatest differential in the reporting of methamphetamine use. Daily concordance between EMA and questionnaire self-reports was not homogenous across participants; specifically, congruent reports were less likely among non-white participants and more likely among those who had graduated from college. Lastly, on days in which each substance was reported to have been used via EMA text response, use was significantly less likely to be reported in the ACASI questionnaire. These findings have implications for both the design and interpretation of substance use research among MSM.

In general, EMA captured more frequent substance use than ACASI. For example, methamphetamine use was reported nearly twice as frequently and alcohol use roughly 14% more frequently via EMA as compared to ACASI. Participants in our study were not compensated differentially based on their EMA responses; in addition, EMA text exchanges could be expedited by reporting no substance use. Therefore, participants had no material or time-related incentive to over-report their substance use by EMA text. Both ACASI and EMA may minimize concerns related to social desirability bias through the use of electronic, self-administered questionnaires as opposed to in-person interviews [23,24]. However, the frequent, repeated assessments of EMA methods are designed to minimize the duration of participant recall and reliance on autobiographical memory, which can be distorted even after relatively short intervals [6,7]. For these reasons, we hypothesize that daily EMA texting may provide a more accurate and reliable assessment of substance use behaviors compared to questionnaires with longer recall periods, such as the ACASI, or traditional in-person interviews. In addition to potential advantages related to data quality, EMA methods can ease the burden of data collection on study participants by allowing them to report information conveniently and privately during their daily routines and using their own personal devices.

In our multivariable analysis, participants of color were less likely to report concordantly between EMA and ACASI, for both methamphetamine and alcohol use. In prior studies comparing self-reported data to biological tests of substance use, African Americans were more likely to underreport substance use [25-29]. This may be the result of either a lower level of trust and greater reluctance among African Americans to disclose sensitive information to someone of different race, or cultural variations in the interpretation of survey questions [30-32]. African American participants also reported a higher frequency of substance use by EMA text compared to ACASI, suggesting that text-based EMA methods may be more acceptable to this population and may help to mitigate any systematic underreporting among this racial/ethnic subgroup.

We also found that the likelihood of daily concordance was significantly less for days during which participants reported substance use by text. In the case of binge alcohol use, likelihood of daily concordance also decreased as the number of drinks reported by text increased. These findings could be a function of either the acute and residual effects of each substance (as suggested by the association between number of drinks and binge alcohol use reporting concordance) or a greater difficulty in recalling exact temporal details of behaviors that occur more frequently. Although we were not able to assess the validity of our self-reported measures, several studies have demonstrated the validity of EMA reporting in relation to objective biochemical markers of cocaine, tobacco, and alcohol use [17,33,34]. If the EMA-collected data were an accurate representation of substance use behaviors, this finding suggests vulnerabilities with ACASI-collected substance use data in analyses where substance use is the exposure or outcome of interest. In this case, accurate recall and reporting of substance use behaviors would differentially depend on whether or not the participant actually used the substance, which would systematically bias any analytic findings (ie, recall bias) [35].

Limitations

This study has several limitations. First, our small sample size limits both our statistical power to identify significant correlates of daily reporting concordance and our ability to explore differences across a wider array of racial/ethnic groups. Secondly, our convenience sample may not be generalizable to all substance-using MSM. We also had no day-to-day objective measures against which to validate either the EMA or ACASI self-reported measures of substance use; therefore, we cannot make conclusions as to the validity of either method. In addition, daily assessments of the number of drinks consumed in the previous day have been shown to elicit lower reported quantities compared to real-time hourly reporting [36]. Thus, although our assessments are mainly concerned with any substance use on a given day as opposed to the specific quantity consumed, our daily text message measure may be affected by recall bias. Finally, our analyses were restricted to days for which both EMA and ACASI data were available, which may have biased our results if EMA nonresponses or missed ACASI interviews were associated with substance use and likelihood of concordance. Although there were high rates of compliance for ACASI Surveys (95% overall), overall text compliance was modest (68% overall for each substance). An assessment of predictors of EMA text message engagement among the current study sample found that white participants had higher response rates than participants of color, and those who had at least some college education were less likely to stop responding to texts for a prolonged period [37]. This variability in compliance among different demographic groups may further affect the generalizability of the data assessed in this analysis. It is important for future studies to assess the characteristics, concordance, and validity of these data collection methods among samples with elevated reporting compliance.

Conclusions

Our findings have important implications for the collection of self-reported data in substance use research, particularly among MSM. We found that methamphetamine and alcohol use were reported more frequently with daily EMA text compared to retrospective ACASI, concordance varied among different racial/ethnic subgroups and level of education, and reported substance use in EMA text was associated with lower daily concordance with retrospective ACASI. These findings suggest that EMA methods may provide more complete reporting about frequent, discrete behaviors such as substance use, which may be particularly valuable among subgroups more likely to underreport such behaviors.