Internet and mobile phone technology have opened new ways to deliver health-related counseling and therapy making health care more accessible to people, especially for those who suffer from chronic illness and live far from hospitals and qualified therapists. Several advantages over face-to-face treatments such as convenience, reduced cost, and the ability to adjust plans and feedback to a participant’s individual needs are reported [1]. Treatments delivered by the internet can also enhance health literacy and health-related knowledge and support people to cope with their health problems [2]. Challenges also exist, such as technological problems and absence of face-to-face interaction. When transforming face-to-face interventions into online interventions, there is a need to evaluate if, and how, the principles of the intervention are applied in the new modus, but only a few studies in this field are reported [3]. To avoid Type I (ie, reject the null hypothesis) and Type II (ie, reject the false null hypothesis) errors, due to misinterpretation of intervention results, it is essential to plan, monitor and evaluate Treatment Fidelity (TF). The National Institutes of Health (NIH) established a Behavior Change Consortium (BCC) and defined TF as “the methodological strategies used to monitor and enhance the reliability and validity of behavioral interventions.” TF is a continuous evaluation of different aspects of the intervention essential to enhance reliability and validity of the independent variables. A high level of TF assures that the treatment protocol is performed as intended. Although approaches to establish TF in Web-based interventions differ from those used in interventions delivered in person, the same fidelity evaluation requirements need to be applied [4].

The NIH BCC also developed a TF protocol with the following 5 principal areas: (1) study design that builds on strategies to ensure that the stated hypotheses can adequately be tested in relation to the underlying theory and clinical processes, (2) provider training to ensure that all treatment providers have been satisfactorily trained to deliver the intervention, (3) delivery of treatment that build on strategies, for example treatment monitoring, to ensure that the intervention is delivered as intended, (4) treatment receipt strategies that involve assessing and optimizing the degree to which the participant understands and demonstrates knowledge to use treatment skills, and (5) treatment enactment that involves assessing and optimizing of the degree to which the participant applies the skills learned in treatment in daily life. [5,6]. According to the results of a study performed by Eaton and colleagues [4], the NIH BCC Treatment Fidelity Guidelines were relevant when establishing TF for online delivered interventions.

The purpose of this study was to investigate the therapists’ adherence [7] to the treatment protocol and participants’ and therapists’ experience with the mobile phone delivered intervention with electronic diaries (e-diaries) and written feedback based on Acceptance and Commitment Therapy (ACT) for persons with Diabetes Mellitus Type 2 (DMT2). Based on this investigation the 5 areas of TF recommended by NIH BCC were evaluated.

In addition to the recommendations from the NIH BCC, it is important to emphasize that an evaluation of TF of technology-based behavioral interventions carries additional challenges due to their dynamic and highly individualized nature, and elements associated with such characteristics need therefore to be evaluated as well [8]. By evaluating fidelity, it is possible to investigate if theory-based processes of the intervention are the primary mechanism of change outcomes and if they allow for precise replication and comparison amongst interventions [9].

ACT is referred to as the third generation of Cognitive Behavior Therapy (CBT). Based on Functional Contextualism Philosophy and Relational Frame Theory [10], the objective of ACT is to improve patients’ functioning and quality of life by increasing psychological flexibility, referred to as the ability to behave in accordance with life values and long-term goals also when interfering thoughts, emotions, and bodily symptoms are present [11]. Psychological flexibility can be achieved as the result of the treatment combining the 6 processes of ACT (see Table 1). These processes are overlapping and interrelated and can be introduced in different orders [10-12]. In most ACT-based treatments, exercises and metaphors are frequently used to discuss behaviors that may appear counterintuitive [13,14].

ACT shares critical features with traditional CBT approaches, such as behavior activation. Also, differences exist, as acceptance and cognitive defusion strategies are relatively unique for ACT [11,15]. The use of therapy based on theory, such as ACT, increases the chance that interventions aiming to support self-management in chronic illness will be effective. Clinical psychologists and health care professionals in related disciplines also play a vital role in the treatment of people with long-term conditions [16]. ACT has been evaluated in several randomized controlled trials for persons with DMT2 and other chronic conditions, and results confirm the utility of this approach for improving health outcomes [14,17-20].

The intervention concept, involving the use of mobile technology to deliver the treatment based on behavioral therapy in a written format, was developed and tested first in the Netherlands in persons with irritable bowel syndrome (IBS) [21]. Later the intervention concept was refined and tested in Norway in persons with chronic widespread pain (CWP) and DMT2, respectively [22,23]. The IBS and CWP studies were randomized controlled trials (RCTs), whereas the DMT2 was a pilot. The RCTs showed positive results concerning catastrophizing, acceptance, and illness impact at a 3 and 5-month follow-up, respectively [21,22]. The participants’ subjective experiences of the interventions in all studies were mostly positive. A summarized description of these studies is presented elsewhere [24]. Recently, a fidelity examination of the CWP study was published, showing a high level of treatment integrity [25]. We argued in this paper that the methodology also could be applied to other similar intervention concepts. Due to the significance of TF in behavior change interventions, and the need for more studies in this area, the present study aims to investigate TF of the DMT2 pilot study.

In this section the strategies applied in the DMT2 pilot study aimed to ensure the 5 areas of TF recommended by NIH BCC, will be presented. These areas are: (1) study design, (2) provider training, (3) delivery of treatment, (4) treatment receipt, and (5) treatment enactment.

An investigation of the therapist's adherence to the treatment protocol and participants’ and therapist's experience with the intervention was done to assess the TF. Further, a checklist, containing the list of criteria (25 items) developed by Borrelli and colleagues [28], was applied. The 25 items are divided into the 5 TF categories (Design, Training, Delivery, Receipt, and Enactment) given the possibility to evaluate all the 5 areas of fidelity recommended by the NIH BCC. The authors of this checklist defined “high treatment fidelity” as those studies that have kappa=.80 or greater proportion adherence to their checklist across all strategies.

To complete the fidelity checklist, in addition to the results presented in this section, the overview of the DMT2 pilot study (Table 2) and its results (Multimedia Appendix 4), were evaluated. The purpose was to answer questions about treatment design, treatment receipt. and treatment enactment. Table 3 shows that 4/25 (16%) items of the fidelity checklist were not applicable to this study due the design DMT2 being a single arm pilot study. A total of 20/21 (95%) remaining items of the fidelity checklist were identified in the present study. These results, according to Borrelli and colleagues [28], indicate that the DMT2 study had a high level of TF.

The qualitative analysis resulted in a refinement of a complete coding scheme that was previously developed in a comparable study [25]. The refined coding scheme included 12 codes, reflecting six ACT-processes (values, committed action, contact with the present moment, self as context, acceptance, and cognitive defusion) and 6 motivation and communication strategies (advice, empathic statements, stimulate participation, general information and educational information), see Figure 1. The ACT process, self as context, was not present in the data material of the previous pain study [25]. This previous study also identified a motivation and communication code, called creative communication, which was not present in the data of the current study.

The refined coding scheme was used to refine the codebook with definitions (see Table 4). The 251/251 (100%) feedback text messages, which were divided into 722/722 (100%) text segments, were coded based on the refined coding scheme and codebook. Table 5 illustrates examples of text segments and codes.

The interrater reliability was calculated based on 336 text segments, delivered to 6/11 (55%) included participants. The number of observed agreements was 291/336 (86.61%) of the observations). The number of agreements expected by chance was 45.9/336 (13.65%) of the observations), resulting in kappa=.85 (95% CI 0.80-0.89) and SE 0.02. Table 6 displays the distribution of the codes within the dataset.

Out of the total text segments coded, 240/722 (33.2%) were ACT-consistent and 482/722 (66.8%) were coded as motivation or communication codes. All 11 participants received text segments that represented the V and CA codes. Of this 7/11 (64%) received text segments representing the CD code, 6/11 (55%) the AC code, 5/11 (45%) the PM code and 3/11 (27%) the SC code (Table 7).

To increase treatment integrity and consequently the validity of behavioral interventions and implementation, assessment of TF is necessary. The methods of implementing and evaluating TF need to be adjusted for Web-based technology interventions due to the variability within and across these interventions [6]. In this DMT2 pilot study, the 5 areas of TF recommended by NIH BCC were applied. Another TF framework that could have been applied in this study is described by Dabbs and colleagues [8]. Their TF framework was developed for Web-based technology-interventions and also recommends the evaluation of participant’ acceptance of the intervention using the Technology Acceptance Model scales [35]. The Acceptance Model scales were not applied in the DMT2 pilot study. Still, owing to the importance to investigate the technology acceptance for the fidelity evaluation in mHealth interventions [8], several questions about this theme were included in the self-reported questionnaire developed to evaluate the DMT2 pilot study [23]. The positive results regarding participants’ acceptance of the used technology reported in the DMT2 pilot study may have contributed to the positive results of the current study regarding treatment receipt and enactment.

The evaluation of the TF, in the current study, was based on the analysis of the delivered feedback messages and by assessing participants’ and therapist's experience with the intervention. The analysis of the 251/251 (100%) written feedback messages was possible by dividing them into 722/722 (100%) text segments and subsequent coding of each segment. A comparison of the refined coding scheme and codebook with the ones developed in a previous study [25], revealed 2 differences. The first was the presence of the self as context and the second was the absence of creative communication. This result confirms that ACT processes can successfully be delivered in a written format, including self as context. Although the ACT process self as context was not present within the provided feedback in the CWP-study, it was included as a component in the e-diaries. The use of communication strategies is a necessity in all behavior change interventions. Motivation elements contribute, along with support, to stimulate the participants to complete the intervention. In the present study, 6/7 (85.7%) codes (compared with the previous study) [25] related to motivation/communication strategies were found after the qualitative content analysis. The use of creative communication (absent in the present study) can be perceived as a therapist style of communication and may not influence the results of the fidelity analysis.

Analysis of texts has become a valuable research tool in numerous areas and coding is a crucial part of these analyses. For establishing TF, a prerequisite for such analysis is to ensure firm consistency between the text and the coding. Quality control of this consistency is essential to relate the results to the treatment [36]. In the present study, we achieved an excellent level of interrater reliability. This result confirmed that the coding scheme and the codebook were reliable tools for analyzing all feedback messages making further analyses possible. In short, the results of the feedback analyses showed that all participants received the text segments coded as “values” (V) and “committed action” (CA). These two ACT-processes are essential to build up participants' understanding of the therapeutic process. Not all participants received text segments representing all ACT codes. It is important to emphasize that although the development of feedback messages was based on ACT, this does not imply that all ACT processes are required for the treatment of each participant.

The excellent representation of the different ACT codes in the feedback messages (Table 7) confirms that these were developed according to the ACT framework and principles (Multimedia Appendix 1). This means that the intervention was delivered according to the chosen theory and as intended, indicating that the TF regarding study design, provider training, and treatment delivery was effective.

The use of ACT-elements may have contributed to the positive results achieved in the DMT2 pilot study [23]. As the impact of Web-based interventions increases, theory is used more extensively [16]. According to Riley (2011), theories may need to be revised to fit the new format of Web-based and mobile interventions [37]. Our experience is that applying ACT in a written format is well suited for Web-based interventions and enables and facilitates the analysis regarding TF. A total of 806/2231 (36%) of the coded text-segments in the previous study [25] were ACT codes against 240/722 (33%) in the present study. These results can indicate that communication and motivation strategies are essential elements of the feedback messages to deliver ACTs therapeutic processes in a written format. This hypothesis needs further investigation.

Despite the positive effects achieved in ACT-based interventions, it is common for the achieved effects to diminish over time, with the return of old cognitions and activity patterns [38,39]. Other studies also show that the long-term effect of cognitive behavioral therapies is generally limited [40,41]. This may indicate a need for more continuity and a longer duration of the intervention to support self-management in people with chronic illness. The results regarding participants’ (self-reported questionnaire and interviews) and therapist's experience with the project indicate that the therapist was capable and qualified and that the participants acquired knowledge regarding their diabetes, treatment skills and how to apply this knowledge in their daily life (treatment receipt and enactment). Despite the positive results, the use of a self-reported questionnaire to assess the therapist’s experience with the project would have been more reliable. The fidelity evaluation was done by applying the checklist developed by Borelli and colleges and showed a high level of TF. This fidelity assessing tool was based on the 5 areas of TF recommended by NIH BCC being suitable for this study purpose supporting the findings of a previous study [4].

TF in online delivered intervention is an emerging area of research [42]. Therefore, methods of evaluating TF adapted to the different kinds of online interventions is needed. In the present study, the reliable and precise evaluation of therapist's adherence was possible due to a coding scheme and a codebook developed in a previous study [25]. A similar method was applied in a recent study aimed to develop and evaluate a scale for assessing therapist fidelity in Web-delivered cognitive behavior therapy [42]. The concept of using a coding system to evaluate a behavior intervention delivered in a written format provides input for future studies.

The coding scheme applied in the present study showed to be reliable and valid to investigate several areas of TF. In this study, a coding scheme and a codebook for ACT-oriented Web-based personal feedback developed in a previous study were refined maintaining the high level of interrater reliability. Results based on the refined coding scheme showed that the feedback provided complied with core ACT principles. The reliability of the investigation of the therapists’ competence could be improved by the development of self-reported questionnaires to be filled in by the therapists.

The fact that the DMT2 intervention required e-diaries 3 times daily could be perceived as potential burden for the participants. The high percentage of dropouts 4/15 (26.7%) indicates that participants believe this as well. The DMT2 pilot study percentage of dropouts is equivalent to results that are reported in a comparable study [22]. However, it is important to emphasize that the participants who left the DMT2 pilot study never started the intervention. Despite being favorable to the intervention at the first meeting, they believed that to participate in the project would be too time-consuming, especially when completing three e-diaries a day. In contrast, all participants who started completed the intervention. Their experience with completing the diaries was assessed and reported elsewhere [23]. Only 2/11 (18.2%) participants would prefer fewer e-diaries and questions. A comparable study reported a higher level of participants burden, 3/6 (50%) participants considered the number of questions in each diary to be too high [43]. The number of questions in the DMT2 was fewer compared to this similar study, and this can explain the better results. However, it is essential to bear in mind the need to reduce the burden on the respondents as much as possible, as it has an adverse effect on the respondents’ motivation and thereby impacts negatively on their response quality and TF [44]. It is essential to identify the ideal number of diaries and questions delivered per day to decrease the participant burden and subsequently reduce the risk of dropouts. Furthermore, it is important to gain an in-depth understanding of participants’ reasons for early withdrawal. We encourage other researchers to consider this when planning large-scale studies requiring a large sample size and TF evaluations.

Regarding providers burden, the DMT2 pilot study revealed that it was time-consuming for the therapist to give feedback. The system did not show historical information as a summary, and it was necessary to navigate through several pages to get needed information. Also, the therapist had to look at the feedback history to avoid repeating information. After the first month, when a feedback bank was created, the time used to formulate the feedback text messages was reduced to 15–20 min, a level that providers reported as suitable [23].

Current Web-based interventions, which utilize written personalized therapeutic feedback tend to be time-consuming for the health personnel involved [22,23]. The possibilities within information technology regarding automation of parts of the feedback messages should, therefore, be explored [25]. As discussed later in the practical implication section, the automation of the feedback messages using algorithms would be possible to secure the TF in these kinds of interventions regarding study design, monitoring and treatment delivery. The potential to use artificial intelligence to analyze the relation between diaries, feedback and results continuously, is a fascinating area for future research.

The results of the present study showed that provider training was done according to the NIH BCC recommendations. The small number of participants (n=11) and the therapist (n=1), enabled the training and monitoring of the therapist as described in the method section. For a large-scale intervention, a systematic TF project should include standardized therapist training program with a focus on the appropriate diagnosis and the appropriate treatment approach. The standardization allows for the specific skills required for intervention delivery to be improved and accentuated within providers regardless of inherent differences [6]. The therapists should also attain a certification after a comprehensive training process. All the therapists should have a bachelor degree in health care like medicine, psychology, physiotherapy or nursing because of the physical and mental symptoms of the participants’ disease and the nature of the intervention (formulation of feedback messages based on information from the e-diaries and ACT) [24]. The monitoring process could be done by midterm randomized audits.

As shown in Table 2, the results of the DMT2 pilot study were positive. As expected, due to the small sample size the results were not statistically significant. Because the results of the present study indicate a high level of TF it is possible to assume the DMT2 pilot intervention results were reliable. This again provides a firm basis to recommend a RCT. This can provide a more qualified answer regarding whether the offered intervention will help more people with DMT2 to achieve self-management and increase their quality of life.

As mentioned in the current paper, by evaluating fidelity, it is possible to investigate if theory-based processes of the intervention are the primary mechanism of change outcomes and if they allow for precise replication and comparison amongst interventions.

The main contribution of this paper to the use of Web-based mobile technology in health literature regarding TF is the method of analyzing therapist’s adherence to the treatment protocol. The method presented in the current study allows more precise results when compared to the methods that are commonly used in TF studies based on observation and interpretation from another professional. Results that are based on the evaluation of people may carry several biases (ie, personal experience, humor, knowledge) that may influence the results.

With the method used in the present study, it was possible to demonstrate that the intervention based on ACT was performed as proposed, confirming that the chosen theory was the primary mechanism of change outcomes. However, the use of the method of evaluation of therapists’ adherence proposed in the current study is limited to a treatment delivered in a written format. To the best of our knowledge, this is the second study that analyses therapists’ adherence to the study protocol using this method.

A reliable and valid coding system as defined in this study is essential for exploring therapeutic change processes in this type of mobile phone-delivered interventions. The coding scheme has the potential to lay a foundation for the automation of feedback messages, together with a bank or database of these messages. Automatic feedback could be generated from a database and combined with individualized feedback if the diaries indicate this to be required. The intervention could then be developed as an application for mobile phones, reducing therapist time and costs. Making such an application available as support for clinical practices and in maintenance treatments would help treat people who do not have easy access to health care services. The use of new and innovative technology to make this kind of behavior change interventions more effective, while still taking care of the patients’ individual needs, suggests an exciting future area of research. The first step would be to develop and test the concept of automation in an RCT. In a further development of the intervention, it would be interesting to explore the effects of using more technologically advanced capabilities to gather rich and complex data. This could include sensors to measure activity levels and context-triggered diary questions [37,45]. Also, the automated feedback on registered data could be provided in progress charts, graphs, and summaries. Educational information could be given by interactive animations or videos [46].

Web-based interventions are becoming increasingly popular and appear to be an essential and cost-effective supplement to everyday health care in the near future. The evaluation of TF is essential to interpret the results of interventions and its underlying working mechanisms. A different methodology is used to deliver interventions via the internet. To achieve TF, the evaluation methods must be adjusted to each kind of online treatment. By doing this, the TF in Web-based interventions can be maintained and monitored reliably.