This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Web-based interventions are becoming an alternative of treatment aimed to support behavioral changes and several advantages over traditional treatments are reported. New ways of delivering an intervention may result in new challenges regarding monitoring of treatment fidelity (TF) which is essential to ensure internal and external validity. Despite the importance of the theme, only a few studies in this field are reported.
To examine TF of a mobile phone delivered intervention based on Acceptance and Commitment Therapy (ACT) with electronic diaries and written situational feedback for persons with diabetes mellitus type 2, the recommendations from the Behavior Change Consortium (BCC) established by The National Institutes of Health (NHI) were applied. To analyze fidelity, they recommend 5 areas to be investigated (1) design of the study, (2) provider training, (3) delivery of treatment, (4) receipt of treatment, and (5) enactment of treatment. In the current study, these areas were examined based on the analysis of therapists’ adherence to the treatment protocol and participants’ and therapists’ experience with the intervention.
To investigate the therapists’ adherence to the treatment protocol, a total of 251 written feedback text messages were divided into text segments. Qualitative thematic analyses were then performed to examine how ACT and other therapeutic processes were used in the feedback by the therapists. For the therapists’ and participants’ experience analysis, participants answered a self-reported questionnaire and participated in 2 interviews. The therapists continuously reported their experiences to the researcher responsible for the project.
The results show high adherence to the TF strategies 20/21 (95%) applicable items of the fidelity checklist recommended by NHI BCC were identified in the present study. Measured provider skill acquisition post-training was the only item absent in the fidelity checklist. The results also show high therapists’ adherence to the treatment protocol. All ACT processes (values, committed action, acceptance, contact with the present moment, self as context and cognitive defusion) were found in the coded text segments of the feedback in addition to communication and motivation strategies. For 336/730 (46%) of total possible text segments coded independently by 2 researchers, the interrater reliability measured by Cohen’s kappa was .85. The evaluation of participants’ and therapists’ experience with the intervention was generally positive.
Based on the analyses of therapists’ adherence to the treatment protocol grounded by ACT-principles and participants’ and therapists’ experience with the intervention, the 5 areas of TF recommended by NHI BCC were analyzed indicating a high level of TF. These results ensure an appropriate level of internal and external validity of the study and reliable intervention results and facilitate a precise replication of this intervention concept. Web-based psychological interventions to support people with chronic conditions are becoming increasingly more common. This study supports the results from a previous study which indicated that ACT could be reliably delivered in a written web-based format.
ClinicalTrials.gov NCT01297049; https://clinicaltrials.gov/ct2/show/NCT01297049 (Archived by WebCite at http://www.webcitation.org/70WC4Cm4T)
Internet and mobile phone technology have opened new ways to deliver health-related counseling and therapy making health care more accessible to people, especially for those who suffer from chronic illness and live far from hospitals and qualified therapists. Several advantages over face-to-face treatments such as convenience, reduced cost, and the ability to adjust plans and feedback to a participant’s individual needs are reported [
The NIH BCC also developed a TF protocol with the following 5 principal areas: (1) study design that builds on strategies to ensure that the stated hypotheses can adequately be tested in relation to the underlying theory and clinical processes, (2) provider training to ensure that all treatment providers have been satisfactorily trained to deliver the intervention, (3) delivery of treatment that build on strategies, for example treatment monitoring, to ensure that the intervention is delivered as intended, (4) treatment receipt strategies that involve assessing and optimizing the degree to which the participant understands and demonstrates knowledge to use treatment skills, and (5) treatment enactment that involves assessing and optimizing of the degree to which the participant applies the skills learned in treatment in daily life. [
The purpose of this study was to investigate the therapists’ adherence [
In addition to the recommendations from the NIH BCC, it is important to emphasize that an evaluation of TF of technology-based behavioral interventions carries additional challenges due to their dynamic and highly individualized nature, and elements associated with such characteristics need therefore to be evaluated as well [
ACT is referred to as the third generation of Cognitive Behavior Therapy (CBT). Based on Functional Contextualism Philosophy and Relational Frame Theory [
ACT shares critical features with traditional CBT approaches, such as behavior activation. Also, differences exist, as acceptance and cognitive defusion strategies are relatively unique for ACT [
Acceptance and Commitment Therapy subprocesses.
Subprocess | Definition |
Values | The deeply meaningful elements in a person's life. Values concern the ideals we have and how we want to live. |
Committed action | Specific and concrete action plans guided by one's values, which also takes into account and anticipate barriers on the way. |
Acceptance | Openness to experience, urges, emotions, and thoughts allowing them to come and go without a struggle. |
Contact with the present moment | Being fully aware of the psychological and environmental events with openness, interest, receptiveness and without judgment. |
Self as context | Allows people to be aware of psychological content without linking it to their personal identity. |
Cognitive defusion | A process where people learn how to gain a perspective regarding one's thoughts, and thus manage to see their own thoughts as an outside observer and therefore avoid being affected by them. |
The intervention concept, involving the use of mobile technology to deliver the treatment based on behavioral therapy in a written format, was developed and tested first in the Netherlands in persons with irritable bowel syndrome (IBS) [
The current study has a mixed-method design, encompassing both qualitative and quantitative data aimed to investigate the TF of a mobile phone delivered intervention with e-diaries and written situational feedback based on ACT to support persons with DMT2.
In this section the strategies applied in the DMT2 pilot study aimed to ensure the 5 areas of TF recommended by NIH BCC, will be presented. These areas are: (1) study design, (2) provider training, (3) delivery of treatment, (4) treatment receipt, and (5) treatment enactment.
A therapist wrote the feedback text messages in the DMT2 pilot study. She had a background in health care sciences (nursing) and attended a 3-day theoretical and practical course in ACT for clinical purpose held by Professor Steven Hayes, the founder of ACT. The therapist also received training, by the research coordinator, in how to analyze the information contained in the daily e-diaries and to write the feedback text based on the e-diaries and ACT.
The training in writing of feedback text messages lasted for 1 month. The feedback texts were provided for 2 persons with DMT2 who participated in a pre-pilot during the development phase of the intervention. These feedback messages were composed in cooperation with a multidisciplinary group including, a diabetes researcher, a diabetes nurse, a communication researcher, and a nutritionist.
Overview of the diabetes mellitus type 2 pilot study by Nes et al [
Protocol | Description |
Aim | To develop and test the feasibility of a mobile phone delivered intervention based on ACT to support self-management in persons with diabetes mellitus type 2. |
Design | This was a feasibility pilot study. The intervention lasted for 12 weeks and started with a personal instructional meeting followed by daily e-diaries and feedback via a mobile phone. At a scheduled diary-completion time and to access the feedback, the participant received a short message service (SMS) text message with a link to a secure website. In this website, the diary questions could be answered and submitted back to the server, and the available feedback message could be read. There were 4 audio files with mindfulness and relaxation exercises available on the mobile phones. |
Diaries | The participants completed the e-diaries 3 times daily. The diaries included 16-19 questions chosen for supporting self-monitoring (level of blood glucose, diet, medicine, and achieved activities) and awareness of health behavior, thoughts, feelings and applied self-management strategies. Most of the questions were answered by choosing predefined alternatives or by scoring on a 6-point Likert scale. The diaries also included a comment field giving participants the opportunity to write a short personal message to the therapist. |
Feedback | A therapist had immediate access to submitted diaries and used the situational information to formulate personalized feedback based on Acceptance and Commitment Therapy. The purpose of the diaries and the situational feedback was to stimulate self-management. Daily written situational feedback (except weekends) was given during the first month followed by weekly feedback during the next 2 months. A multi-disciplinary group supported the development of the feedback during the first period of the study. The therapist used information from the 3 latest submitted diaries. There was no limitation on the length of the feedback. |
Therapist | A nurse that was trained in Acceptance and Commitment Therapy. |
Setting and recruitment | The intention was to recruit persons with type 2 diabetes through general practitioners and to include 10-15 participants to test the feasibility of the intervention in this patient group. Because of the difficulty in recruiting participants through their general practitioners, the social network of the researchers was also informed about the project and persons were asked if they knew potential candidates. The potential candidates with type 2 diabetes that met the inclusion criteria received a letter describing the study. Those interested in participating met the responsible researcher and received additional information. After receiving complementary information, the patients who agreed to participate in the project signed an informed consent form. All patients were followed by and received standard care from their general practitioners. |
Outcomes | The primary outcome was the level of glycosylated hemoglobin (HBA1c). This blood test shows the average level of blood sugar over the previous 2 to 3 months. This indicates how well a person with diabetes is being controlled over time. |
Study Sample and Data Collection | Of the 11/15 (73.3%) participants included in the study completed the intervention. The data were collected at researcher’s and general practitioner’s office. The baseline in the first meeting with the patients (T1) and immediately after 12-week intervention period (T2). The participants were interviewed twice. The first time halfway through and the second time at the end of intervention period. |
Statistical Analysis | Descriptive statistics as means and frequencies were calculated using IBM SPSS version 18 statistical software. A descriptive summary of the information extracted from the interviews was made, the content was analyzed, and themes identified. |
Effect | Most of the participants reported positive life style changes. The response rate to daily registration entries was good and few technical problems were encountered. The mean HBA1c level the week before inclusion was 7.4% (SD 1.1%) and 6.9% (SD 0.8%) at the end of intervention. More detailed results regarding outcomes are reported elsewhere [ |
Feasibility | At the end of the intervention, the participants received a questionnaire to assess their experience with the study. The questionnaire had 5 main areas with the number of items varying from 8 to 20: (1) participation in the project (12 items), (2) use of mobile phone (20 items), (3) daily diaries (12 items), (4) the received feedback (12 items), and (5) self-management (8). The scoring range in the answers was on 5-point Likert scales from 0 ‘‘totally disagree’’ to 5 ‘‘totally agree.’’ The mean for participation in the project was 4.2 (SD 0.5), for the use of a mobile phone it was 3.3 (SD 0.2), for diaries it was 4.4 (SD 0.4), for feedback it was 4.0 (SD 0.5), and for self-management it was 3.4 (SD 0.6). The participants also answered 7 questions about the project structure. There were 2 semi-structured interviews with each participant performed. |
Conclusion | The described intervention is feasible and was evaluated as supportive and meaningful. The developed mobile phone application seems a promising tool for supporting patients with type 2 diabetes to make important lifestyle changes. |
To ensure that the treatment was delivered as intended, a framework with principles for the development of feedback based on ACT was created (see
In the first individual meeting with the potential participants of the DMT2 project, the intervention was presented and explained. For those who agreed to participate, a mobile phone and an instruction manual containing all the necessary information were provided. The researcher (AAGN) instructed each participant on how to use the mobile phone, how to complete the e-diaries and how to access and read the feedback messages. The participants also received education about the importance of identifying life values and corresponding goals according to the ACT framework. To reinforce this information, they also received a workbook with voluntary written exercises aimed to help them in this process. Contact details were included in the manual. The first week of the intervention was a run-in-period, intended to familiarize the participants with the mobile phone, the e-diaries, and the feedback. All these described practices were performed to increase the participants’ level of confidence to understand and undertake treatment-related behavioral and cognitive strategies. The fidelity of participant receipt of treatment is essential for Web-based studies due to the missing or minimal in-person contact, where many interactions rely on written communication where providers miss nonverbal cues of comprehension [
Fidelity of treatment enactment is vital to ensure that participants are regularly engaging with cognitive and behavioral skills emphasized during the treatment. Cognitive skills such as value identification, goal setting, and behavioral skills as action planning were reinforced to help participants meet their goal related to blood sugar level, exercise, and diet. The main way to stimulate the participants’ adherence to these cognitive skills was through the daily e-diaries and personalized written feedback. In the evening e-diaries, the participants evaluated if the feedback was helpful regarding recommended physical activities, diet, blood sugar level and being aware of what was important for oneself. They could also indicate if the feedback was helpful or not. If the response was that the feedback was unhelpful, the therapist stimulated the participants to use the comment field to report their needs. Based on this daily report the therapist managed to help the participants with their difficulties, thereby increasing participants’ satisfaction with feedback and treatment enactment.
An investigation of the therapist's adherence to the treatment protocol and participants’ and therapist's experience with the intervention was done to assess the TF. Further, a checklist, containing the list of criteria (25 items) developed by Borrelli and colleagues [
The first part of the investigation of TF was performed by assessing the therapist’s adherence to the therapy model by comparing the intended content of the feedback (see
The feedback messages were analyzed qualitatively using a combined deductive and inductive approach [
As a second step, the data was re-analyzed inductively using an editing organizing style to identify other therapeutic processes (ie, those not explicitly related to ACT). This iterative process involved a systematic reading of all data material where relevant observations were continuously coded and refined for further interpretation [
Interrater reliability is a statistical measure used to examine the agreement between 2 or more people on the assignment of categories of a categorical variable [
To ensure the reliability of the current study, interrater reliability was assessed in 2 phases. In the first phase, 4 researchers participated in the process. They received a decoded selection of data from 5 participants of the DMT2 study where all ACT categories supposedly were represented, and the results were compared. These results are reported in an earlier publication [
When the interrater reliability assessment produced excellent results (kappa>.75), the remaining data material (106/251, 42% feedback messages was coded by AAGN using the refined codebook.
The second part of the fidelity investigation involved the assessment of the participant and therapist overall experience with the intervention. The purpose was to evaluate treatment receipt and treatment enactment. The participants completed a questionnaire at the end of the intervention period and took part in 2 individual interviews (see
The first part of the results will be presented in accordance with the completed checklist of the 5 areas of TF recommended by the NIH BCC (Study design, Provider training, Treatment delivery, Receipt, and Enactment). Also, the results of the analyses that made it possible to complete the fidelity checklist as therapist's adherence to the treatment protocol (qualitative analysis, interrater reliability, and coding of all feedback messages) will be presented. The results of participants’ experience with the therapist and therapy and therapist’s experience with the intervention are summarized in the
To complete the fidelity checklist, in addition to the results presented in this section, the overview of the DMT2 pilot study (
The qualitative analysis resulted in a refinement of a complete coding scheme that was previously developed in a comparable study [
Assessment of treatment fidelity strategies developed by Borrelli and colleagues [
Treatment fidelity strategies | Present | Absent | Not Applicable | ||
Treatment design | |||||
Length of contact session(s) | X | — | — | ||
Number of contacts | X | — | — | ||
Content of treatment | X | — | — | ||
Duration of contact over time | X | — | — | ||
Length of contact session(s) | — | — | X | ||
Number of contacts | — | — | X | ||
Content of treatment | — | — | X | ||
Duration of contact over time | — | — | X | ||
Mention of provider credentials | X | — | — | ||
Mention of a theoretical model or clinical guidelines on which the intervention is based | X | — | — | ||
Description of how providers were trained | X | — | — | ||
Standardized provider training | X | — | — | ||
Measured provider skill acquisition post training | — | X | — | ||
Described how provider skills maintained over time | X | — | — | ||
Included method to ensure that the content of the intervention was being delivered as specified | X | — | — | ||
Included method to ensure that the dose of the intervention was being delivered as specified | X | — | — | ||
Included mechanism to assess if the provider adhered to the intervention plan | X | — | — | ||
Assessed nonspecific treatment effects | X | — | — | ||
Used treatment manual | X | — | — | ||
Assessed subject comprehension of the intervention during the intervention period | X | — | — | ||
Included a strategy to improve subject comprehension of the intervention above and beyond what is included in the intervention | X | — | — | ||
Assessed subject’s ability to perform the intervention skills during the intervention period | X | — | — | ||
Included a strategy to improve subject performance of intervention skills during the intervention period | X | — | — | ||
Assessed subject performance of the intervention skills assessed in settings in which the intervention might be applied | X | — | — | ||
Assessed strategy to improve subject performance of the intervention skills in settings in which the intervention might be applied | X | — | — |
Refined coding scheme.
Codebook with refined codes definitions.
Processes and strategies | Code | Definition | |
Values | V | Stimulate patient’s reflection on their own values and the values’ impact on their life. Helping the patient to identify the difference between goals and values. Stimulate awareness of gratitude and enthusiasm regarding feelings for increasing awareness of values. |
|
Committed action | CA | Encourage the patient to committed behavior related to their own values through reflecting on strategies related to well-planned actions, barriers and follow-up. Stimulate planning of activities in the form of value-oriented goals. |
|
Contact with present moment | PM | Stimulate breathing exercises for relaxation and variety in activities. Stimulate attention and awareness of internal and external experiences in the present moment. |
|
Cognitive defusion | CD | Stimulate awareness of thought processes instead thought content. Stimulate understanding in thought content as a result of the context in which thoughts are a product of the specific situation(s). |
|
Acceptance | AC | Encourage the patient to make active choices to act in accordance with their values, despite the difficult thoughts, emotions and physical sensations that are unpleasant, but which we cannot directly eliminate or reduce. |
|
Self as context | SC | Stimulate the patient be aware of psychological content without linking it to their personal identity, creating a sense of distance between one’s self and one’s thoughts. |
|
Behavioral support | BS | Support the patient in the change process by recognizing the patient's willingness and efforts to change behavior. Motivate the patient in the change process through the use of praise and positive words. Provide confirmation of the patient's coping strategies. Encourage the patient to be their own supporter by practicing self-talk. Use specific records from patient diary forms in order to deliver a Message. Summarize developments during the follow-up period to emphasize key elements. |
|
Advise | AD | Give specific and constructive advice for appropriate behavior or to specific situation. |
|
Empathic statements | ES | Recognize the patient's experiences and feelings, showing empathy, understanding and respect. |
|
Stimulate participation | SP | Encourage patient to provide written information to ensure the most individualized follow-up. |
|
General information | GI | Provide information of a general, not therapeutic purpose |
|
Educational information | EI | Explain scientific purpose and significance related to advice, training and or intervention |
The refined coding scheme was used to refine the codebook with definitions (see
The interrater reliability was calculated based on 336 text segments, delivered to 6/11 (55%) included participants. The number of observed agreements was 291/336 (86.61%) of the observations). The number of agreements expected by chance was 45.9/336 (13.65%) of the observations), resulting in kappa=.85 (95% CI 0.80-0.89) and SE 0.02.
Out of the total text segments coded, 240/722 (33.2%) were ACT-consistent and 482/722 (66.8%) were coded as motivation or communication codes. All 11 participants received text segments that represented the V and CA codes. Of this 7/11 (64%) received text segments representing the CD code, 6/11 (55%) the AC code, 5/11 (45%) the PM code and 3/11 (27%) the SC code (
Example of feedback text messages divided in text segments. AD: advise; BS: behavioral support; CA: committed action; EI: education information; ES: empathetic statements; GI: general information.
Feedback Text Messages Number | Code | Text segments |
3 | CA | Hi. Yesterday I wrote a little about life values. If one of your life values is to achieve better health, you may set up some goals to live by according to this life value. An example of this might be that you may eat and drink according to your physician’s recommendation. The goals should not be extensive, but narrow and feasible. The mobile phone you borrowed from the project has several utilities. Have you tried to use the software for diabetes that is uploaded there? You can, for example, set up goals regarding food and drink for the day and for the week. At the end of the day, record what you have eaten and been drinking, and the system provides feedback (smile faces), depending on how you performed in relation to your own personal goals. |
3 | EI | The software uses the terms high and low carbs. It may be a little unclear what these terms mean, so I'm going to give you a little explanation now. High carbs are carbohydrates that are rapidly absorbed by the body and rapidly increases blood sugar. Examples of drink and food that contain a lot of carbohydrates are sugary drinks like soft drinks and pasta respectively. Low carbs are carbohydrates that are slowly absorbed by body and give a slower blood sugar rise like bread with a lot of whole grains and fiber and vegetables. |
3 | GI | In the manual you received from Ann in the beginning of the project you can also find this explanation with other examples. |
3 | ES | Feel free to write in the text field if you have any questions! Have a nice day. Regards Helen. |
4 | BS | Hi. It looks to me like you have a plan for the day concerning taking of your medications, blood sugar control and eat and drink as recommended. I see that you manage to achieve these goals. That’s great! |
4 | CA | You wrote in the middle of the day yesterday that you were at work, and that you had the opportunity to do physical exercises. Do you want to increase your activity level? Yesterday I wrote about setting up goals according to your life values. The goals shall not be to extensive, but narrow and achievable. Even when you set goals that you believe are achievable, there may be barriers that impede you from reaching them. It may be helpful to think about any barriers in relation to the goals you set up to yourself. Barriers may include time, effort or different thoughts and feelings. Time Barriers are anything that prevent many people from moving toward their goals. Can you think of any barriers that impede you from reaching your goals? Challenging work and a hectic schedule are barriers that prevent people being as physically active as they want. If you experience the same, you may reflect on what strategies you can use to reach your goals, despite the fact that time is limited. |
4 | GI | A long weekend is coming up and you will not get feedback until Monday. I will take a look at the diaries you will fill out during these days, to give you feedback on Tuesday. |
4 | AD | During this period, I would suggest that you reflect on what I have written to you in the feedback text messages so far. This because reflection can lead to awareness of things that are important to you. I wish you a nice Pentecost weekend! Regards Helen |
Distribution of the codes used in the interrater reliability analyses. Dashes indicate the absence of codes. AC: acceptance; AD: advise; BS: behavioral support; CA: committed action; CD: cognitive defusion; EI: education information; ES: empathetic statements; GI: general information; PM: contact with present moment; SC: self as context; SP: stimulate participation; V: values.
Code | V | CA | PM | CD | AC | SC | BS | AD | ES | SP | GI | EI | SUM |
V | 29 | — | 3 | — | — | — | — | — | — | — | — | — | 32 |
CA | 3 | 46 | — | — | — | — | — | — | — | — | — | — | 49 |
PM | — | — | 1 | — | — | — | — | — | — | — | — | — | 1 |
CD | — | — | — | 9 | 6 | — | — | — | — | — | — | 2 | 17 |
AC | — | — | — | — | 3 | — | — | — | — | — | — | — | 3 |
SC | — | — | — | — | — | 6 | — | — | — | — | — | — | 6 |
BS | — | 2 | 2 | 3 | — | — | 75 | — | 7 | — | — | — | 89 |
AD | — | 1 | 3 | — | — | — | — | 29 | 1 | — | — | — | 34 |
ES | — | 1 | — | — | — | — | — | — | 11 | — | — | — | 12 |
SP | — | — | — | — | — | — | — | 1 | — | 7 | — | — | 8 |
GI | — | 2 | — | — | — | — | 1 | — | 3 | 2 | 45 | — | 53 |
EI | — | — | — | 1 | — | — | — | — | 1 | — | — | 30 | 32 |
SUM | 32 | 52 | 9 | 13 | 9 | 6 | 76 | 30 | 23 | 9 | 45 | 32 | 336 |
Overview of all data material coded. Dashes indicate the absence of codes. AC: acceptance; AD: advise; BS: behavioral support; CA: committed action; CD: cognitive defusion; EI: education information; ES: empathetic statements; GI: general information; P: participant; PM: contact with present moment; SC: self as context; SP: stimulate participation; TF: total feedback messages; TTS: total of text segments; V: values.
P | TF | V | CA | PM | CD | SC | AC | BS | AD | ES | SP | CC | GI | EI | TTS |
1 | 21 | 6 | 14 | — | 2 | — | — | 14 | 7 | 4 | 2 | — | 11 | 2 | 62 |
2 | 25 | 12 | 22 | 3 | — | — | — | 29 | 8 | 27 | 7 | — | 15 | 5 | 128 |
3 | 15 | 3 | 7 | — | 2 | — | 2 | 7 | 5 | 2 | 6 | — | 9 | 6 | 49 |
4 | 18 | 7 | 13 | — | — | — | — | 14 | 2 | 2 | 9 | — | 12 | 3 | 62 |
5 | 27 | 4 | 12 | — | 3 | — | 2 | 17 | 6 | 5 | 6 | — | 6 | 6 | 69 |
6 | 26 | 6 | 14 | — | 3 | — | 4 | 18 | 10 | — | 1 | — | 6 | 2 | 64 |
7 | 13 | 2 | 4 | — | — | — | — | 12 | 2 | 7 | — | — | 5 | 7 | 39 |
8 | 27 | 5 | 12 | 2 | — | 1 | 2 | 14 | 3 | 2 | 3 | — | 5 | 10 | 59 |
9 | 24 | 6 | 4 | 2 | 3 | — | 3 | 9 | 12 | 9 | 3 | — | 8 | 7 | 66 |
10 | 28 | 7 | 10 | 2 | 3 | 1 | — | 15 | 3 | 4 | 1 | — | 6 | 8 | 60 |
11 | 27 | 6 | 10 | 2 | 3 | 4 | 3 | 15 | 4 | 4 | 1 | — | 6 | 6 | 64 |
SUM | 251 | 64 | 124 | 11 | 19 | 6 | 16 | 164 | 62 | 66 | 39 | — | 89 | 62 | 722 |
To increase treatment integrity and consequently the validity of behavioral interventions and implementation, assessment of TF is necessary. The methods of implementing and evaluating TF need to be adjusted for Web-based technology interventions due to the variability within and across these interventions [
The evaluation of the TF, in the current study, was based on the analysis of the delivered feedback messages and by assessing participants’ and therapist's experience with the intervention. The analysis of the 251/251 (100%) written feedback messages was possible by dividing them into 722/722 (100%) text segments and subsequent coding of each segment. A comparison of the refined coding scheme and codebook with the ones developed in a previous study [
Analysis of texts has become a valuable research tool in numerous areas and coding is a crucial part of these analyses. For establishing TF, a prerequisite for such analysis is to ensure firm consistency between the text and the coding. Quality control of this consistency is essential to relate the results to the treatment [
The excellent representation of the different ACT codes in the feedback messages (
The use of ACT-elements may have contributed to the positive results achieved in the DMT2 pilot study [
Despite the positive effects achieved in ACT-based interventions, it is common for the achieved effects to diminish over time, with the return of old cognitions and activity patterns [
TF in online delivered intervention is an emerging area of research [
The coding scheme applied in the present study showed to be reliable and valid to investigate several areas of TF. In this study, a coding scheme and a codebook for ACT-oriented Web-based personal feedback developed in a previous study were refined maintaining the high level of interrater reliability. Results based on the refined coding scheme showed that the feedback provided complied with core ACT principles. The reliability of the investigation of the therapists’ competence could be improved by the development of self-reported questionnaires to be filled in by the therapists.
The fact that the DMT2 intervention required e-diaries 3 times daily could be perceived as potential burden for the participants. The high percentage of dropouts 4/15 (26.7%) indicates that participants believe this as well. The DMT2 pilot study percentage of dropouts is equivalent to results that are reported in a comparable study [
Regarding providers burden, the DMT2 pilot study revealed that it was time-consuming for the therapist to give feedback. The system did not show historical information as a summary, and it was necessary to navigate through several pages to get needed information. Also, the therapist had to look at the feedback history to avoid repeating information. After the first month, when a feedback bank was created, the time used to formulate the feedback text messages was reduced to 15–20 min, a level that providers reported as suitable [
Current Web-based interventions, which utilize written personalized therapeutic feedback tend to be time-consuming for the health personnel involved [
The results of the present study showed that provider training was done according to the NIH BCC recommendations. The small number of participants (n=11) and the therapist (n=1), enabled the training and monitoring of the therapist as described in the method section. For a large-scale intervention, a systematic TF project should include standardized therapist training program with a focus on the appropriate diagnosis and the appropriate treatment approach. The standardization allows for the specific skills required for intervention delivery to be improved and accentuated within providers regardless of inherent differences [
As shown in
As mentioned in the current paper, by evaluating fidelity, it is possible to investigate if theory-based processes of the intervention are the primary mechanism of change outcomes and if they allow for precise replication and comparison amongst interventions.
The main contribution of this paper to the use of Web-based mobile technology in health literature regarding TF is the method of analyzing therapist’s adherence to the treatment protocol. The method presented in the current study allows more precise results when compared to the methods that are commonly used in TF studies based on observation and interpretation from another professional. Results that are based on the evaluation of people may carry several biases (ie, personal experience, humor, knowledge) that may influence the results.
With the method used in the present study, it was possible to demonstrate that the intervention based on ACT was performed as proposed, confirming that the chosen theory was the primary mechanism of change outcomes. However, the use of the method of evaluation of therapists’ adherence proposed in the current study is limited to a treatment delivered in a written format. To the best of our knowledge, this is the second study that analyses therapists’ adherence to the study protocol using this method.
A reliable and valid coding system as defined in this study is essential for exploring therapeutic change processes in this type of mobile phone-delivered interventions. The coding scheme has the potential to lay a foundation for the automation of feedback messages, together with a bank or database of these messages. Automatic feedback could be generated from a database and combined with individualized feedback if the diaries indicate this to be required. The intervention could then be developed as an application for mobile phones, reducing therapist time and costs. Making such an application available as support for clinical practices and in maintenance treatments would help treat people who do not have easy access to health care services. The use of new and innovative technology to make this kind of behavior change interventions more effective, while still taking care of the patients’ individual needs, suggests an exciting future area of research. The first step would be to develop and test the concept of automation in an RCT. In a further development of the intervention, it would be interesting to explore the effects of using more technologically advanced capabilities to gather rich and complex data. This could include sensors to measure activity levels and context-triggered diary questions [
Web-based interventions are becoming increasingly popular and appear to be an essential and cost-effective supplement to everyday health care in the near future. The evaluation of TF is essential to interpret the results of interventions and its underlying working mechanisms. A different methodology is used to deliver interventions via the internet. To achieve TF, the evaluation methods must be adjusted to each kind of online treatment. By doing this, the TF in Web-based interventions can be maintained and monitored reliably.
Framework and principles for development of feedback based on ACT.
Example of a sequence of feedback from the first week of the project
Interview guide for evaluation of the DMT2 pilot study.
Assessment of provider’s and participants’ evaluation of intervention.
acceptance
advise
Acceptance and Commitment Therapy
Audit of Diabetes Dependence Quality of Life
Behavior Change Consortium
behavioral support
committed action
cognitive defusion
Cognitive Behavior Therapy
chronic widespread pain
diabetes mellitus type 2
electronic diaries
education information
empathetic statements
general information
glycosylated hemoglobin
irritable bowel syndrome
National Institutes of Health
Problem Areas in Diabetes
contact with present moment
randomized controlled trial
self as context
short message service
stimulate participation
treatment fidelity
values
AAGN and HE were responsible for the development of DMT2 pilot study based on a concept invented by SvD. AAGN and EAB refined the coding scheme and codebook and performed inter-rate reliability test. AAGN coded all data material and drafted the manuscript. All authors contributed to the process of revision and approved the final manuscript. A warm thank to the participants helping in the development of this intervention. The authors were funded by their own institutions.
None declared.