Published on 27.02.15 in Vol 3, No 1 (2015): Jan-Mar
Qualitative Evaluation of a Text Messaging Intervention to Support Patients With Active Tuberculosis: Implementation Considerations
Background: Tuberculosis (TB) remains a major global public health problem and mobile health (mHealth) interventions have been identified as a modality to improve TB outcomes. TextTB, an interactive text-based intervention to promote adherence with TB medication, was pilot-tested in Argentina with results supporting the implementation of trials at a larger scale.
Objective: The objective of this research was to understand issues encountered during pilot-testing in order to inform future implementation in a larger-scale trial.
Methods: A descriptive, observational qualitative design guided by a sociotechnical framework was used. The setting was a clinic within a public pulmonary-specialized hospital in Argentina. Data were collected through workflow observation over 115 days, text messages (n=2286), review of the study log, and stakeholder input. Emerging issues were categorized as organizational, human, technical, or sociotechnical considerations.
Results: Issues related to the intervention included workflow issues (eg, human, training, security), technical challenges (eg, data errors, platform shortcomings), and message delivery issues (eg, unintentional sending of multiple messages, auto-confirmation problems). System/contextual issues included variable mobile network coverage, electrical and Internet outages, and medication shortages.
Conclusions: Intervention challenges were largely manageable during pilot-testing, but need to be addressed systematically before proceeding with a larger-scale trial. Potential solutions are outlined. Findings may help others considering implementing an mHealth intervention to anticipate and mitigate certain challenges. Although some of the issues may be context dependent, other issues such as electrical/Internet outages and limited resources are not unique issues to our setting. Release of new software versions did not result in solutions for certain issues, as specific features used were removed. Therefore, other software options will need to be considered before expanding into a larger-scale endeavor. Improved automation of some features will be necessary, however, a goal will be to retain the intervention capability to be interactive, user friendly, and patient focused. Continued collaboration with stakeholders will be required to conduct further research and to understand how such an mHealth intervention can be effectively integrated into larger health systems.
JMIR mHealth uHealth 2015;3(1):e21
Adherence to long-term medication therapy in the outpatient setting remains a global health challenge, particularly for tuberculosis (TB) treatment [, ]. The World Health Organization (WHO) and others have called for patient-centered approaches that tailor interventions to meet patient needs [ - ]. Regular supervision and support provides opportunities for education and problem monitoring, and promotes medication adherence [ ]. Directly observed therapy (DOT) has been the predominant TB medication management strategy since the 1950s [ , ]. However, DOT is challenging for patients and health care providers, due to limited resources, operation expenses, and daily travel burden [ - ].
Mobile health (mHealth) interventions utilize portable devices, such as mobile phones, to provide health services [, ]. Mobile phones are increasingly prevalent across the globe [ ] and there are a growing number of researchers assessing mHealth’s impact on health outcomes [ , ]. Promising evidence suggests that mHealth interventions can enhance health services in low- and middle-income countries [ - ]. However, uptake at a larger scale can be slow despite growing evidence of the potential benefits [ , ]. Previous studies of mHealth interventions in TB management have focused on DOT using the mobile phone’s video features [ - ], traditional phone calls to mobile phones to remind patients to take their medication [ ], or sending short message service (SMS) text messages asking participants to respond with the time they took their medication [ ]. The most common type of mHealth intervention reported in the literature is one-way SMS text messaging [ ].
Although mHealth technology offers promise to aid in management of chronic conditions, there remains insufficient evidence to inform larger-scale implementation . Leading experts recommend rigorous research of mHealth potential, as well as the implementation challenges, and warn against skipping outcome evaluations, which could threaten the understanding of the long-term value of mHealth [ ]. Small-scale and pilot implementations are needed to provide evidence of acceptability and feasibility and can suggest ways to improve mHealth interventions to avoid larger-scale implementation pitfalls [ , ].
The objective of this research was to understand implementation issues encountered during pilot-testing, and to identify system improvements that will inform future implementation in a larger-scale trial.
The research was based on descriptive observational qualitative design [, ] guided by an adapted sociotechnical framework (see below). This study was the second phase of an interventional study that explored feasibility, acceptability, and initial efficacy of the interactive TextTB intervention to support patients with active TB (described below) [ ]. Data were collected through workflow observations over 115 days and were tracked in a study log (eg, process, problems, barriers, and one-on-one and team meeting discussions). Text messages (n=2286) were reviewed for content related to technical challenges, and verbal and written stakeholder input—feedback to questions based on the sociotechnical framework—was collected. The setting was a clinic within a public pulmonary-specialized hospital in Buenos Aires, Argentina. The study was approved by the University of Utah Institutional Review Board (IRB) and an independent research ethics board of Hospital Italiano, in Buenos Aires, Argentina.
For this study, sociotechnical models from Cornford et al  and Barber et al [ ] were adapted. These models integrate classic Donabedian Structure-Process-Outcome quality improvement concepts with a sociotechnical perspective to understand health information technology implementation outcomes [ , ]. In addition, a Rapid Assessment Process (RAP) was used as a process guide [ ]. The RAP, adapted for informatics evaluation from ethnography and other qualitative methods, has been shown to be useful for explaining health technology implementation success or failure, and for providing feedback for system improvements [ ]. Informatics interventions, including mHealth interventions, often occur in naturalistic settings where certain variables are outside of the investigator's control. The basic definitions, matrix structure, and the framework from the models were adapted a priori to articulate variables relevant to this specific mHealth intervention (see ).
|Technical approach||Social approach|
|System function||Human perspective||Organizational context|
|Structure||Technical detail and content: computer-based intervention details||Adapting work conditions/requirements of intervention implementation||Requirements for sustainability: costs, management, and equipment needs|
|Process||Information processing||Participation of patient/health care team, social interaction||Altered practice and delivery of service|
|Outcome||Technical performance: efficiency and reliability||Quality of service and individual outcomes||Global effect: lessons learned, potential application to other settings|
aAdapted from Cornford et al  and Barber et al [ ].
The Intervention and Team
The TextTB intervention pilot study occurred from December 2011 through April 2013. Details of that study are reported elsewhere [, ]. In brief, participants were randomized to the intervention group (TextTB, n=18) or control group (paper documentation, n=19) for the first 2 months of active TB treatment. Participants in the TextTB group were asked to (1) send an initial SMS text message to confirm connection with the system at enrollment, (2) text daily to confirm they took their TB medication that day, and (3) text any questions or concerns. They received confirmation that their text messages were received, or they received query messages if they failed to send notifications. They also received twice weekly educational text messages that were based on the Information-Motivation-Behavior skills model [ - ]. The team members were the study principal investigator (PI), a regional TB director/pulmonologist, the lead regional TB social worker, a regional TB staff member, two TB-specialized clinic registered nurses (RNs), and the hospital TB program director/pulmonologist.
FrontlineSMS version 126.96.36.199 was the platform selected to send, receive, and manage text messages . FrontlineSMS is an open-source, free software program that is installed on a laptop and functions with a GSM modem and a local SIM card. The GSM modem fits into the laptop and holds the SIM card. The SIM card includes the imbedded chip required for cell phone transmission [ ]. The card contains identification numbers, controls which phone services the user can access, and can be moved between mobile devices. Together, the GSM modem and SIM card allow the computer to function like a mobile phone to send and receive text messages across a mobile phone network [ ].
Hands-on training and written directions in Spanish on how to operate the FrontlineSMS platform were provided by the PI to four team members. One team member primarily managed the daily patient interactions, while another did so on occasion or during vacation periods. Two pulmonologists were available for consultation for participant questions that were technical or needed expert advice (eg, recommendations for potential allergic reaction). Other nonparticipating hospital staff members were introduced to the intervention during a hospital-wide conference focusing on TB case presentations and current and future goals for TB management.
Data from the combined sources (eg, study log, text messages, and stakeholder feedback) were assessed for implementation issues and categorized based on the evaluation framework (see). An iterative, interpretive, and flexible process was applied as recommended for RAP methods [ ]. Analytic validity was strengthened with member checking through local stakeholder review, group consensus, and by drawing from multiple data sources. The PI conducted the initial categorization based on theoretical framework definitions. The other investigators reviewed the categorizations, which were iteratively refined until group consensus was achieved. An onsite champion (eg, local expert) who could serve as a liaison to clinical staff facilitated the evaluation process and provided member checking. Ash et al noted the importance of an onsite expert as well [ ]. Based on the specific issues identified in this study and the pragmatic experience of the investigators, potential modifications for a larger-scale trial implementation were outlined. Analysis was supported using ATLAS.ti version 6 (GmbH, Berlin, 2009).
Organizational considerations included issues that could impact sustainability, delivery of services, and potential application to other settings. Issues encountered that impacted the intervention included equipment security, intervention-related costs, and workflow changes.
The original plan was for the study laptop to be located at the nursing station, but it became apparent that the laptop might be unattended at times throughout the day, raising the risk for a security breach or outright theft. Therefore, the laptop was moved to the regional TB office located upstairs in the same building. Security concerns also required that the laptop be turned off and put away in a locked cabinet when study staff members were not present, which meant that the software was available only during clinic office hours, rather than around the clock. Participants were informed that responses would be provided only within clinic hours, Monday through Friday, and emergencies were to be directed through standard routes. However, powering down the program caused other problems—see technical and sociotechnical consideration sections.
The study had associated set-up costs. The FrontlineSMS software was open source and available free of charge . However, there were costs associated with the purchase of the laptop computer, GSM modem, and SIM card. The GSM modem was replaced due to technical issues, incurring additional cost.
The majority of ongoing costs were associated with text messaging over a mobile phone network (ie, SMS text messaging). A majority of participants reported having basic feature mobile phones (26/37, 70%) and pay-as-you-go mobile phone plans (22/37, 59%). In Argentina, text messages are free to receive and ranged from ARS (Argentinian peso) $0.60 to $1 (US $0.10-$0.23) to send for pay-as-you-go service plans, according to the participants and mobile phone service provider websites. However, these fees could vary substantially due to service provider promotions or by purchasing a package with unlimited text messaging for a given number of days. Based on cost range per text message, the intervention averaged US $12.80 to $29.44 per participant for a 2-month period for messages sent and received.
Although there are options for online mobile phone service providers to send text messages at a discounted rate, there were logistical problems with this service. Initially, credit was purchased as a bundle from one of the suggested online mobile phone credit providers (Clickatell). However, in order to use the credit in Argentina, the phone number had to remain a US or European number, which would have required the patients to send messages to an international number, greatly increasing the costs and potentially challenging the legitimacy of the intervention being conducted by local team members. Therefore, we chose to use a local phone number. Other cost-related considerations included two phones reported as lost or stolen, and credit used up sooner than anticipated in some instances, in which case the monthly credit was added to the patient account early.
Workflow Changes and Monitoring
Initially, the intervention was to be managed by the nurses. However, because the laptop had to be moved to a more secure area upstairs, the nurses were unable to consistently leave their area to run the intervention. The technician volunteered to be the primary operator of the messaging platform. The regional director managed the platform when others were on sick leave or on vacation. An Excel file was developed to help visually track participant stage of treatment and notifications, and to clearly identify those who did not send notifications. The technician estimated between 15 minutes to 1 hour per day was required to review and respond to the text messages. This time was broken up into two to three intervals between work responsibilities.
Another workflow impact resulted from tracking and managing mobile phone credit compensation. Participants were provided with mobile phone credit at study onset and at the beginning of the month to compensate for cost of texting daily and at the end of each month for study participation. Although adding credit was easy, it required leaving the hospital to go to a kiosk nearby where credit could be added with the phone number and name of the mobile provider. This task was additionally complicated because participants were enrolled in the study at various times and cash was required for the transaction.
Although not directly related to the mHealth intervention, the paper-based medical charting system in place was a challenge for collecting final treatment outcomes. Manual chart review, to identify and collect treatment outcomes, was time consuming and accompanied by occasional missing or incomplete data. When considering future trials at a larger scale, the integration of the mHealth data into the paper-based medical records will be challenging or impossible.
There was an unexpected TB medication shortage at the regional and national TB program levels during the study. The shortage and outage of some medication was documented in the study log from December 1, 2012 to the last text message regarding the medication shortage sent on March 20, 2013. However, availability or amount of stocked medication may have been an issue for a longer period. Because of the shortage, medication was distributed to participants for shorter time periods, for example, 5 days or 2 weeks rather than the standard 1- to 2-month supply. In some instances, only one of the medications was allocated and patients were told to keep checking back for the missing medication. Although not the focus of our study, the medication shortage may have impacted treatment outcomes, and text messages were added to inform participants when medication became available.
Human Perspective Considerations
Human/social considerations focused on requirements for implementation (eg, training), team participation, change in attitudes or beliefs, and perception of the quality of the intervention.
It was necessary to train study personnel to use the messaging platform, and an extended period of technical support was required. Training was provided by the PI to the regional TB director/physician, a social worker, two nurses, and a technician. One nurse and the technician were computer novices and required instruction on basic computer functions (eg, how to save a file, how to copy and paste text). All trained team members were able to review and send text messages using the platform. All intervention participants knew how to send text messages. Participants were given verbal and written instructions on the desired format for notification messages. However, there were still problems with inconsistent formatting of notification messages (discussed below in the auto-confirmation section).
Team Participation and Process
Responding to patients’ questions was often a collaborative process. The TB technician solicited assistance from team members when necessary. The technician had worked in the hospital for nearly 20 years, had conducted TB testing, had basic TB knowledge, and was familiar with the hospital/clinic setting. The office was without walls or dividers and the regional TB director (pulmonologist/TB specialist) and TB social worker were seated nearby and could provide rapid assistance.
Attitudes and Beliefs
The team indicated that the ideal would be for patients to be referred to local centers for close monitoring during their TB treatment course. However, they all agreed that the intervention appeared to be beneficial for those receiving treatment by self-administration and especially for those who lived in rural or semirural settings where access to health care was challenging. The technician, who primarily managed the intervention, indicated that he felt he was truly able to help participants through this mode of delivery. He noted that the intervention seemed to be very useful to participants, especially for those who had many questions, concerns, or needed advice.
Some of the staff, however, believed that self-administration functioned well despite a documented high rate of treatment abandonment. As a result, there were some staff who hesitated to act with respect to potential abandonment before a full month had passed. In one case, the patient contact information was not entered into the medical record and, therefore, follow-up could not be initiated. The intervention did serve to identify a patient who had been documented in the notification records as abandoning treatment, but who had actually transferred to a different health care facility and had noted doing so in one of her text messages.
Technical System Function (Platform) Considerations
A number of the technical issues encountered during the pilot test were able to be remedied, while others will need further consideration for a larger-scale trial. The technical issues, including software quirks, errors in data exporting, and software inefficiencies affected the intervention flow and analysis of results, as well as the reliability of the program.
The platform software allowed messages to be organized in multiple ways (eg, by contact, sent or received, or all). However, it was identified that a “0” needed to be added to the beginning of each mobile phone number in order to match the messages to an individual. In addition, although two local area codes were used interchangeably for calls within the province, the software recognized only one (eg, 11, and not 15).
Scheduling of Reminders
The reminders software feature was used to set up the package of educational messages for automatic delivery at predetermined times, twice weekly.illustrates the steps necessary to create one message (eg, select date, frequency, add content). Patients entered the study on a rolling basis, therefore, the package of messages had to be created for each participant. It took 10 to 15 minutes to create each package. By trial and error it was identified that the software had a maximum capacity of 100 reminders. The system did not provide a notification or alert to indicate why new messages could not be created or that maximum capacity was near or reached. Once this issue was identified, sent messages had to be deleted at regular intervals for new ones to be added.
Impacting Cost and Data Analysis
Accents marks, commonly used in the Spanish language, were recognized as multiple characters. Despite not exceeding the 160-character limit, this miscount cued the platform to send multiple messages (see). Once this problem was identified, accent marks were removed from the educational messages.
In addition, problems were encountered when exporting messages with accent marks for analysis. The software has a function to export data into an Excel file. However, letters with associated accent marks were changed to unrecognizable characters during this process. One example is the Spanish word for “I took”, which is “Tomé” that was changed to “TomÃ©.” The messages had to be reviewed individually and corrected in the exported data file, resulting in increased time for data analysis.
In addition, messages were time stamped 2 hours earlier than when they were sent or received. Consultation with the program staff suggested that the cause of the time discrepancy might have been because the study computer was set up in a different time zone. Changing the time zone on the computer did not correct the problem. This issue warrants further investigation to assure precision time stamping in the future.
Reliability of Software
Inconsistencies were identified between the confirmation of messages sent within the platform and messages delivered after messages were not received on the study mobile phone. It was determined that when the mobile phone credit was depleted, messages were still shown as sent within the platform even though they were not sent by the mobile phone carrier. There was no link between the mobile phone carrier or credit amount remaining on the SIM card and the FrontlineSMS software. To address this problem, the mobile credit balance had to be regularly checked on the service provider website. This was another task to remember to do on a regular basis to avoid episodes where messages were believed to be delivered, but were not.
Larger System Issues Impacting Technical System Function
Network coverage variability and Internet and electricity outages were outside of the investigators’ control, but impacted intervention implementation. The impact of data loss due to poor or no network coverage was minimal for the pilot study, but could be significant for a larger-scale implementation. There were participants who reported loss of network coverage when traveling as the cause for missing days’ worth of notifications. In addition, one of the main mobile phone service providers had a 3-day outage and no messages could be sent or received, which likely resulted in loss of data. The company provided a compensation of ARS $10 (about US $3.5) for the loss in service.
In addition, there were 10 days logged in the field notes during which there was no Internet access and days without electricity. Access to the Internet is required to download the FrontlineSMS program, but after installment, Internet access was no longer required because the software sends messages over a mobile phone network instead of over the Internet. Disruption in Internet access could be a concern for cloud-based programs. For several days, electricity was out for all, or part, of the work day. The loss of electricity did not cause a delay in messages being received or sent because the laptop had a long battery life. However, extended periods without electricity could be a significant problem for larger-scale implementations. In addition, when the radio was on in the office, a notable static disturbance in the music was heard when texts were received.
Sociotechnical considerations reflected the interactions between people and technology and focused primarily on issues that affected study participants. Issues included inconsistent message delivery, deletion of reminder settings, challenges to using keywords for auto-confirmation messages, and messages delivered out of sync.
Inconsistent Message Delivery
For the first 2 months, reminder messages were sent to participants with some replying that they had sent their notification earlier. It was eventually identified that depending on the order of opening the software (eg, modem first and then FrontlineSMS, or vice versa) messages would go to the modem and not be delivered to the messaging platform. It was later recognized that the modem had a storage maximum at which time messages needed to be deleted in order to receive new ones. Initially, the modem inbox was reviewed daily and messages (n=55) were manually transferred into the database. Then a new modem (different model) was incorporated into the trial and there was fewer instances of data not being received. Remaining issues appeared to be caused by problems with service coverage, rather than the modem.
Deletion of Reminder Settings
The educational messages set up using the reminder feature were also not consistently delivered at the prespecified time and date. When reminders were sent, a green check mark was displayed to the left of the message row in the software window. The reminders that were not checked had to be reviewed and resent. It was later discovered that powering down the program caused the automatic reminder settings to be erased. As previously mentioned, the study computer had to be powered off and stored because of security issues.
Challenges to Using Keywords to Send Auto-Confirmation Messages
The keyword feature was used to set up auto-reply messages to daily notifications of self-administration of medication. In order for the software to recognize a keyword, it must be the first word of the message. It was identified that the software was also sensitive to case, accent marks, and punctuation. As a result, multiple similar keywords were added (eg, upper-/lowercase, with/without accent marks) in response to the different ways participants formatted messages. Although participants received both verbal and written instructions for the formatting of notification messages (eg, “Tome 3R 4B” for “I took 3 red and 4 white pills”), message format often varied (eg, “hello”, “good day”, or other first words). This led to failure of the system to auto-confirm receipt of notification, requiring the technician to manually review messages and respond. Some keywords were added in attempt to accommodate communication patterns (eg, “hola!!!”). However, the number and variety of keywords also led to auto-confirmation messages being sent in error (n=17) when, for example, a question rather than a notification was received.
Messages Delivered Out of Sync
Another outcome to shutting down the study computer nightly and over the weekend due to security reasons was that some messages were delivered out of sync. That is, if a message was sent outside of the office hours, the auto-response was delayed until the software was powered on. For example, if a participant sent notifications over the weekend, multiple acknowledgment messages were sent first thing Monday morning. These messages that were delivered out of sync caused confusion for some participants. Some participants texted back that they received a confirmation message, but had not yet taken their medications that day.
The goal of this research was to identify and describe practical issues that need to be considered before widely disseminating the TextTB intervention. This research uncovered generalizable issues that others may want to consider prior to developing and implementing an interactive mHealth intervention. Understanding what worked and what likely would not work at a larger scale is important for planning. Potential solutions were identified for some of the problems (see). The potential solutions need to be evaluated for effectiveness in future research.
|Adding phone credit was time consuming and credit was depleted early for some participants||Free texting|
|Contract with local mobile phone carriers|
Trial mobile messaging app that allows exchange of messages without SMS charge
|Messaging platform shortcomings||Modify open-source software or consider other available platforms|
|Message package upload and tailored delivery|
Alerts to health care team
Visualization of treatment course
|Retain interactive feature to promote patient/health care personnel relationship|
|Identify or develop software that provides message selection options (eg, an app)|
|Structured messages (improve ease of patient reporting, maximize automaticity of program features, and reduce manual intervention)|
Free text option for questions and responses
|Patient tracking||Further partner with stakeholders (eg, National TB Program director, local health care centers)|
|Have dedicated intervention management staff|
|Confirm transfers to another health care facility|
|Case contact tracing|
|Involve health care provider in care management|
|Implement intervention for full treatment course|
The texting platform software allowed for a large number of contacts and enabled multiple view options. The software allowed interactivity, personalization, and messaging functionality without requiring access to the Internet. Although we attempted to make the intervention as automated as possible, the team also wanted to assure that it could allow personalized communication with the patients, and not be viewed as too automated.
Intervention Areas to Strengthen
We experienced issues of delayed response and multiple acknowledgment messages. After this study concluded, we learned from other researchers (personal communication) that powering off the software can cause the reminder feature to be inactivated, which would explain the inconsistencies we experienced. Our solution was a manual review of sent messages and resending unsent messages. The solution for the other researchers was to leave the computers running continuously, although they have experienced issues caused by software updates or the computer accidentally being unplugged. However, in our situation the computer needed to be powered off and stored daily because of security issues.
Patients sent notifications that they took their medication in a number of ways, although verbal and written directions were provided, which caused inefficiencies and messages sent in error. For widespread dissemination, software that provides message selection options (ie, an app instead of basic messaging) might be useful. Selecting from structured messages, with an option to include free text questions, could improve patient reporting, maximize automaticity of program features, and reduce the need for manual responses. However, the use of structured messages may also make the intervention less personal.
The software was not built specifically for the application for which it was used in this study and, therefore, we expected to find some misalignment of features. A new Web-based version of FrontlineSMS, version 2, is now available. It is advertised as more intuitive, easier to use for creating and managing messages, and capable of managing larger volumes of messages . However, in the new version, a number of key features we used, such as the translation manager and reminders, are currently not available. This exemplifies a challenge to implementing open-source software. It is difficult to purposefully use a previous version of software once the new one is released, and the new version may be quite different.
Even though sufficient credit for the intervention was added to patients’ mobile phones at the beginning of each month, running out of credit early was a problem for some participants. Texting and mobile phone use habits are unique to each individual. For example, one participant indicated that when she had available funds, her usual mobile phone habit was to purchase a package that allowed unlimited calls and texting for 5 days. She would use her phone as much as possible during these periods. She reported having to use her mother’s phone during the study because she had used all her credit within a short time. In addition, the workflow associated with adding phone credit for each patient at local kiosks, although feasible for a pilot study, would likely be prohibitive on a larger scale. An ideal solution for this issue would be to establish a free to-text-in number.
Easy access to the computer is an essential feature for workflow integration. The original plan was for the intervention to be managed by the nurses. When equipment security concerns required the laptop to be moved upstairs, the nurses were unable to consistently leave their area to run the program. Nurses are often overburdened with multiple tasks and patient interruptions. Leaving the nursing area daily to monitor the messages was not feasible. With large numbers of patients, the ideal solution would be to hire one individual with TB specialty knowledge to manage the messaging.
Factors Unrelated to the Intervention
Potentially, extended periods of Internet and electrical outages could have caused major problems with the research. Internet and electrical outages can occur more often in low- and middle-income countries. Hoffman et al  reported challenges of Internet access in Kenya with downtime due to frayed network cables and slow system access. Because the software we used did not require the Internet to operate, outages experienced during the study period were not a limiting factor to the continuity of intervention delivery. We used a computer that had a long battery life. There were a few participants, however, that reported depleted mobile phone batteries after long periods of power outages. Extended electrical outages could certainly be anticipated to cause problems for any mHealth intervention.
Loss of data was a major frustration. Data loss is not unique to the technology used in this study. Hoffman et al  reported technical and transmission problems resulting in an estimated 25% loss of data, and Wade et al [ ] reported technical problems with variable phone signal strength.
Use of the Sociotechnical Framework
For this study, we adapted a framework [, ] in which sociotechnical concepts (ie, technology, person, organization) were matched to classic structure-process-outcome concepts to create a matrix. There were limitations to our approach. The matrix was simple and explicit, but the interaction between components, which is a key part of sociotechnical systems theory, remained implicit and could have been overlooked. The matrix also did not account for broader contextual issues like electrical outages or a national medication shortage, and the approach does not explicate interpersonal interactions. Sociotechnical approaches have provided a powerful framework within which to analyze reasons for uptake and performance with many types of information and communications technology, and can contribute to the design of those systems [ ]. Much of the richness and nuance of the sociotechnical approach was lost with our approach. However, it was pragmatically useful for the purposes of analyzing and categorizing implementation issues.
The field of mHealth is rapidly evolving, and frameworks designed specifically to evaluate mHealth interventions are beginning to emerge. For example, Leon et al  applied a health system framework to assess community-based health services system challenges in South Africa. Their framework included larger context and sustainability issues and identified four key dimensions: government stewardship, organizational systems, technical systems, and financial systems. Hirschhorn et al [ ] and Aranda-Jan et al [ ] conducted systematic reviews to identify potentially important domains for scaling up and evidence of which mHealth components worked and did not work in Africa. Mohr et al [ ] focused on principles from behavior change theories in the context of mHealth.
The purpose of this research was to identify key areas to improve for further testing of the intervention in a larger sample. We succeeded in identifying and developing solutions for most of the problems we encountered, although a few issues remained unresolved. Issue tracking, determining the cause of problems, and developing solutions occurred on an ad hoc basis. Although informatics experts were a part of the PI dissertation committee, on-the-ground IT support was limited. As such, some of the issues encountered may have been managed proactively with a local IT expert. In addition, the cost estimate was for the texting component only, and did not factor in staff time—such estimates are recommended for the next research phase.
Intervention challenges were largely manageable during the pilot study, but when evaluating these challenges for a larger-scale trial, issues to be addressed were identified with potential solutions outlined here. These findings may help others considering implementing an mHealth intervention to mitigate these challenges. Although some of the issues may be context dependent, other issues such as electricity and Internet outages, limited supplies, and human resources are not unique issues to our setting. Release of new software versions does not necessarily result in solutions for certain issues, particularly when specific features are removed. As such, other software options will be considered prior to moving to a larger-scale study. Improved automation of some features was recognized as necessary for use with a larger sample, however, a goal will be to retain the intervention capability to be interactive, user friendly, and meet the needs of the patients and the health care team. Continued collaboration with stakeholders will be required to conduct further research and to understand how such an mHealth intervention can be effectively integrated into larger health systems.
The authors would like to thank the study participants, Mirta Etchevarria and Daniel Cardinale for participating in data collection, and other hospital staff not directly involved in this study. We also want to thank Christine Pickett for manuscript editing. We would also like to thank the following agencies for funding this study: the National Institute of Nursing Research (NINR) of the National Institutes of Health (NIH) for the Ruth L. Kirschstein National Research Service Award (NRSA) (grant no. F31NR012614), and Sigma Theta Tau Gamma Rho Chapter. SI receives training funding through the NINR of the NIH, Comparative and Cost-Effectiveness Research Training for Nurse Scientists Award (T32NR014205). The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
SI contributed to the research development, implementation, analysis, and manuscript drafting. KS contributed to the adaptation of the theoretical framework, analysis, and manuscript preparation. SB, PP, and DT contributed to protocol development, analysis, and manuscript preparation. CC contributed to data collection, manuscript preparation, and verification of findings.
Conflicts of Interest
- Bayer R, Wilkinson D. Directly observed therapy for tuberculosis: history of an idea. Lancet 1995 Jun 17;345(8964):1545-1548. [Medline]
- Morris K. WHO sees DOTS. The Lancet 1997 Mar;349(9055):857. [CrossRef]
- Barclay E. Text messages could hasten tuberculosis drug compliance. Lancet 2009 Jan 3;373(9657):15-16. [Medline]
- Kaplan WA. Can the ubiquitous power of mobile phones be used to improve health outcomes in developing countries? Global Health 2006;2:9 [FREE Full text] [CrossRef] [Medline]
- Tuberculosis Coalition for Technical Assistance. International Standards for Tuberculosis Care (ISTC).: The Hague: Tuberculosis Coalition for Technical Assistance; 2006. URL: http://www.who.int/tb/publications/2006/istc_report.pdf [accessed 2015-02-12] [WebCite Cache]
- World Health Organization. Treatment of Tuberculosis: Guidelines. 4th edition. Geneva, Switzerland: Stop TB Department, World Health Organization; 2010. URL: http://apps.who.int/iris/bitstream/10665/44165/1/9789241547833_eng.pdf?ua=1&ua=1 [WebCite Cache]
- Hill PC, Stevens W, Hill S, Bah J, Donkor SA, Jallow A, et al. Risk factors for defaulting from tuberculosis treatment: a prospective cohort study of 301 cases in the Gambia. Int J Tuberc Lung Dis 2005 Dec;9(12):1349-1354. [Medline]
- Khan MA, Walley JD, Witter SN, Imran A, Safdar N. Costs and cost-effectiveness of different DOT strategies for the treatment of tuberculosis in Pakistan. Directly Observed Treatment. Health Policy Plan 2002 Jun;17(2):178-186 [FREE Full text] [Medline]
- Sanchez AI, Bertolozzi MR. Beyond DOTS (Directly Observed Treatment Short-Course) in tuberculosis' control: interfacing and sharing needs. Rev Lat Am Enfermagem 2009;17(5):689-694 [FREE Full text] [Medline]
- Källander K, Tibenderana J, Akpogheneta OJ, Strachan DL, Hill Z, ten Asbroek AH, et al. Mobile health (mHealth) approaches and lessons for increased performance and retention of community health workers in low- and middle-income countries: a review. J Med Internet Res 2013;15(1):e17 [FREE Full text] [CrossRef] [Medline]
- World Health Organization. mHealth: New Horizons for Health Through Mobile Technologies. Geneva, Switzerland: WHO Press, World Health Organization; 2011. URL: http://www.who.int/goe/publications/goe_mhealth_web.pdf [accessed 2015-02-12] [WebCite Cache]
- The World in 2014: ICT Facts and Figures. Geneva, Switzerland: International Telecommunication Union; 2014 Apr. URL: http://www.itu.int/en/ITU-D/Statistics/Documents/facts/ICTFactsFigures2014-e.pdf [accessed 2015-02-12] [WebCite Cache]
- de Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone messaging for facilitating self-management of long-term illnesses. Cochrane Database Syst Rev 2012;12:CD007459. [CrossRef] [Medline]
- Fiordelli M, Diviani N, Schulz PJ. Mapping mHealth research: a decade of evolution. J Med Internet Res 2013;15(5):e95 [FREE Full text] [CrossRef] [Medline]
- Lester R, Karanja S. Mobile phones: exceptional tools for HIV/AIDS, health, and crisis management. Lancet Infect Dis 2008 Dec;8(12):738-739. [CrossRef] [Medline]
- Wei J, Hollin I, Kachnowski S. A review of the use of mobile phone text messaging in clinical and healthy behaviour interventions. J Telemed Telecare 2011;17(1):41-48. [CrossRef] [Medline]
- Kumar S, Nilsen W, Pavel M, Srivastava M. Mobile health: revolutionizing healthcare through transdisciplinary research. Computer 2013 Jan;46(1):28-35 [FREE Full text]
- Chaiyachati KH, Loveday M, Lorenz S, Lesh N, Larkan LM, Cinti S, et al. A pilot study of an mHealth application for healthcare workers: poor uptake despite high reported acceptability at a rural South African community-based MDR-TB treatment program. PLoS One 2013;8(5):e64662 [FREE Full text] [CrossRef] [Medline]
- Chib A, van Velthoven MH, Car J. mHealth adoption in low-resource environments: A review of the use of mobile healthcare in developing countries. J Health Commun 2015 Jan;20(1):4-34. [CrossRef] [Medline]
- DeMaio J, Schwartz L, Cooley P, Tice A. The application of telemedicine technology to a directly observed therapy program for tuberculosis: a pilot project. Clin Infect Dis 2001 Dec 15;33(12):2082-2084 [FREE Full text] [CrossRef] [Medline]
- Hoffman JA, Cunningham JR, Suleh AJ, Sundsmo A, Dekker D, Vago F, et al. Mobile direct observation treatment for tuberculosis patients: a technical feasibility pilot using mobile phones in Nairobi, Kenya. Am J Prev Med 2010 Jul;39(1):78-80. [CrossRef] [Medline]
- Wade VA, Karnon J, Eliott JA, Hiller JE. Home videophones improve direct observation in tuberculosis treatment: a mixed methods evaluation. PLoS One 2012;7(11):e50155 [FREE Full text] [CrossRef] [Medline]
- Kunawararak P, Pongpanich S, Chantawong S, Pokaew P, Traisathit P, Srithanaviboonchai K, et al. Tuberculosis treatment with mobile-phone medication reminders in northern Thailand. Southeast Asian J Trop Med Public Health 2011 Nov;42(6):1444-1451. [Medline]
- Mohammed S, Siddiqi O, Ali O, Habib A, Haqqi F, Kausar M, et al. User engagement with and attitudes towards an interactive SMS reminder system for patients with tuberculosis. J Telemed Telecare 2012 Oct;18(7):404-408. [CrossRef] [Medline]
- Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling up mHealth: where is the evidence? PLoS Med 2013;10(2):e1001382 [FREE Full text] [CrossRef] [Medline]
- Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [CrossRef] [Medline]
- Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI -statement on reporting of evaluation studies in health informatics. Yearb Med Inform 2009:23-31. [Medline]
- Sandelowski M. Whatever happened to qualitative description? Res Nurs Health 2000 Aug;23(4):334-340. [Medline]
- Sandelowski M. What's in a name? Qualitative description revisited. Res Nurs Health 2010 Feb;33(1):77-84. [CrossRef] [Medline]
- Iribarren SJ, Beck S, Pearce PF, Chirico C, Etchevarria M, Cardinale D, et al. TextTB: A mixed method pilot study evaluating acceptance, feasibility, and exploring initial efficacy of a text messaging intervention to support TB treatment adherence. Tuberc Res Treat 2013;2013:349394 [FREE Full text] [CrossRef] [Medline]
- Cornford T, Doukidis G, Forster D. Experience with a structure, process and outcome framework for evaluating an information system. Omega 1994 Sep;22(5):491-504. [CrossRef]
- Barber N, Cornford T, Klecun E. Qualitative evaluation of an electronic prescribing and administration system. Qual Saf Health Care 2007 Aug;16(4):271-278 [FREE Full text] [CrossRef] [Medline]
- Ash J, Sittig DF, McMullen CK, Guappone K, Dykstra R, Carpenter J. A rapid assessment process for clinical informatics interventions. AMIA Annu Symp Proc 2008:26-30 [FREE Full text] [Medline]
- Iribarren SJ, Beck SL, Pearce PF, Chirico C, Etchevarria M, Rubinstein F. mHealth intervention development to support patients with active tuberculosis. JournalMTM 2014 Jul 28;3(2):16-27. [CrossRef]
- Fisher JD, Amico KR, Fisher WA, Harman JJ. The information-motivation-behavioral skills model of antiretroviral adherence and its applications. Curr HIV/AIDS Rep 2008 Nov;5(4):193-203. [Medline]
- Fisher JD, Fisher WA, Bryan AD, Misovich SJ. Information-motivation-behavioral skills model-based HIV risk behavior change intervention for inner-city high school youth. Health Psychol 2002 Mar;21(2):177-186. [Medline]
- FrontlineSMS FrontlineCloud. Washington, DC: FrontlineSMS URL: http://www.frontlinesms.com/ [accessed 2015-02-12] [WebCite Cache]
- NowSMS. WhyteLeafe, UK: Now Wireless Limited What is a GSM modem? (or GPRS modem? or 3G modem?) URL: http://www.nowsms.com/faq/what-is-a-gsm-modem [accessed 2015-02-12] [WebCite Cache]
- Coiera E. Putting the technical back into socio-technical systems research. Int J Med Inform 2007 Jun;76 Suppl 1:S98-S103. [CrossRef] [Medline]
- Leon N, Schneider H, Daviaud E. Applying a framework for assessing the health system challenges to scaling up mHealth in South Africa. BMC Med Inform Decis Mak 2012;12:123 [FREE Full text] [CrossRef] [Medline]
- Hirschhorn LR, Talbot JR, Irwin AC, May MA, Dhavan N, Shady R, et al. From scaling up to sustainability in HIV: potential lessons for moving forward. Global Health 2013;9:57 [FREE Full text] [CrossRef] [Medline]
- Aranda-Jan CB, Mohutsiwa-Dibe N, Loukanova S. Systematic review on what works, what does not work and why of implementation of mobile health (mHealth) projects in Africa. BMC Public Health 2014;14:188 [FREE Full text] [CrossRef] [Medline]
- Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions. J Med Internet Res 2014;16(6):e146 [FREE Full text] [CrossRef] [Medline]
|DOT: directly observed therapy|
|IRB: Institutional Review Board|
|mHealth: mobile health|
|NIH: National Institutes of Health|
|NINR: National Institute of Nursing Research|
|NRSA: National Research Service Award|
|PI: principal investigator|
|RAP: Rapid Assessment Process|
|RN: registered nurse|
|SMS: short message service|
|WHO: World Health Organization|
Edited by G Eysenbach; submitted 25.10.14; peer-reviewed by C Aranda-Jan, G Tsafnat; comments to author 19.11.14; revised version received 19.12.14; accepted 19.12.14; published 27.02.15
©Sarah J Iribarren, Katherine A Sward, Susan L Beck, Patricia F Pearce, Diana Thurston, Cristina Chirico. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 27.02.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.