This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
eHealth apps often fail to improve clinical outcomes due to poor integration with clinical workflow—the sequence and personnel needed to undertake a series of tasks for clinical care. Our central thesis is that eHealth interventions will be more effective if the clinical workflow is studied and taken into consideration for intervention implementation. This paper aims to provide an introductory tutorial on when and how to use a clinical workflow analysis to guide the implementation of eHealth interventions. The tutorial includes a step-by-step guide to conducting a clinical workflow analysis in planning for eHealth implementation. We began with a description of why a clinical workflow analysis is best completed before the implementation of eHealth interventions. Next, we described 4 steps needed to perform the clinical workflow analysis: the identification of discrete workflow components, workflow assessment, triangulation, and the stakeholder proposal of intervention implementation. Finally, we presented a case study of a clinical workflow analysis, which was conducted during patient visits of patients aged 11 or 12 years from 4 diverse pediatric or family medicine clinics to plan the implementation of a tablet-based app for adolescent vaccination. Investigators planning the implementation of new eHealth interventions in health care settings can use the presented steps to assess clinical workflow, thereby maximizing the match of their intervention with the clinical workflow. Conducting a prospective workflow study allows for evidence-based planning, identifying potential pitfalls, and increasing stakeholder buy-in and engagement. This tutorial should aid investigators in increasing the successful implementation of eHealth interventions.
Within health care settings, one of the most common reasons eHealth apps fail to effectively increase the health outcomes they are designed to aid and improve quality of care is incompatibility between the app and clinical workflow [
Several methods for assessing clinical workflow are available [
According to the Consolidated Framework for Implementation Research (CFIR), a well-known implementation research framework, interventions are not inherently compatible with specific settings, and adaptation is required to maximize success [
This paper offers a simple-to-follow methodology for studying clinical workflow and planning for implementation with clinic staff. It provides tools adapted from previously published strategies to aid researchers in identifying clinic-specific adaptations that will increase the compatibility of eHealth interventions with clinical workflow and thus improve outcomes [
Clinical workflow assessments for implementation planning can follow 4 steps (
Steps for conducting a workflow assessment.
Step | Purpose | Methods | Example tools |
Identify discrete workflow components | Define what is necessary to make the intervention work | Select locations, interactions, and tasks | Review direct observation checklist ( |
Workflow assessment | Create a model of the clinical workflow | Direct observation, interviews, sensor-based investigations, EHRa audit logs, and job task diaries | Review direct observation form ( |
Triangulation | Confirm rigor of clinical workflow model | Direct observation, interviews, sensor-based investigations, EHR audit logs, and job task diaries | Review semistructured template ( |
Stakeholder proposal | Plan intervention implementation based on stakeholder preferences | Interviews | Review semistructured template ( |
aEHR: electronic health record.
The first step in conducting a clinical workflow analysis for improving eHealth implementation planning is to identify the discrete components of the clinical workflow that need to be measured. Owing to the multitude of tasks occurring during clinical visits, it is imperative to clearly define what activities researchers should track to ensure that each is collected and documented consistently. Tracked activities may fall under 3 observable workflow categories: location, interactions, and tasks. Location refers to where and how long individuals are physically present in specific areas of a clinic. Interactions include face-to-face conversations and moments in which health records or patients transition from one provider to another. Tasks include a review of health records, measures collected (eg, weight, blood pressure), interventions administered (eg, giving prescription or vaccine), and administrative functions (eg, patient check-in or scheduling). For each task considered, the actors (patient, provider, or support staff) may vary.
To identify the discrete workflow components for the intervention, consider what is necessary to make the intervention work as intended in the setting. Imagine how the intervention will be integrated into the clinic and considered the following questions: At what time during a clinic visit is the most useful for the patient and provider to access the intervention? What specific tasks need to occur to have the intervention in the hands of those who need it at the right time during the clinical encounter? What types of clinical situations require alternative planning? When choosing the discrete workflow components of interest, it is important to consider the minimum necessary rule and evaluate tasks directly relevant to an intervention’s implementation that do not compromise patient privacy or impede care delivery.
A second step in conducting a workflow analysis to improve eHealth intervention implementation is to assess the clinical workflow. Several strategies can be used to assess clinical workflow, including direct observation, clinical staff reporting, interviews with staff or patients, sensor-based investigations, EHR audit logs, and job task diaries [
Workflow assessment via direct observation can occur in 3 steps: (1) identify who you wish to observe, (2) complete informed consent with potential participants, and (3) directly observe the workflow using standardized data collection tools to systematically record observations. Before direct observation, engage with clinic leadership to ensure that the observed activities will not affect medical staff employment. Before the observation days, explain the informed consent to the identified staff so that they can thoughtfully consider participation. Observations should be scheduled for clinic days in which relevant patients have scheduled visits. Once patients express interest to clinical staff, patient consent can be obtained. Care must be taken to avoid disrupting the standard clinical flow to maintain high-quality research and clinical care. Although the number of observed patients can be influenced by practical considerations (eg, travel expenses and clinical receptiveness), to achieve a valid sample, patients should be observed until additional observations no longer present new information (ie, saturation is achieved) [
The use of a standardized observation form increases the rigor and reproducibility of workflow observations and can be incorporated into any workflow assessment method chosen [
A third step in conducting a clinical workflow analysis for improving eHealth intervention implementation is method triangulation: the verification of findings from a different viewpoint by answering similar questions with a different method [
All the options mentioned for primary workflow assessment are possible methods for triangulation [
It is useful for stakeholders or actual clinical users to evaluate how the intervention will be compatible with their clinical workflow. Stakeholder opinions on implementation can be obtained during the triangulation phase through interviews or planning meetings. The planning process should follow the community participatory principles of engagement and participation so that the clinic staff feel that their insights and expertise are critical to the project [
We developed a tablet-based eHealth app for primary care clinics called
Our pilot trial demonstrated that the intervention successfully increased initiation of our main target, human papillomavirus (HPV) vaccine [
We identified 3 workflow-related conditions necessary for the success of the intervention. First, a parent needed 5-10 minutes to complete the application before the child saw the provider. Second, the provider needed to view the app results before (or during) the time spent with the patient. Third, the ordering and administration of vaccines needed to occur following parents’ and providers’ app use. As such, we needed to identify a time during a visit when the parent could complete the intervention, a method for getting the completed intervention to the provider for review, and ensure that both steps took place before the time when vaccines need to be ordered during a clinical encounter. Thus, we selected the following discrete workflow components of interest: (1) patient locations, interactions, and tasks before meeting with the provider; (2) provider interactions with the patient and patient records; (3) the timing of vaccine record review; and (4) the timing of vaccine administration. We restricted our workflow components to activities that occurred outside the clinic exam rooms in order to preserve patient privacy, encourage patient participation, and follow the minimum necessary rule for data collection.
To enhance the rigor of our direct observations, we created 2 standardized checklists by adapting the Arkansas Foundation for Medical Care Workflow Assessment Checklist found on the AHRQ website [
We created checklists in the MediDocs platform (EnMedical Systems), a software designed to capture medical information that includes a click to timestamp functionality. To ensure the usability and functionality of the Medidocs platform and enhance observer comfort with the procedures, observers created an example list of coffee shop procedures and then pilot tested the Medidocs platform at a local coffee shop by tracking employee and customer interactions. A coffee shop was selected because there were many individuals who followed the same procedures, and personal health information was not included.
We observed the clinical workflow at 4 primary care clinics that agreed to participate in the intervention implementation trial. The participating clinics were pediatric or family medicine specialties, served a range of patients with Medicaid insurance (14%-90% of patients), and were owned privately, by the university, or by health systems. Before the observation days, a lead clinical stakeholder at each clinic invited all medical staff who interacted with patients aged 11-12 years to attend an information session. Information sessions were held during each clinic’s preferred schedule (ie, during lunchtime or a staff meeting), food was provided, and research staff explained the study and informed consent forms. Clinic staff were given the opportunity to ask questions and return signed informed consent forms at any time before the observation days. The direct observation of clinical staff was restricted to those staff members who had returned signed consent forms.
To minimize clinic disruption and research staff travel expenses, we aimed to observe approximately 3 participants over 1-2 days at each clinic. Clinical staff selected days when at least one patient aged 11-12 years was scheduled. On each observation day, clinic staff invited all parents of patients aged 11-12 years to participate and referred interested parents to research staff who were present in the clinic. The research staff obtained consent through a Health Insurance Portability and Accountability Act waiver of documentation of consent and collected data without identifiers. Research staff conducting observations had a minimum of a bachelor’s degree in a scientific or social science field, had existing relationships with the clinic, and were trained in research and practice facilitation. We chose not to interview patients about the acceptability of completing the eHealth intervention during a clinic visit because parents were very receptive to completing the eHealth intervention during our pilot, and we enhanced parents’ acceptability of the app content with focus groups [
With 12 patients as a practical target, we observed 13 patient visits between August 2016 and March 2017. By the 13th visit, we were no longer observing new variations in the workflow components. Thus, saturation was reached, and further data collection was not needed [
For the triangulation of our direct observation, we interviewed clinical stakeholders (group interviews of a clinical support staff member and the lead physician at each clinic) because involving the clinic staff in the development process creates a sense of ownership and increases buy-in consistent with the CFIR [
After noting considerable similarities across clinics and visit types, we created a composite flowchart that represented the workflow at all 4 clinics (
While a patient completed check-in and triage, physicians attended to other patients. Once notified that a patient was ready and waiting in an exam room, physicians reviewed the patient’s record for a few minutes immediately before entering the exam room. Depending on the clinic, patient records were reviewed via paper files placed immediately outside an exam room or electronically. Following the patient-physician interaction, physicians at each clinic communicated verbally with a nurse or medical assistant about additional care needs (eg, vaccine administration). The clinical staff administered ordered services. Finally, the patient returned to the front desk and scheduled another appointment or received the required instructions.
Example clinical workflow diagram of patient visits (those aged 11-12 years) with the proposed intervention tablet placement. MA: medical assistant.
Clinicians suggested additional staff double-checking that the tablets were distributed. Participants mentioned that checking for tablet distribution during triage would be compatible with redundancy checks already taking place within a clinical workflow. “We always double-check ages and things anyway...so if I see that the patient is 11 or more, I can always grab [the tablet].” Another site noted that it would be helpful to use a team approach to enhance intervention dissemination. To accomplish this, front office staff could highlight the patients eligible for the intervention during their morning patient reviews. For example, one clinician stated:
They [front desk staff] identify all the 11- and 12-year-olds coming in for the following day, and they somehow highlight, red flag, that particular patient, and when we discuss patients coming in...it can just be mentioned. So, everyone, front desk staff, and clinicians, will know that a patient needs an iPad.
At all 3 of these clinics, physicians preferred to review the intervention results at the same time they reviewed the patient records. Clinicians suggested physically placing the tablet with the encounter forms and patient records or accessing the intervention results on their computer when reviewing patient records. As one physician explained, “Usually I look through the patient chart before I walk into the room, so I would just do that [review intervention results] right after.” In cases where patients were unable to complete the intervention in the waiting room, physicians felt that the parent could continue to complete the tablet while the physician interacted with the adolescent. In explaining this alternative, a physician noted:
...at age 11 or 12, I do more interaction with the child versus interaction with the parent. So, I can say, “Oh, by the way mom, can you go ahead and fill this out for me?”
One clinic that already used a tablet system suggested a slightly modified implementation. The staff of this clinic proposed that the new intervention be distributed in a similar fashion to their current tablet system (ie, given by nurse to parent in the exam room). If the parent finished before the nurse completed triage activities and left the exam room, then the nurse would collect the tablet from the parent and place it outside the exam room for the physician to review. If a parent was still completing the tablet intervention during the completion of triage activities, the nurse would alert the physician that the parent still had the tablet. In this case, physicians reviewed the tablet results upon entering the room alongside the patient. One physician explained:
I’d bring it up and say, “Let me see how you feel about the HPV vaccine,” and if they are ready to have it, “great,” and if not, I would discuss if they had a hesitation.
Clinicians identified 3 potential barriers to intervention implementation. First, clinicians reported that they do not typically review and administer vaccines during acute care visits. Although it was noted that parents would still be amenable to completing the intervention during most acute care visits, it was less clear how medical staff could alter their work processes to incorporate speaking about and, potentially, administering vaccines during acute visits. This issue was first mentioned in relation to the lack of an established process for checking vaccine records for patients presenting for an acute care visit, with staff reporting, “We don’t usually check ahead for the sick visit. Just for the wells.”
Second, physicians mentioned patient conditions that would preclude their administration of vaccines (eg, a high fever) or cause disruptions to how patients move through the clinic, thus interfering with when a tablet could be given to a patient to complete the intervention. One clinician explained that “...if we know that [a patient] is really, really contagious, we won’t even do two minutes [in the waiting room], as soon as they come in we put them in an empty room.” Clinicians were open to the idea of administering the intervention in these acute visit scenarios, for example, stating, “We could implement [the tablet] at some point, like once we’ve discussed the issue or whatever they’ve come in for,” but noting that more work would need to be done in order to optimize the process for acute visits.
The final barrier mentioned was the physician time constraints. One physician mentioned that, despite the simplicity of the tablet, competing demands may cause him to forgo reviewing intervention results stating
We present a general strategy and an example for conducting a workflow assessment to enhance the implementation of an eHealth intervention in primary care. With this strategy, we were able to identify specific timing, staffing, and management processes that may enhance intervention implementation. Potential barriers related to the implementation of the intervention were identified and addressed by stakeholders. Incorporating clinical stakeholders into the design of the implementation plan enhanced stakeholder buy-in, a critical component of successful implementation of eHealth interventions [
Conducting the workflow assessment before intervention implementation allows for adaptations consistent with the adaptable periphery of interventions in the CFIR. By using a direct observation approach to assess clinical workflow, we were able to identify how and when our eHealth intervention should be incorporated. Clinicians’ participation in verifying the workflow and proposing an implementation strategy allowed for additional options to be proposed before implementation. For example, clinicians added a double-check of the tablet distribution and provided contingency plans for when parents did not have enough time to finish the app before seeing the provider. Although the clinicians were unaware that our prior pilot had issues with front office staff forgetting to give the tablets to parents, incorporation of clinicians in the planning process pre-emptively addressed this shortcoming in our second implementation. Moreover, while we had anticipated a workflow tailored to each clinic based on prior research [
By engaging stakeholders with implementation planning, we were able to identify potential barriers, discourage clinic staff adaptation of the core intervention, and build clinician buy-in. Consistent with previous primary care workflow research [
There are 3 important limitations of the case study presented that could be improved in future workflow assessments. First, the workflow assessment focused on physicians and patients. Although physician and patient visits are the most frequent opportunity for administering adolescent vaccines, a workflow assessment that included nurses and other clinical staff could be useful if the intervention was implemented in nurse-only visits. Second, our triangulation and stakeholder engagement strategies focused on physicians and clinical support staff. In particular, as front office staff were identified as integral to the implementation plan, including front office staff in the workflow assessment and intervention implementation planning would likely improve planning and garner the needed support from these critical stakeholders. Third, our case study presented 4 clinics with 3 observations each and, in turn, the small number of observations increases the risk of undue influence of outliers if the visits are not representative of standard care practices. However, this size is reasonable for a small implementation study and is supported by thematic saturation [
Our proposed workflow assessment strategy and supporting case study have 3 strengths relative to the design and execution. First, the use of 2 strategies to assess workflow enhances the rigor and validity of the results. Second, the use of separate observers for the patient and the physician was relatively novel and ensured that overlapping activities were observed. This strategy of one observer to each actor can be expanded to any clinical setting. Third, the benefit of involving the clinic staff in implementation planning with regard to stakeholder ownership and engagement cannot be understated. Participating clinicians are interested in seeing the intervention succeed and are aware that some work-arounds may alter the core of the intervention.
Conducting a prospective workflow assessment can enhance the acceptability and performance of eHealth interventions in clinical practice. We presented a 4-step strategy to increase the compatibility between eHealth interventions and clinical workflow, as compatibility is one of the main predictors of clinician use [
Direct observation checklist for patients and clinic staff. Boxes above and below the tables create timestamps upon clicking.
Semistructured interview questions and subquestions were used to evaluate workflow observations.
Agency for Healthcare Research and Quality
Consolidated Framework for Implementation Research
electronic health record
human papillomavirus
The authors thank Jennifer Mauck (Master of Public Health) and Lori M Drum (Licensed Mental Health Counselor and Certified Clinical Research Coordinator) for their assistance with data collection. They also thank Shivani Desai (Master of Public Health) for manuscript formatting assistance. This research was funded by the National Institutes of Health, R21 CA202011 (principal investigator: SS).
SS led all aspects of the project. ES, MM, MV, LT, and SS recruited the clinics. SS, NR, ES, LT, MV, and MM planned the study. ES and MV performed direct observations and conducted semistructured interviews. SS and JT analyzed and interpreted the data and drafted the manuscript. JT, NR, ES, and SS created the tables and figures. All authors have reviewed and approved the final manuscript.
None declared.