eHealth apps often fail to improve clinical outcomes due to poor integration with clinical workflow—the sequence and personnel needed to undertake a series of tasks for clinical care. Our central thesis is that eHealth interventions will be more effective if the clinical workflow is studied and taken into consideration for intervention implementation. This paper aims to provide an introductory tutorial on when and how to use a clinical workflow analysis to guide the implementation of eHealth interventions. The tutorial includes a step-by-step guide to conducting a clinical workflow analysis in planning for eHealth implementation. We began with a description of why a clinical workflow analysis is best completed before the implementation of eHealth interventions. Next, we described 4 steps needed to perform the clinical workflow analysis: the identification of discrete workflow components, workflow assessment, triangulation, and the stakeholder proposal of intervention implementation. Finally, we presented a case study of a clinical workflow analysis, which was conducted during patient visits of patients aged 11 or 12 years from 4 diverse pediatric or family medicine clinics to plan the implementation of a tablet-based app for adolescent vaccination. Investigators planning the implementation of new eHealth interventions in health care settings can use the presented steps to assess clinical workflow, thereby maximizing the match of their intervention with the clinical workflow. Conducting a prospective workflow study allows for evidence-based planning, identifying potential pitfalls, and increasing stakeholder buy-in and engagement. This tutorial should aid investigators in increasing the successful implementation of eHealth interventions.JMIR Mhealth Uhealth 2021;9(3):e18534
Within health care settings, one of the most common reasons eHealth apps fail to effectively increase the health outcomes they are designed to aid and improve quality of care is incompatibility between the app and clinical workflow [- ]—the series of tasks conducted to complete clinical care in what order and by whom. Incompatibilities often stem from a wait-and-see method of implementation common to many eHealth interventions whereby an intervention is introduced into a clinical setting and the clinical workflow either adjusts to accommodate or does not [ , ]. Frequently, a lack of integration with clinical workflow results in clinical staff creating work-arounds or adaptations that interfere with the core components of the intervention [ - , ]. For example, nurses reported checking boxes on forms that were not applicable to move the screens forward and others reported supplementing the electronic records with handwritten notes on paper [ ]. Many eHealth apps would be more effective if a tailored implementation plan was created based on a systematic clinical workflow analysis performed before implementation [ ].
Several methods for assessing clinical workflow are available [- ]. Most notably, the Agency for Healthcare Research and Quality (AHRQ) has a digital toolkit of resources to guide investigators in studying clinical workflow [ ]. In addition, Ozkaynak et al [ ] have written an overview chapter of clinical workflow, including definitions of workflow, several workflow evaluation approaches, and examples of visualizing workflow [ ]. Other studies describe select methods of studying clinical workflow, including electronic health record (EHR) audit logs, direct observation by trained external staff, clinic staff reporting, and sensor-based recordings [ - ]. Despite evidence that an intervention’s compatibility with the local setting is pertinent to success [ ], there is limited information clearly describing how to increase compatibility with clinical workflow before implementation within the eHealth literature.
According to the Consolidated Framework for Implementation Research (CFIR), a well-known implementation research framework, interventions are not inherently compatible with specific settings, and adaptation is required to maximize success [, ]. Following this meta-theory, interventions can be adapted as long as the changes occur within the adaptable periphery—components that do not compromise the intervention’s core components essential for the affect. One type of adaptation to improve user compatibility, a key construct from the diffusion of innovations theory incorporated within the CFIR inner setting [ , , ] that can frequently be made without affecting the intervention’s core components, is integration with the clinic’s workflow. To ensure that adaptations enhance compatibility and remain within the adaptable periphery, it is suggested that the investigators and clinic staff collaborate to develop adaptations. In addition to engaging the clinical staff stakeholders, a key construct of CFIR and a critical component of integrating eHealth into primary care, this collaboration increases the likelihood of meeting two additional constructs within the CFIR process domain: identifying clinical champions and garnering the support of opinion leaders [ , , ].
This paper offers a simple-to-follow methodology for studying clinical workflow and planning for implementation with clinic staff. It provides tools adapted from previously published strategies to aid researchers in identifying clinic-specific adaptations that will increase the compatibility of eHealth interventions with clinical workflow and thus improve outcomes [, ]. We present 4 steps for assessing clinical workflow and improving intervention compatibility with clinical settings, and a case study illustrating these steps. First, identify which components of the intervention are critical and when these components need to occur during a clinic visit. Second, choose from a variety of described methods to observe the existing clinical workflow relevant to the intervention. Third, confirm their findings using a second workflow assessment strategy. Finally, consult the clinic staff on the best ways to adapt the intervention compatibility with the confirmed clinical workflow. The purpose of this paper is to provide a tutorial for eHealth researchers on how to assess clinical workflow and use the knowledge gained to adapt interventions for maximal implementation success.
Assessing Clinical Workflow
Workflow Assessment Steps
Clinical workflow assessments for implementation planning can follow 4 steps (): (1) the identification of discrete workflow components, (2) workflow assessment, (3) triangulation, and (4) the stakeholder proposal of intervention implementation.
|Identify discrete workflow components||Define what is necessary to make the intervention work||Select locations, interactions, and tasks||Review direct observation checklist () and other checklists to select clinical practices [ ]|
|Workflow assessment||Create a model of the clinical workflow||Direct observation, interviews, sensor-based investigations, EHRa audit logs, and job task diaries||Review direct observation form ()|
|Triangulation||Confirm rigor of clinical workflow model||Direct observation, interviews, sensor-based investigations, EHR audit logs, and job task diaries||Review semistructured template ()|
|Stakeholder proposal||Plan intervention implementation based on stakeholder preferences||Interviews||Review semistructured template ()|
aEHR: electronic health record.
Step 1: Identification of Discrete Workflow Components
The first step in conducting a clinical workflow analysis for improving eHealth implementation planning is to identify the discrete components of the clinical workflow that need to be measured. Owing to the multitude of tasks occurring during clinical visits, it is imperative to clearly define what activities researchers should track to ensure that each is collected and documented consistently. Tracked activities may fall under 3 observable workflow categories: location, interactions, and tasks. Location refers to where and how long individuals are physically present in specific areas of a clinic. Interactions include face-to-face conversations and moments in which health records or patients transition from one provider to another. Tasks include a review of health records, measures collected (eg, weight, blood pressure), interventions administered (eg, giving prescription or vaccine), and administrative functions (eg, patient check-in or scheduling). For each task considered, the actors (patient, provider, or support staff) may vary.
To identify the discrete workflow components for the intervention, consider what is necessary to make the intervention work as intended in the setting. Imagine how the intervention will be integrated into the clinic and considered the following questions: At what time during a clinic visit is the most useful for the patient and provider to access the intervention? What specific tasks need to occur to have the intervention in the hands of those who need it at the right time during the clinical encounter? What types of clinical situations require alternative planning? When choosing the discrete workflow components of interest, it is important to consider the minimum necessary rule and evaluate tasks directly relevant to an intervention’s implementation that do not compromise patient privacy or impede care delivery.
Step 2: Workflow Assessment
A second step in conducting a workflow analysis to improve eHealth intervention implementation is to assess the clinical workflow. Several strategies can be used to assess clinical workflow, including direct observation, clinical staff reporting, interviews with staff or patients, sensor-based investigations, EHR audit logs, and job task diaries [, , - ]. We focused on direct observation by a member of the research staff because this method offers advantages over other possible choices, including an in-depth analysis of complex interactions and clinical work-arounds, limited equipment and technology needs, and the potential to increase clinical staff engagement [ , , ].
Workflow assessment via direct observation can occur in 3 steps: (1) identify who you wish to observe, (2) complete informed consent with potential participants, and (3) directly observe the workflow using standardized data collection tools to systematically record observations. Before direct observation, engage with clinic leadership to ensure that the observed activities will not affect medical staff employment. Before the observation days, explain the informed consent to the identified staff so that they can thoughtfully consider participation. Observations should be scheduled for clinic days in which relevant patients have scheduled visits. Once patients express interest to clinical staff, patient consent can be obtained. Care must be taken to avoid disrupting the standard clinical flow to maintain high-quality research and clinical care. Although the number of observed patients can be influenced by practical considerations (eg, travel expenses and clinical receptiveness), to achieve a valid sample, patients should be observed until additional observations no longer present new information (ie, saturation is achieved) [, ].
The use of a standardized observation form increases the rigor and reproducibility of workflow observations and can be incorporated into any workflow assessment method chosen . The standardized observation form should include a simple method to record the location and primary actors for each of the discrete workflow components of interest. Multiple data collection forms and observers may be necessary because clinics have several actors attending to each patient simultaneously. Examples of standardized observation forms, including an adaptable Microsoft Access database, can be found in the AHRQ health information technology workflow assessment toolkit [ ]. Including timestamps of observed activities in the standardized observation form enables researchers to conduct what is considered the gold standard measurement for clinical workflow assessment, a time and motion study [ ]. Time and motion studies evaluate the activities and duration of each activity of the clinical workflow.
Step 3: Triangulation
A third step in conducting a clinical workflow analysis for improving eHealth intervention implementation is method triangulation: the verification of findings from a different viewpoint by answering similar questions with a different method . Triangulation is a common strategy for enhancing the rigor of qualitative studies [ ]. If triangulation reveals small deviations, the workflow should be updated. If triangulation reveals large deviations, additional data collection should be considered.
All the options mentioned for primary workflow assessment are possible methods for triangulation [, - ]. An additional option for triangulation is member checking of results, whereby the observed workflow is presented visually to the stakeholders via a sequential task analysis, line graphs, flow charts, and diagrams [ , , , ]. The AHRQ provides several examples of how to present workflow observations as flowcharts [ ].
Step 4: Stakeholder Proposal of Intervention Implementation
It is useful for stakeholders or actual clinical users to evaluate how the intervention will be compatible with their clinical workflow. Stakeholder opinions on implementation can be obtained during the triangulation phase through interviews or planning meetings. The planning process should follow the community participatory principles of engagement and participation so that the clinic staff feel that their insights and expertise are critical to the project . Examples of critical research staff attitudes include mutual respect and genuineness, transparent processes, and balancing power [ ]. Critical components for the success of stakeholder engagement include genuine partnerships, strategic selection of clinicians at each site to be involved in the project, and accommodation of stakeholder needs (eg, patient care responsibilities) [ ]. A framework for operationalizing the engagement of partners, including examples of practices from research projects, is available from the Patient-Centered Outcomes Research Institute [ ]. Finally, research staff should use this opportunity to discover or gain understanding into further adaptation needed for special situations (eg, acute visits vs wellness visits).
Overview: Protect Me 4
We developed a tablet-based eHealth app for primary care clinics called Protect Me 4. The app is intended for use by parents of adolescents at the beginning of their child’s appointment to help parents learn about recommended vaccines and explore educational information addressing common hesitations. At the time of the appointment, the app also prompts providers of due vaccines and parents’ selected hesitations to prepare the provider to lead more efficient and effective vaccine discussions.
Our pilot trial demonstrated that the intervention successfully increased initiation of our main target, human papillomavirus (HPV) vaccine . The app’s overall effectiveness was limited because only 8% (57/1062) of the eligible families used our app. On the basis of poststudy interviews with clinic staff, we hypothesized that the clinic workflow was not aligned with the implementation strategy. Thus, before our second study with an enhanced version of the app, we conducted a workflow analysis at 4 participating primary care clinics to understand how we could adapt our intervention to improve compatibility. All activities were approved by the University of Florida Institutional Review Board.
Step 1: Identification of Discrete Workflow Components
We identified 3 workflow-related conditions necessary for the success of the intervention. First, a parent needed 5-10 minutes to complete the application before the child saw the provider. Second, the provider needed to view the app results before (or during) the time spent with the patient. Third, the ordering and administration of vaccines needed to occur following parents’ and providers’ app use. As such, we needed to identify a time during a visit when the parent could complete the intervention, a method for getting the completed intervention to the provider for review, and ensure that both steps took place before the time when vaccines need to be ordered during a clinical encounter. Thus, we selected the following discrete workflow components of interest: (1) patient locations, interactions, and tasks before meeting with the provider; (2) provider interactions with the patient and patient records; (3) the timing of vaccine record review; and (4) the timing of vaccine administration. We restricted our workflow components to activities that occurred outside the clinic exam rooms in order to preserve patient privacy, encourage patient participation, and follow the minimum necessary rule for data collection.
Step 2: Workflow Assessment
To enhance the rigor of our direct observations, we created 2 standardized checklists by adapting the Arkansas Foundation for Medical Care Workflow Assessment Checklist found on the AHRQ website . The first checklist ( ) standardized recording observations of the patient from check-in to check-out and included fields about (1) patient locations, interactions, and tasks before meeting with the provider; (2) the timing of vaccine record review; and (3) the timing of vaccine delivery. The second checklist standardized the recording observations of the provider and included fields on provider interactions with patient records.
We created checklists in the MediDocs platform (EnMedical Systems), a software designed to capture medical information that includes a click to timestamp functionality. To ensure the usability and functionality of the Medidocs platform and enhance observer comfort with the procedures, observers created an example list of coffee shop procedures and then pilot tested the Medidocs platform at a local coffee shop by tracking employee and customer interactions. A coffee shop was selected because there were many individuals who followed the same procedures, and personal health information was not included.
We observed the clinical workflow at 4 primary care clinics that agreed to participate in the intervention implementation trial. The participating clinics were pediatric or family medicine specialties, served a range of patients with Medicaid insurance (14%-90% of patients), and were owned privately, by the university, or by health systems. Before the observation days, a lead clinical stakeholder at each clinic invited all medical staff who interacted with patients aged 11-12 years to attend an information session. Information sessions were held during each clinic’s preferred schedule (ie, during lunchtime or a staff meeting), food was provided, and research staff explained the study and informed consent forms. Clinic staff were given the opportunity to ask questions and return signed informed consent forms at any time before the observation days. The direct observation of clinical staff was restricted to those staff members who had returned signed consent forms.
To minimize clinic disruption and research staff travel expenses, we aimed to observe approximately 3 participants over 1-2 days at each clinic. Clinical staff selected days when at least one patient aged 11-12 years was scheduled. On each observation day, clinic staff invited all parents of patients aged 11-12 years to participate and referred interested parents to research staff who were present in the clinic. The research staff obtained consent through a Health Insurance Portability and Accountability Act waiver of documentation of consent and collected data without identifiers. Research staff conducting observations had a minimum of a bachelor’s degree in a scientific or social science field, had existing relationships with the clinic, and were trained in research and practice facilitation. We chose not to interview patients about the acceptability of completing the eHealth intervention during a clinic visit because parents were very receptive to completing the eHealth intervention during our pilot, and we enhanced parents’ acceptability of the app content with focus groups [, ].
With 12 patients as a practical target, we observed 13 patient visits between August 2016 and March 2017. By the 13th visit, we were no longer observing new variations in the workflow components. Thus, saturation was reached, and further data collection was not needed . However, we did continue to observe a variety of wait times but were able to conclude that sufficient time was typically available before the patient saw the provider.
Step 3: Triangulation
For the triangulation of our direct observation, we interviewed clinical stakeholders (group interviews of a clinical support staff member and the lead physician at each clinic) because involving the clinic staff in the development process creates a sense of ownership and increases buy-in consistent with the CFIR [, , ]. We chose to visually present the workflow using a common flowcharting strategy [ ]. The moderator was the same research staff who conducted direct observations. Each group followed a semistructured interview guide ( ). To review the observed workflow, flowcharts of the acute and preventive care workflows were displayed on large poster boards. A moderator explained the flowchart, asked about any missing components, and solicited opinions on accuracy. Clinic staff were compensated US $20 for interview participation in approximately 20-minute sessions. Interviews were recorded and transcribed. The flowchart was updated for minor deviations. Evaluation activities used a postpositivist approach to observe and understand the main activities involved in clinic appointments.
After noting considerable similarities across clinics and visit types, we created a composite flowchart that represented the workflow at all 4 clinics (). Our example flowchart shows that patients first arrived and were checked in with front office staff. Then, patients spent up to 59 minutes in the waiting room without staff contact (mean 12.6 min, SD 16.1 min in addition to time spent completing the study consent). In the 2 clinics, wait time was substantially less for acute than preventive visits. Next, a nurse or medical assistant triaged the patients’ presenting issue and collected basic vital information. Patients were then left alone in the exam room for a range of 2-25 minutes (mean 10.9 min, SD 8.8 min).
While a patient completed check-in and triage, physicians attended to other patients. Once notified that a patient was ready and waiting in an exam room, physicians reviewed the patient’s record for a few minutes immediately before entering the exam room. Depending on the clinic, patient records were reviewed via paper files placed immediately outside an exam room or electronically. Following the patient-physician interaction, physicians at each clinic communicated verbally with a nurse or medical assistant about additional care needs (eg, vaccine administration). The clinical staff administered ordered services. Finally, the patient returned to the front desk and scheduled another appointment or received the required instructions.
Step 4: Stakeholder Proposal of Intervention Implementation
Clinicians suggested additional staff double-checking that the tablets were distributed. Participants mentioned that checking for tablet distribution during triage would be compatible with redundancy checks already taking place within a clinical workflow. “We always double-check ages and things anyway...so if I see that the patient is 11 or more, I can always grab [the tablet].” Another site noted that it would be helpful to use a team approach to enhance intervention dissemination. To accomplish this, front office staff could highlight the patients eligible for the intervention during their morning patient reviews. For example, one clinician stated:
They [front desk staff] identify all the 11- and 12-year-olds coming in for the following day, and they somehow highlight, red flag, that particular patient, and when we discuss patients coming in...it can just be mentioned. So, everyone, front desk staff, and clinicians, will know that a patient needs an iPad.
At all 3 of these clinics, physicians preferred to review the intervention results at the same time they reviewed the patient records. Clinicians suggested physically placing the tablet with the encounter forms and patient records or accessing the intervention results on their computer when reviewing patient records. As one physician explained, “Usually I look through the patient chart before I walk into the room, so I would just do that [review intervention results] right after.” In cases where patients were unable to complete the intervention in the waiting room, physicians felt that the parent could continue to complete the tablet while the physician interacted with the adolescent. In explaining this alternative, a physician noted:
...at age 11 or 12, I do more interaction with the child versus interaction with the parent. So, I can say, “Oh, by the way mom, can you go ahead and fill this out for me?”
One clinic that already used a tablet system suggested a slightly modified implementation. The staff of this clinic proposed that the new intervention be distributed in a similar fashion to their current tablet system (ie, given by nurse to parent in the exam room). If the parent finished before the nurse completed triage activities and left the exam room, then the nurse would collect the tablet from the parent and place it outside the exam room for the physician to review. If a parent was still completing the tablet intervention during the completion of triage activities, the nurse would alert the physician that the parent still had the tablet. In this case, physicians reviewed the tablet results upon entering the room alongside the patient. One physician explained:
I’d bring it up and say, “Let me see how you feel about the HPV vaccine,” and if they are ready to have it, “great,” and if not, I would discuss if they had a hesitation.
Clinicians identified 3 potential barriers to intervention implementation. First, clinicians reported that they do not typically review and administer vaccines during acute care visits. Although it was noted that parents would still be amenable to completing the intervention during most acute care visits, it was less clear how medical staff could alter their work processes to incorporate speaking about and, potentially, administering vaccines during acute visits. This issue was first mentioned in relation to the lack of an established process for checking vaccine records for patients presenting for an acute care visit, with staff reporting, “We don’t usually check ahead for the sick visit. Just for the wells.”
Second, physicians mentioned patient conditions that would preclude their administration of vaccines (eg, a high fever) or cause disruptions to how patients move through the clinic, thus interfering with when a tablet could be given to a patient to complete the intervention. One clinician explained that “...if we know that [a patient] is really, really contagious, we won’t even do two minutes [in the waiting room], as soon as they come in we put them in an empty room.” Clinicians were open to the idea of administering the intervention in these acute visit scenarios, for example, stating, “We could implement [the tablet] at some point, like once we’ve discussed the issue or whatever they’ve come in for,” but noting that more work would need to be done in order to optimize the process for acute visits.
The final barrier mentioned was the physician time constraints. One physician mentioned that, despite the simplicity of the tablet, competing demands may cause him to forgo reviewing intervention results stating, “If I’m in a rush, or I’m behind, or if we have lots of add-ons, what have you...I mean, it’s simple enough to just log-in and everything, but it’s that extra step.”
We present a general strategy and an example for conducting a workflow assessment to enhance the implementation of an eHealth intervention in primary care. With this strategy, we were able to identify specific timing, staffing, and management processes that may enhance intervention implementation. Potential barriers related to the implementation of the intervention were identified and addressed by stakeholders. Incorporating clinical stakeholders into the design of the implementation plan enhanced stakeholder buy-in, a critical component of successful implementation of eHealth interventions . By conducting the workflow analysis before the implementation of an eHealth intervention, researchers may avoid potential implementation barriers that limit the reach and effectiveness of the intervention.
Conducting the workflow assessment before intervention implementation allows for adaptations consistent with the adaptable periphery of interventions in the CFIR. By using a direct observation approach to assess clinical workflow, we were able to identify how and when our eHealth intervention should be incorporated. Clinicians’ participation in verifying the workflow and proposing an implementation strategy allowed for additional options to be proposed before implementation. For example, clinicians added a double-check of the tablet distribution and provided contingency plans for when parents did not have enough time to finish the app before seeing the provider. Although the clinicians were unaware that our prior pilot had issues with front office staff forgetting to give the tablets to parents, incorporation of clinicians in the planning process pre-emptively addressed this shortcoming in our second implementation. Moreover, while we had anticipated a workflow tailored to each clinic based on prior research , conducting the workflow assessment allowed us to simplify our approach with a standardized implementation plan that could be applied to all clinics with minimal tailoring.
By engaging stakeholders with implementation planning, we were able to identify potential barriers, discourage clinic staff adaptation of the core intervention, and build clinician buy-in. Consistent with previous primary care workflow research , we found that the clinical workflow differed between acute and well visits, suggesting that it will be more challenging to use the app during acute visits. However, clinicians suggested work-arounds for acute visits that were inconsistent with the core of the app. For example, parents using the app after their initial discussion with the provider allows clinicians to address immediate concerns but does not allow the app to aid parent-provider conversations. Learning of this potential work-around before implementation allowed research staff to explain the inconsistencies between the work-around and the core of the intervention and work with clinicians to develop alternative solutions (eg, limiting use of the app to nonurgent acute visits). Finally, and most importantly, through their involvement with the planning, clinicians gained a sense of ownership, which is expected to translate to a more successful implementation.
There are 3 important limitations of the case study presented that could be improved in future workflow assessments. First, the workflow assessment focused on physicians and patients. Although physician and patient visits are the most frequent opportunity for administering adolescent vaccines, a workflow assessment that included nurses and other clinical staff could be useful if the intervention was implemented in nurse-only visits. Second, our triangulation and stakeholder engagement strategies focused on physicians and clinical support staff. In particular, as front office staff were identified as integral to the implementation plan, including front office staff in the workflow assessment and intervention implementation planning would likely improve planning and garner the needed support from these critical stakeholders. Third, our case study presented 4 clinics with 3 observations each and, in turn, the small number of observations increases the risk of undue influence of outliers if the visits are not representative of standard care practices. However, this size is reasonable for a small implementation study and is supported by thematic saturation .
Our proposed workflow assessment strategy and supporting case study have 3 strengths relative to the design and execution. First, the use of 2 strategies to assess workflow enhances the rigor and validity of the results. Second, the use of separate observers for the patient and the physician was relatively novel and ensured that overlapping activities were observed. This strategy of one observer to each actor can be expanded to any clinical setting. Third, the benefit of involving the clinic staff in implementation planning with regard to stakeholder ownership and engagement cannot be understated. Participating clinicians are interested in seeing the intervention succeed and are aware that some work-arounds may alter the core of the intervention.
Conducting a prospective workflow assessment can enhance the acceptability and performance of eHealth interventions in clinical practice. We presented a 4-step strategy to increase the compatibility between eHealth interventions and clinical workflow, as compatibility is one of the main predictors of clinician use [, ]. Multiple options are presented for each of the 4 steps: (1) the identification of discrete workflow components, (2) workflow assessment, (3) triangulation, and (4) the stakeholder proposal for intervention implementation. By following the included step-by-step guide, eHealth researchers can study clinical workflows to design eHealth implementation strategies that complement rather than compete with clinical care. More importantly, through engagement of clinical staff in implementation planning, research staff can increase the clinical staff buy-in to the intervention and are able to stress that adaptations, while possible, need to be implemented only after considering the effects on the core intervention.
The authors thank Jennifer Mauck (Master of Public Health) and Lori M Drum (Licensed Mental Health Counselor and Certified Clinical Research Coordinator) for their assistance with data collection. They also thank Shivani Desai (Master of Public Health) for manuscript formatting assistance. This research was funded by the National Institutes of Health, R21 CA202011 (principal investigator: SS).
SS led all aspects of the project. ES, MM, MV, LT, and SS recruited the clinics. SS, NR, ES, LT, MV, and MM planned the study. ES and MV performed direct observations and conducted semistructured interviews. SS and JT analyzed and interpreted the data and drafted the manuscript. JT, NR, ES, and SS created the tables and figures. All authors have reviewed and approved the final manuscript.
Conflicts of Interest
Direct observation checklist for patients and clinic staff. Boxes above and below the tables create timestamps upon clicking.PDF File (Adobe PDF File), 330 KB
Semistructured interview questions and subquestions were used to evaluate workflow observations.PDF File (Adobe PDF File), 97 KB
- Trivedi MH, Kern JK, Marcee A, Grannemann B, Kleiber B, Bettinger T, et al. Development and implementation of computerized clinical guidelines: barriers and solutions. Methods Inf Med 2002;41(5):435-442. [Medline]
- Liberati EG, Ruggiero F, Galuppo L, Gorli M, González-Lorenzo M, Maraldi M, et al. What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci 2017 Sep 15;12(1):113 [FREE Full text] [CrossRef] [Medline]
- Djalali S, Ursprung N, Rosemann T, Senn O, Tandjung R. Undirected health IT implementation in ambulatory care favors paper-based workarounds and limits health data exchange. Int J Med Inform 2015 Nov;84(11):920-932. [CrossRef] [Medline]
- Yeung MS, Lapinsky SE, Granton JT, Doran DM, Cafazzo JA. Examining nursing vital signs documentation workflow: barriers and opportunities in general internal medicine units. J Clin Nurs 2012 Apr;21(7-8):975-982. [CrossRef] [Medline]
- Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci 2016 Oct 26;11(1):146 [FREE Full text] [CrossRef] [Medline]
- Staras SAS, Vadaparampil ST, Livingston MD, Thompson LA, Sanders AH, Shenkman EA. Increasing human papillomavirus vaccine initiation among publicly insured Florida adolescents. J Adolesc Health 2015 May;56(5 Suppl):40-46 [FREE Full text] [CrossRef] [Medline]
- Sedlmayr B, Patapovas A, Kirchner M, Sonst A, Müller F, Pfistermeister B, et al. Comparative evaluation of different medication safety measures for the emergency department: physicians' usage and acceptance of training, poster, checklist and computerized decision support. BMC Med Inform Decis Mak 2013 Jul 29;13:79 [FREE Full text] [CrossRef] [Medline]
- Yadav K, Chamberlain JM, Lewis VR, Abts N, Chawla S, Hernandez A, et al. Designing real-time decision support for trauma resuscitations. Acad Emerg Med 2015 Sep;22(9):1076-1084 [FREE Full text] [CrossRef] [Medline]
- Cady RG, Finkelstein SM. Mixed-methods approach for measuring the impact of video telehealth on outpatient clinic triage nurse workflow. Comput Inform Nurs 2013 Sep;31(9):439-449 [FREE Full text] [CrossRef] [Medline]
- Cresswell KM, Mozaffar H, Lee L, Williams R, Sheikh A. Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals. BMJ Qual Saf 2017 Jul;26(7):542-551. [CrossRef] [Medline]
- Vigoda MM, Gencorelli F, Lubarsky DA. Changing medical group behaviors: increasing the rate of documentation of quality assurance events using an anesthesia information system. Anesth Analg 2006 Aug;103(2):390-395. [CrossRef] [Medline]
- What is workflow? Agency for Healthcare Research and Quality. URL: https://digital.ahrq.gov/health-it-tools-and-resources/evaluation-resources/workflow-assessment-health-it-toolkit/workflow [accessed 2021-01-19]
- Ozkaynak M, Unertl K, Johnson S, Brixey J, Haque S. Clinical workflow analysis, process redesign, and quality improvement. In: Clinical Informatics Study Guide. Cham, Switzerland: Springer International Publishing; 2016:135-161.
- Rule A, Chiang MF, Hribar MR. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020 Mar 01;27(3):480-490 [FREE Full text] [CrossRef] [Medline]
- Fontanesi J, De Guire M, Chiang J, Holcomb K, Sawyer M. Applying workflow analysis tools to assess immunization delivery in outpatient primary care settings. Jt Comm J Qual Improv 2000 Nov;26(11):654-660. [CrossRef] [Medline]
- Ampt A, Westbrook J, Creswick N, Mallock N. A comparison of self-reported and observational work sampling techniques for measuring time in nursing tasks. J Health Serv Res Policy 2007 Jan;12(1):18-24. [CrossRef] [Medline]
- Vankipuram M, Kahol K, Cohen T, Patel VL. Toward automated workflow analysis and visualization in clinical environments. J Biomed Inform 2011 Jun;44(3):432-440 [FREE Full text] [CrossRef] [Medline]
- Pagel C, Bates DW, Goldmann D. How to avoid common pitfalls of health IT implementation. NEJM Catal Innov Care Deliv. 2020. URL: https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0048 [accessed 2021-03-02]
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 07;4:50 [FREE Full text] [CrossRef] [Medline]
- Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004;82(4):581-629 [FREE Full text] [CrossRef] [Medline]
- Rogers EM. Diffusion of innovations, 4th Edition. New York, NY: Simon and Schuster; 2010.
- CFIR Research Team. Constructs. Consolidated Framework for Implementation Research (CFIR). 2021. URL: https://cfirguide.org/constructs/ [accessed 2021-01-15]
- Bally ELS, Cesuroglu T. Toward integration of mHealth in primary care in the Netherlands: a qualitative analysis of stakeholder perspectives. Front Public Health 2019;7:407 [FREE Full text] [CrossRef] [Medline]
- Unertl KM, Weinger MB, Johnson KB, Lorenzi NM. Describing and modeling workflow and information flow in chronic disease care. J Am Med Inform Assoc 2009;16(6):826-836 [FREE Full text] [CrossRef] [Medline]
- Workflow assessment for health IT toolkit. Agency for Healthcare Research and Quality. URL: https://digital.ahrq.gov/health-it-tools-and-resources/evaluation-resources/workflow-assessment-health-it-toolkit [accessed 2021-01-19]
- Checklist. Agency for Healthcare Research and Quality. URL: https://healthit.ahrq.gov/health-it-tools-and-resources/evaluation-resources/workflow-assessment-health-it-toolkit/all-workflow-tools/checklist [accessed 2021-01-19]
- Carayon P, Karsh BT, Cartmill RS. Job task diary. Prepared by the Center for Quality and Productivity Improvement, University of Wisconsin - Madison, under Contract No. HHSA 290-2008-10036C. Rockville, MD: Agency for Healthcare Research and Quality; 2010. URL: https://digital.ahrq.gov/sites/default/files/docs/workflowtoolkit/JobTaskDiary.pdf [accessed 2021-03-02]
- Wu DTY, Smart N, Ciemins EL, Lanham HJ, Lindberg C, Zheng K. Using EHR audit trail logs to analyze clinical workflow: a case study from community-based ambulatory clinics. AMIA Annu Symp Proc 2017;2017:1820-1827 [FREE Full text] [Medline]
- Bratt JH, Foreit J, Chen PL, West C, Janowitz B, de Vargas T. A comparison of four approaches for measuring clinician time use. Health Policy Plan 1999 Dec;14(4):374-381. [CrossRef] [Medline]
- Lopetegui M, Yen P, Lai A, Jeffries J, Embi P, Payne P. Time motion studies in healthcare: what are we talking about? J Biomed Inform 2014 Jun;49:292-299 [FREE Full text] [CrossRef] [Medline]
- Kannampallil T, Li Z, Zhang M, Cohen T, Robinson DJ, Franklin A, et al. Making sense: sensor-based investigation of clinician activities in complex critical care environments. J Biomed Inform 2011 Jun;44(3):441-454 [FREE Full text] [CrossRef] [Medline]
- Palinkas LA. Qualitative and mixed methods in mental health services and implementation research. J Clin Child Adolesc Psychol 2014;43(6):851-861 [FREE Full text] [CrossRef] [Medline]
- Padgett DK. Qualitative methods in social work research, 2nd ed. Los Angeles, CA: Sage Publications Inc; 2008:1-304.
- Work flow assessment checklist. Arkansas Foundation for Medical Care. 2005. URL: https://healthit.ahrq.gov/sites/default/files/docs/workflowtoolkit/Workflow_Assessment_Checklist.pdf [accessed 2021-01-19]
- Flick U, von Kardoff E, Steinke I. A companion to qualitative research. London: SAGE Publications Ltd; 2004:1-448.
- Creswell JW, Miller DL. Determining validity in qualitative inquiry. Theory Into Practice 2000 Aug;39(3):124-130. [CrossRef]
- Holman GT, Beasley JW, Karsh B, Stone JA, Smith PD, Wetterneck TB. The myth of standardized workflow in primary care. J Am Med Inform Assoc 2016 Jan;23(1):29-37 [FREE Full text] [CrossRef] [Medline]
- Flowchart. Agency for Healthcare Research and Quality. URL: https://digital.ahrq.gov/health-it-tools-and-resources/evaluation-resources/workflow-assessment-health-it-toolkit/all-workflow-tools/flowchart [accessed 2021-01-19]
- Rhodes SD, Tanner AE, Mann-Jackson L, Alonzo J, Simán FM, Song EY, et al. Promoting community and population health in public health and medicine: a stepwise guide to initiating and conducting community-engaged research. J Health Dispar Res Pract 2018;11(3):16-31 [FREE Full text] [Medline]
- Forsythe LP, Ellis LE, Edmundson L, Sabharwal R, Rein A, Konopka K, et al. Patient and stakeholder engagement in the PCORI pilot projects: description and lessons learned. J Gen Intern Med 2016 Jan;31(1):13-21 [FREE Full text] [CrossRef] [Medline]
- Sheridan S, Schrandt S, Forsythe L, Hilliard TS, Paez KA, Advisory Panel on Patient Engagement (2013 inaugural panel). The PCORI engagement rubric: promising practices for partnering in research. Ann Fam Med 2017 Mar;15(2):165-170 [FREE Full text] [CrossRef] [Medline]
- Theis RP, Wells BA, Staras SAS. "I can be the Judge of What's Serious": a qualitative pilot study of parents' responses to messaging about side effects of the HPV vaccine. Matern Child Health J 2020 Apr;24(4):456-461. [CrossRef] [Medline]
- Guest G, Bunce A, Johnson L. How many interviews are enough?: An experiment with data saturation and variability. Field Methods 2006 Feb 01;18(1):59-82. [CrossRef]
- Jun S, Plint AC, Campbell SM, Curtis S, Sabir K, Newton AS. Point-of-care cognitive support technology in emergency departments: a scoping review of technology acceptance by clinicians. Acad Emerg Med 2018 May;25(5):494-507 [FREE Full text] [CrossRef] [Medline]
|AHRQ: Agency for Healthcare Research and Quality|
|CFIR: Consolidated Framework for Implementation Research|
|EHR: electronic health record|
|HPV: human papillomavirus|
Edited by L Buis; submitted 20.04.20; peer-reviewed by D Pförringer, M Ozkaynak; comments to author 29.06.20; revised version received 16.09.20; accepted 22.02.21; published 31.03.21Copyright
©Stephanie Staras, Justin S Tauscher, Natalie Rich, Esaa Samarah, Lindsay A Thompson, Michelle M Vinson, Michael J Muszynski, Elizabeth A Shenkman. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 31.03.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.