Background: ClinicalConnect, a federated clinical viewer for South West Ontario, Canada, launched a mobile interface in June 2012.
Objective: The aim of the study was to assess usability of the mobile interface and the perceived impact on productivity of health care providers and quality of healthcare delivery.
Methods: A survey was conducted using the System Usability Scale (SUS) and questionnaires designed to measure productivity and quality based on Canada Health Infoway's Benefits Evaluation framework.
Results: The mean SUS score was 67 based on 77 responses. The mean scores for productivity and quality were 3.37 (N=74) and 3.62 (N=71), respectively, on a 5-point Likert scale where 3 was neutral.
Conclusions: Users perceived the mobile interface of ClinicalConnect as useful but were neutral about the ease of use.
Mobile devices are rapidly becoming a part of everyday life as both communication and information tools. Recent studies indicate that this is true in the case of health care providers as well. Health care providers often use their mobile devices to access educational material, pharmaceutical compendiums, practice guidelines, clinical pathways, and electronic medical records .
Electronic health information exchange (HIE), in which patients’ clinical data is efficiently shared between care delivery settings, is expected to produce a number of quality improvements and cost savings . ClinicalConnect is a federated clinical viewer that provides health information exchange between health services and care providers of South West Ontario [ ]. It is considered to be one of the largest federated HIE models in North America. A mobile version of ClinicalConnect was developed and deployed in June 2012.
Health care providers often fail to realize the intended effects of their eHealth systems due to inadequate usability . Despite the widely held belief that the computerization of health information systems contributes to improved quality of patient care and care management, studies have found medical staff attitudes toward computerization to be negative [ ]. In order to discern whether the ClinicalConnect mobile version was meeting user requirements, an evaluation survey was conducted in November 2013. A range of survey tools was used for the assessment with a focus on usability, and the perceived impact of the mobile version on the quality of patient care and productivity of health care providers.
The System Usability Scale (SUS), a nonproprietary validated survey tool , was used to assess the usability of the ClinicalConnect mobile interface. SUS includes 10 statements presented on a 5-point Likert scale, which results in an overall score from 0 to 100 that indicates the perceived usability of the interface.
Canada Health Infoway published a Benefits Evaluation Indicators Technical Report in 2006 that was subsequently updated in 2012 to provide guidance for benefits evaluation planning related to information and communications technology (ICT) in health care . The framework proposes the indicators that could have an impact on productivity and quality of health care. Based on these indicators, productivity and quality questionnaires using a 5-point Likert scale [ ] were designed. Productivity and quality questionnaires had 8 ( ) and 10 questions ( ), respectively.
The SUS, productivity, and quality survey tools were hosted and administered using SurveyMonkey .
Five hundred and four health care providers who expressed interest in participating in periodic evaluations while signing up for the HIE service were sent an e-mail with a link to the survey. A second reminder e-mail was sent if no response was received from a provider. One hundred and ten providers responded to the survey, though some surveys were incomplete. Seventy-seven responses were obtained for SUS. On productivity and quality scores, we received 74 and 71 responses, respectively.
Results of the SUS questionnaire were recoded and normalized, and the mean SUS score and the standard deviation were then recorded. The Likert-type questions for productivity and quality were independently tabulated. Since all questions within each category measure a single concept, the values were combined into a composite score by calculating the mean and standard deviation. Scores for negatively worded questions (fifth question in the productivity matrix and second and third questions in the quality matrix) were normalized prior to the calculation of the composite score.
Eighty-five (77%) of the respondents were physicians. The mean SUS score was 67 (SD 14.4) with a percentile score of 46.9 . This means that the mobile version of ClinicalConnect can be considered more usable than 46.9% of all products evaluated with the SUS instrument.
depicts the responses to the productivity-related questions and summarizes the quality questions. The mean productivity score was 3.37 (SD 1.06) and the mean quality score was 3.62 (SD 0.99), with 3 being neutral on the 5-point Likert scale.
|Answer options||Strongly agree||Agree||Neither agree nor disagree||Disagree||Strongly disagree|
|n (%)||n (%)||n (%)||n (%)||n (%)|
|Accessing patient/client test results and information on my mobile device has decreased time spent tracking down or waiting for these reports||15(20)||27(36)||17(23)||13(18)||2(3)|
|Using a mobile device allows me to spend more face-to-face time with my patient/client||5(7)||19(26)||31(42)||18(24)||1(1)|
|I have more portability as I can now access ClinicalConnect with a mobile device||19(26)||47(64)||4(5)||3(4)||1(1)|
|The “New Results” feature draws my attention to results quickly||10(14)||34(46)||22(30)||6(8)||2(3)|
|It is difficult to view patient information on my mobile device screen||6(8)||21(28)||22(30)||21(28)||4(5)|
|I can access information faster on my mobile device than on a desktop computer||3(4)||18(24)||20(27)||17(23)||16(22)|
|Treatment decisions are made faster now that I can access patient/client information on my mobile device||8(11)||20(27)||32(43)||14(19)||0(0)|
|Access to patient information on my mobile device allows for better communication between health care providers||14(19)||31(42)||22(30)||7(9)||0(0)|
|Answer options||Strongly agree||agree||Neither agree nor disagree||Disagree||Strongly disagree|
|n (%)||n (%)||n (%)||n (%)||n (%)|
|Education activities are enhanced when the patient/client can view their results on my mobile device with me||7(10)||16(23)||33(46)||14(20)||1(1)|
|There is high risk that a mobile device can be the source of a nosocomial infection||3(4)||11(15)||24(34)||24(34)||9(13)|
|Current results are not easy to access on a mobile device||5(7)||16(23)||16(23)||25(35)||9(13)|
|A mobile device allows faster access to vital patient information facilitating quicker consultation, diagnostic tests, and interventions||14(20)||29(41)||17(24)||11(15)||0(0)|
|Access to patient/client information anywhere, anytime enhances consultations, referrals, and handoffs||17(24)||38(54)||10(14)||5(7)||1(1)|
|I am more confident that my mobile device is cleaner than the desktop computer that is used by multiple people||11(15)||24(34)||23(32)||8(11)||5(7)|
|I can better prioritize my actions to follow-up on test results with the “New Results” feature||6(8)||29(41)||31(44)||4(6)||1(1)|
|I feel more confident in care decisions because I have the information I need at my fingertips||15(21)||30(42)||22(31)||3(4)||1(1)|
|I am less likely to order a duplicate test because I have easier access to current results||15(21)||36(51)||16(23)||3(4)||1(1)|
|HCPs should have access to their patient’s/client’s health information no matter where they are||33(46)||29(41)||7(10)||2(3)||0(0)|
HIE is the process of sharing electronic health information between different providers and organizations. As in most other health information systems, mobile devices are increasingly becoming popular as an HIE platform .
The SUS consists of 10 alternating positively and negatively worded statements scored on a 5-point Likert scale. The mean SUS score for 2324 surveys about usability conducted over a 10-year period was 70.14 with a median score of 75 .
Inadequate usability is a major cause of failure of eHealth systems , especially of mobile platforms. Our data show that health service providers perceive the mobile interface of ClinicalConnect as useful, but are neutral about the ease of use. This pattern has been noticed before in other mHealth interventions [ ]. The ease of use can be affected by factors beyond mobile user interface such as ergonomic and social aspects [ ].
HIE is vital for improving efficiency and quality of health care. It has been demonstrated that perceived usefulness is a stronger predictor of the use of an eHealth technology than the perceived ease of use . Most of our respondents agreed that ClinicalConnect on mobile devices had a positive impact on their productivity and quality. Anywhere, anytime access to patient information on mobile devices was perceived as an important factor in faster consultations, referrals, and handoffs, and in improved communication between health care providers. However, information access on mobile devices was not considered faster than the desktop counterpart, and there was no consensus on whether mobile devices allow more face-to-face time with patients. This corroborates previous studies that showed that physicians consider improvements in the quality domain as the overarching benefit of HIE [ ].
Though HIE programs have demonstrated clinical value in some situations such as emergency departments , a consistent empirical proof of value is lacking [ ]. In a study, emergency physicians reported workflow disruptions from HIE use [ ]. Our study suggests that mobile access to integrated health records is perceived as beneficial, especially to the quality of patient care. However, our sample size may be insufficient and not representative enough of the health care roles for us to statistically draw a precise conclusion.
It is important for HIE system implementations to be integrated into the care practice improvement process . The study shows that bringing data from disparate health care systems to the point of care via mobile HIE systems has a perceived potential for improvements in patient care.
Funding for this ClinicalConnect mobile project and Benefits Evaluation as well as funding for the publication costs was provided by Canada Health Infoway Inc, a non-profit corporation funded by the Government of Canada. The funders had no role in study design, data collection and analysis, or decision to publish the manuscript. In-kind services support was provided by the Hamilton Niagara Haldimand Brant (HNHB) eHealth Office and associated health care providers. The authors would like to acknowledge the support of Dale Anderson, John Haywood, Debra Hutchinson, Rob Lloyd, and Kate Black.
ClinicalConnect is a trademark of the public hospital Hamilton Health Sciences. ClinicalConnect is jointly funded by public hospitals and agencies as well as eHealth Ontario.
Conflicts of Interest
The study was conducted by the HNHB eHealth Office/Hamilton Health Sciences, the regional solution provider deploying ClinicalConnect regionally, and the service delivery partner for the HNHB local health integration network (LHIN).
- Sclafani J, Tirrell TF, Franko OI. Mobile tablet use among academic physicians and trainees. J Med Syst 2013 Feb;37(1):9903 [FREE Full text] [CrossRef] [Medline]
- Lammers EJ, Adler-Milstein J, Kocher KE. Does health information exchange reduce redundant imaging? Evidence from emergency departments. Med Care 2014 Mar;52(3):227-234. [CrossRef] [Medline]
- ClinicalConnect Connecting South West Ontario. URL: http://info.clinicalconnect.ca/CC/ [accessed 2014-10-01] [WebCite Cache]
- Scandurra I, Hägglund M, Persson A, Ahlfeldt RM. Disturbing or facilitating?--on the Usability of Swedish eHealth Systems 2013. Stud Health Technol Inform 2014;205:221-225. [Medline]
- Kagan I, Fish M, Farkash-Fink N, Barnoy S. Computerization and its contribution to care quality improvement: the nurses' perspective. Int J Med Inform 2014 Dec;83(12):881-888. [CrossRef] [Medline]
- Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 2008 Jul 30;24(6):574-594. [CrossRef]
- Lau F. Extending the infoway benefits evaluation framework for health information systems. Stud Health Technol Inform 2009;143:406-413. [Medline]
- Komorita SS. Attitude content, Intensity, and the neutral point on a Likert Scale. J Soc Psychol 1963 Dec;61:327-334. [CrossRef] [Medline]
- SurveyMonkey: Free online survey software & questionnaire tool. URL: https://www.surveymonkey.com/ [accessed 2014-10-01] [WebCite Cache]
- Kalz M, Lenssen N, Felzen M, Rossaint R, Tabuenca B, Specht M, et al. Smartphone apps for cardiopulmonary resuscitation training and real incident support: a mixed-methods evaluation study. J Med Internet Res 2014;16(3):e89 [FREE Full text] [CrossRef] [Medline]
- Bouri N, Ravi S. Going mobile: how mobile personal health records can improve health care during emergencies. JMIR Mhealth Uhealth 2014;2(1):e8 [FREE Full text] [CrossRef] [Medline]
- Vélez O, Okyere PB, Kanter AS, Bakken S. A usability study of a mobile health application for rural Ghanaian midwives. J Midwifery Womens Health 2014;59(2):184-191 [FREE Full text] [CrossRef] [Medline]
- Svanaes D, Alsos OA, Dahl Y. Usability testing of mobile ICT for clinical settings: methodological and practical challenges. Int J Med Inform 2010 Apr;79(4):e24-e34. [CrossRef] [Medline]
- Davis FD. Perceived Usefulness, Perceived Ease of Use,User Acceptance of Information Technology. MIS Q. Society for Information Management and The Management Information Systems Research Center 1989;13(3):319. [CrossRef]
- Khurshid A, Diana ML, Luce SD. Health information exchange: metrics to address quality of care and return on investment. Perspect Health Inf Manag 2012;9:1e [FREE Full text] [Medline]
- Tzeel A, Lawnicki V, Pemble KR. "Hidden" value: how indirect benefits of health information exchange further promote sustainability. Am Health Drug Benefits 2012 Sep;5(6):333-341 [FREE Full text] [Medline]
- Overhage JM, Evans L, Marchibroda J. Communities' readiness for health information exchange: the National Landscape in 2004. J Am Med Inform Assoc 2005;12(2):107-112 [FREE Full text] [CrossRef] [Medline]
- Thorn SA, Carter MA, Bailey JE. Emergency physicians' perspectives on their use of health information exchange. Ann Emerg Med 2014 Mar;63(3):329-337. [CrossRef] [Medline]
|HIE: health information exchange|
|HNHB: Hamilton Niagara Haldimand Brant|
|ICT: information and communications technology|
|LHIN: local health integration network|
|SUS: System Usability Scale|
Edited by G Eysenbach; submitted 06.11.14; peer-reviewed by C Dias, M Kalz; comments to author 20.02.15; revised version received 27.02.15; accepted 23.03.15; published 14.04.15Copyright
©Bell Raj Eapen, Barbara Chapman. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 14.04.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.