This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Medication is the most common intervention in health care, and the number of online consumer information systems within the pharmaceutical sector is increasing. However, online consumer information systems can be a barrier for users, imposing information asymmetries between stakeholders.
The objective of this study was to quantify and compare the usability of an online consumer medication information system (OCMIS) against a reference implementation based on an interoperable information model for patients, physicians, and pharmacists.
Quantitative and qualitative data were acquired from patients, physicians, and pharmacists in this online usability study. We administered 3 use cases and a post hoc questionnaire per user. Quantitative usability data including effectiveness (task success), efficiency (task time), and user satisfaction (system usability scale [SUS]) was complemented by qualitative and demographic data. Users evaluated 6 existing systems and 1 reference implementation of an OCMIS.
A total of 137 patients, 81 physicians, and 68 pharmacists participated in this study. Task success varied from 84% to 92% in patients, 66% to 100% in physicians, and 50% to 91% in pharmacists. Task completion time decreased over the course of the study for all but 2 OCMIS within the patient group. Due to an assumed nonnormal distribution of SUS scores, within-group comparison was done using the Kruskal-Wallis test. Patients showed differences in SUS scores (
The vendor-neutral reference implementation based on an interoperable information model was proven to be a promising approach that was not inferior to existing solutions for patients and physicians. For pharmacists, it exceeded user satisfaction scores compared to other OCMIS. This data-driven approach based on an interoperable information model enables the development of more user-tailored features to increase usability. This fosters data democratization and empowers stakeholders within the pharmaceutical sector.
Every medical decision is dependent on information, and thus, information quality is a key aspect when accessing health related information [
Online consumer medication information systems (OCMIS) try to take on these challenges by being a source for relevant medication information among patients and providers [
In Chile, an emerging middle-income country in Latin America [
After a feature analysis of OCMIS as part of a systematic review [
Implementation research studies focus on real-world scenarios and identify factors that impact the uptake of research findings across multiple levels [
In addition, quantitative data were collected in parallel during user interaction to evaluate task success and task completion time. Data quality for the study was assured through a token system embedded in the process of accessing the study material. Pseudonymized tracking of participants without personal reference was possible, recognizing users that were not invited initially. The study was administered to participants via a URL to a self-hosted webpage where SurveyJS [
Participants of this study had no incentive other than to augment their knowledge about medications and OCMIS. The ethics committee at the Faculty of Medicine of the University of Chile approved this study.
A graphical view of the study procedure is shown in a Business Process Model and Notation (BPMN). After reading the introduction and consenting to participate, the participants are randomly assigned to either the case group, which uses an online consumer medication information system (online system 1...n), or the control group, which uses the reference implementation (control system). A post hoc questionnaire was performed before concluding the study. OCMIS: online consumer medication information system.
In discussion with 2 domain experts from each user group, 6 OCMIS were identified as relevant. For patients, domain experts were head organizers of patient interest groups. Physician experts were academic professionals with expertise in public health, and pharmacist experts were represented through academic professionals. After interacting with each of the platforms, experts selected relevant OCMIS based on the information needed to fulfill typical use cases. OCMIS were categorized as online pharmacies (Farmazon [
In addition to the aforementioned OCMIS, the reference implementation TMED (medical terminology) [
TMED is the result of an effort to create the first vendor-neutral, standardized, and interoperable information database using Fast Healthcare Interoperability Resources (FHIR), a standard developed by Health Level 7. The information model accommodates the Chilean pharmaceutical sector, enabling users to search for and view bioequivalent generic and brand medications as well as innovator products [
Graphical representation of the patient (green), physician (red), and pharmacist (yellow) user groups and their assigned online consumer medication information systems. Online system types include online pharmacies (Pharol, Farmazon), a traditional pharmacy (Salcobrand), medication information aggregators (MINSAL), medication information platforms (CENABAST, ISP), and a self-developed platform (TMED). CENABAST: National Health Service System of Chile; ISP: Public Health Institute of Chile; MINSAL: Ministry of Health of Chile; TMED: medication terminology.
Use cases are part of requirements engineering and are a narrative description of user actions and expected outcomes [
During the course of the study, each participant solved the use case for their group with 3 group-specific scenarios given in consecutive order, based on the use cases defined above. All scenarios are equal in structure and involve finding a medication for a specific medical condition, which facilitated participant learning and familiarization with the OCMIS. Subsequently, we established how OCMIS usability would be evaluated.
Patient: finding a suitable commercial product for a prescription received from a physician.
Physician: finding a suitable commercial product to prescribe for a patient based on a principal active substance indicated for a diagnosis.
Pharmacist: finding a suitable commercial product to restock a pharmacy, based on the need for principal active substances issued by physicians.
Atypical pneumonia, which has a growing prevalence in the Chilean population [
Focal epilepsy, one of the most common neuronal diseases worldwide; the majority of individuals with focal epilepsy (80%) live in low- and middle-income countries [
Hypertension, one of the most common diseases; it affects more than 3.6 million in Chile and 1.3 billion worldwide [
Usability evaluations are critical for assuring user acceptability when designing applications [
The first usability dimension was measured on 3 discrete levels: complete success, partial success, and not successful. Results were aggregated dichotomously over all 3 tasks resolved by the participant by defining anything other than a complete success as not successful. Overall success was achieved if at least 2 tasks were completed successfully by the user.
Task completion time in seconds was measured automatically during the study for each task and user.
User satisfaction was measured using the well-established SUS, which yields a score between 0 and 100 [
A sample size calculation was conducted. Literature suggests a sample size of 12-14 as sufficient to distinguish user satisfaction reliably between websites [
Inclusion and exclusion parameters were defined prior the study. Physicians had to have completed medical school; in Chile, this includes 2 years of practical experience in the field. Pharmacists had to have at least 1 year of professional experience. Patients were only included if they had bought medication at least once in their life. Possible participants were contacted via email invitation among special interest groups (eg, for pharmacists, invitations were sent to members of the College of Pharmaceutical and Biochemical Chemists of Chile). The data collection phase lasted 3 months and was followed up by statistical data analysis.
Initially, group-wise statistical tests were conducted, comparing platforms in terms of task time, task success, and SUS score. If results were statistically significant, an adjusted pairwise examination was performed to identify the significantly different feature. SUS score and task time were compared between OCMIS using the Kruskal-Wallis test for independent samples to compare means. Task success was evaluated using the chi-square test in combination with a standardized Z-score residual post hoc test. The Pearson chi-square test evaluated how likely it is that any observed difference between the sets arose by chance. Its null hypothesis states that the frequency distribution of certain events observed in a sample is consistent with a particular theoretical distribution [
Study participants included 136 patients, 80 physicians, and 67 pharmacists. The overall response rate was 283 of 4849 contacted individuals (5.8%).
To reduce the out-of-pocket spending for patients, 68 of 80 physicians (85%) reported considering the patient’s health insurance when prescribing medications, and 57 of 80 (71%) reported considering the economic situation of the patient.
When asked whether generics are bioequivalent to their respective innovator medication, 33 of 80 of physician participants (41%) stated that they are equal. In contrast, only 30 of 67 pharmacists surveyed (45%) agreed that innovator drugs could be replaced with generics without concern. In addition, 24 of 67 pharmacists (36%) disagreed and 12 of 67 pharmacists (18%) stated some concerns about replacing innovator drugs with generics.
Baseline table of the participants.
Characteristic | Patients (n=136) | Physicians (n=80) | Pharmacists (n=67) | |
Age (years), mean (SD) | 38 (11.2) | 31 (6.2) | 35 (9.2) | |
|
||||
|
Female | 87 (64) | 36 (45) | 30 (45) |
|
Male | 49 (36) | 44 (55) | 37 (55) |
|
||||
|
Limited | 85 (65) | 36 (47) | 45 (68) |
|
Optimal | 46 (35) | 41 (53) | 21 (32) |
Professional experience, mean (SD)b | N/Ac | 6.57 (6.6) | 8.86 (7.8) | |
|
||||
|
Yes | 77 (59) | 60 (78) | 55 (83) |
|
No | 54 (41) | 17 (22) | 11 (17) |
|
||||
|
Yes | N/A | 33 (41) | 24 (36) |
|
No | N/A | 41 (51) | 30 (46) |
|
Other | N/A | 6 (8) | 12 (18) |
|
||||
|
Farmazon | 32 | —e | — |
|
Pharol | 30 | — | — |
|
Salcobrand | 44 | 39 | — |
|
MINSALf | 15 | 18 | — |
|
CENABASTg | — | — | 20 |
|
ISPh | — | — | 28 |
|
TMEDi | 15 | 23 | 19 |
aThese values represent self-assessed health literacy as captured by a single-item, 5-point Likert scale where 1-4 indicated limited health literacy and 5 indicated optimal health literacy.
bProfessional experience was measured in years since graduation from university.
cN/A: not applicable.
dIf Yes was indicated, the participant had used an online consumer medication information system at least once before this study.
eNot applicable.
fMINSAL: Ministry of Health of Chile.
gCENABAST: National Health Service System of Chile.
hISP: Public Health Institute of Chile.
iTMED: medication terminology.
The second usability measure was task success (
On the other hand, physicians’ success was heavily platform-dependent, reaching a completion rate of just 67% on MINSAL and a 100% task success rate using TMED. Pharmacists’ task success rates ranged from 50% on the CENABAST platform to 92% on the ISP platform. TMED performance was in the middle of the group, with 75% of participants successfully completing the tasks.
Binary task success rates for online consumer medication information systems: successful (light) and not successful (dark). CENABAST: National Health Service System of Chile; ISP: Public Health Institute of Chile; MINSAL: Ministry of Health of Chile; TMED: medication terminology.
Median task completion time in seconds for each task is shown in
Median task completion times for patients (left), physicians (center), and pharmacists (right). Times per task 1 (dark), task 2 (lighter), and task 3 (lightest) are shown with a 95% CI. CENABAST: National Health Service System of Chile; ISP: Public Health Institute of Chile; MINSAL: Ministry of Health of Chile; TMED: medication terminology.
Overview of task success, task time, and system usability scale scores for all user groups by online consumer medication information system.
Characteristic | Farmazon |
Pharol |
Salcobrand (nPat=44, nPhyb=39) | MINSALc (nPat=15, nPhy=18) | CENABASTd (nPhae =20) | ISPf (nPha =28) | TMEDg (nPat=15, nPhy=23, nPha=19) | |
|
||||||||
|
Patient | 89.6 | 84.0 | 83.8 | 84.6 | —i | — | 92.3 |
|
Physician | — | — | 97.4 | 66.7 | — | — | 100 |
|
Pharmacist | — | — | — | — | 50.0 | 91.7 | 77.8 |
|
||||||||
|
Patient | 50.33 (27.61) | 60.67 (50.53) | 51.33 (74.03) | 63.68 (61.89) | — | — | 64.33 (32.65) |
|
Physician | — | — | 50.00 (236.52) | 61.00 (478.19) | — | — | 56.67 (179.78) |
|
Pharmacist | — | — | — | — | 42.33 (42.55) | 47.67 (31.54) | 68.00 (33.03) |
|
||||||||
|
Patient | 83.83 (15.18, 78.46-89.74) | 76.38 (19.71, 69.13-84.11) | 66.73 (23.87, 59.52-74.39) | 71.33 (24.72, 58.02-85.31) | — | — | 72.67 (15.36, 64.41-81.32) |
|
Physician | — | — | 79.66 (15.89, 74.61-85.22) | 77.06 (22.45, 65.69-88.78) | — | — | 76.85 (17.23, 69.66-84.60) |
|
Pharmacist | — | — | — | — | 50.63 (22.24, 40.43-61.27) | 79.81 (20.68, 71.87-88.21) | 84.87 (11.62, 79.50-90.71) |
anPat: number of patients.
bnPhy: number of physicians.
cMINSAL: Ministry of Health of Chile.
dCENABAST: National Health Service System of Chile.
enPha: number of pharmacists.
fISP: Public Health Institute of Chile.
gTMED: medication terminology.
hPercentage of aggregated task success rates.
iNot applicable.
jThe median task time is in seconds.
kSUS: system usability scale; scores can be values between 0 and 100.
The third dimension of usability, user satisfaction, proved to have a very high overall internal consistency, as indicated by a Cronbach alpha value of .89 for SUS scores. With one exception each in the patient and pharmacist groups, median SUS scores were above the global average of 68 (SD 12.5) for SUS scores for websites (
SUS box plots: the lower (Q1) and upper (Q3) quartile, representing observations outside the 9-91 percentile range. The diagram also shows the median observation. Data falling outside the Q1-Q3 range are plotted as outliers. CENABAST: National Health Service System of Chile; ISP: Public Health Institute of Chile; MINSAL: Ministry of Health of Chile; SUS: system usability scale; TMED: medication terminology.
The observed mean SUS scores for TMED ranged from 72.5 (SD 15.36) for patients to 76.85 (SD 17.23) for physicians and 84.87 (SD 11.62) for pharmacists (
Transformation of TMED system usability scale scores into percentile ranks, adjectives, and grades for patients, physicians, and pharmacists.
Parameters | Patients | Physicians | Pharmacists |
System usability scale score, mean (SD) | 72.67 (15.36) | 76.85 (17.23) | 84.87 (11.62) |
Percentage | 66.9 | 88.0 | 96.6 |
Adjective | Good | Excellent | Excellent |
Grade (Bangor [ |
C | B | B |
Grade (Sauro & Lewis [ |
B– | A– | A+ |
The null hypothesis was defined as not exhibiting any differences for any of the given aspects (task time, task success, SUS score), with α=.05. Due to data skewness, normality was not assumed and subsequently only nonparametrical tests were performed.
In the patient group, the differences in SUS scores (
For the physician group, the differences in SUS scores (
The results from the pharmacist group indicated a highly significant difference between OCMIS for SUS scores (
In addition to quantitative data, 76 of 136 patients (55%), 36 of 80 physicians (45%), and 31 of 67 pharmacists (46%) provided qualitative feedback about features that they considered desirable for OCMIS. Comments were analyzed for their content and tagged by keyword (
The up-to-date or approximated medication price should be displayed (132 mentions).
Search flexibility should be increased (eg, searching for principal active substances or quality parameters; 11 mentions).
Disambiguation of search terms (eg, phonetic searches) should be provided (10 mentions).
Medication concentrations should be displayed (6 mentions).
Adverse effect information should be provided (6 mentions).
An increased amount of information about medications (eg, kinetics and posology) should be included (4 mentions).
Evidence for medications should be shown (3 mentions).
Filters for information such as dosage or concentration should be implemented (3 mentions).
Integration to other knowledge databases should be considered (3 mentions).
Georeferenced information for pharmacies and stock considerations should be included (2 mentions).
Personal discounts due to insurance coverage should be included in the price calculation (1 mention).
Information neutrality should be a priority (1 mention).
Native mobile applications should be preferred (1 mention).
An online usability study was conducted to evaluate OCMIS on the dimensions of task success (completion), task completion time, and user satisfaction.
The ongoing controversy of whether to prescribe innovator medications or use bioequivalent generic products is reflected within the study population. Generally, physicians are slightly more confident in using generic products than pharmacists.
For patients, online pharmacies (Farmazon and Pharol) seemed to be the most suited to their tasks as indicated by high user satisfaction scores. Task time was significantly lower for the OCMIS of traditional pharmacies when compared to online pharmacies. Task success rates indicated that all platforms seemed to be suited for the use case.
Physicians seemed to have difficulties completing their tasks when using the MINSAL platform, but not when using the OCMIS of traditional pharmacies (Salcobrand) or the reference implementation (TMED).
The user satisfaction scores of pharmacists identified both ISP and TMED as the most usable platforms, with no significant difference in user satisfaction between them. The platform of public medication supplier CENABAST received lower SUS scores and also had lower task success rates.
For the selection of OCMIS, a discussion with 2 professional representatives was conducted; this may not be representative of which OCMIS are used by health care professionals on a national level. However, more than half of the participants indicated an awareness of the OCMIS presented in this study, indicating that the selected OCMIS were relevant. Health literacy was not homogeneous among participants, indicating unequal starting conditions for each participant; however, this reflects reality. Participant recruitment was carried out by email distribution to special interest groups, which might introduce bias as these individuals may have a higher awareness of OCMIS.
Due to the design of online usability studies, a unique combination of advantages was achieved. The study was not moderated and no social desirability response bias [
This study demonstrated that TMED is a promising approach and showed that interoperable, neutral information models can empower stakeholders in context-agnostic medication decisions. Although an independent group should verify these results to avoid any potential bias, TMED was statistically proven to not be inferior to other OCMIS in usability aspects, while offering flexible search and extension capabilities due to its underlying interoperable information model.
Based on the results and qualitative feedback on desired features provided by participants, improvements can be incorporated to alleviate information asymmetries and foster data democratization within the pharmaceutical sector even further by providing user-tailored information. The approach of personalized drug information provision is promising and can serve as a basis for other applications, such as electronic prescriptions, and enable research opportunities through its standardized approach.
Benchmarking of online consumer medication information systems. This benchmark was used in a previous publication [22].
Introduction and overview video for the online usability study.
National Health Service System/La Central Nacional de Abastecimiento
Fast Healthcare Interoperability Resources
Public Health Institute of Chile
Ministry of Health
online consumer medication information system(s)
system usability scale
medication terminology (Spanish: Terminologia de Medicamentos)
The authors thank the following institutions for their support: Centro Nacional en Sistemas de Información en Salud (CORFO 16CTTS-66390), Biomedical Neuroscience Institute (ICM, P09-015-F), DAAD PAGEL CHIP: Chilean Health Info and Process Challenge (DAAD 57220037), DAAD Scholarship: binationally supervised doctorates (DAAD 57314603) and the MOLIT Institute.
We are responsible for the development of TMED, one of the tested online consumer medication information systems. No other conflicts of interest are declared.