Implementation research studies focus on real-world scenarios and identify factors that impact the uptake of research findings across multiple levels [23]. Within the context of this study, OCMIS are understood as sociotechnical systems, and the focus of this study is the human-computer interaction. This online usability study used a two-phased approach: first, there was the preceding pretest phase, which was followed by the main phase for data collection. During the pretest phase, approximately 10% of the expected participants from each group completed the study and provided feedback to researchers about the clarity and understandability of the study contents. Comments about wording obtained during the pretest were recorded as free text in digital form. Validation was performed with 2 native Spanish-speaking expert representatives from each user group and incorporated into the study after discussion. Changes to the study material were only incorporated when both experts agreed unanimously on significance and meaningfulness. Subsequently, the unmoderated main phase was conducted online, where participants acted in an in vivo setting. After the introduction video (Multimedia Appendix 2), the two-step study process was initiated: first, users completed 3 group-specific use cases with a randomly assigned OCMIS, followed by a post hoc questionnaire about the user experience during the study (Figure 1). All contents were administered in the native language. Data about the participants’ self-assessed health literacy [24,25] and OCMIS experience, as well as demographic data, were collected.

In addition, quantitative data were collected in parallel during user interaction to evaluate task success and task completion time. Data quality for the study was assured through a token system embedded in the process of accessing the study material. Pseudonymized tracking of participants without personal reference was possible, recognizing users that were not invited initially. The study was administered to participants via a URL to a self-hosted webpage where SurveyJS [26] was used for questionnaire rendering.

Participants of this study had no incentive other than to augment their knowledge about medications and OCMIS. The ethics committee at the Faculty of Medicine of the University of Chile approved this study.

In discussion with 2 domain experts from each user group, 6 OCMIS were identified as relevant. For patients, domain experts were head organizers of patient interest groups. Physician experts were academic professionals with expertise in public health, and pharmacist experts were represented through academic professionals. After interacting with each of the platforms, experts selected relevant OCMIS based on the information needed to fulfill typical use cases. OCMIS were categorized as online pharmacies (Farmazon [27], Pharol [28]), web presence of a traditional pharmacy (Salcobrand [29]), government-driven (Ministry of Health [MINSAL] [30]; Public Health Institute of Chile [ISP] [31]), and supplier-driven (National Health Service System/La Central Nacional de Abastecimiento [CENABAST] [32]) OCMIS. OCMIS were assigned to user groups based on a decision matrix based on their features to ensure suitability.

In addition to the aforementioned OCMIS, the reference implementation TMED (medical terminology) [33], based on an interoperable information model called Chilean Pharmaceutical Terminology [34], was part of the test bench for all user groups (Figure 2).

TMED is the result of an effort to create the first vendor-neutral, standardized, and interoperable information database using Fast Healthcare Interoperability Resources (FHIR), a standard developed by Health Level 7. The information model accommodates the Chilean pharmaceutical sector, enabling users to search for and view bioequivalent generic and brand medications as well as innovator products [22,34,35]. TMED supports identification of medication type by qualities and features and provides the possibility of grouping medications by principal active substances. However, in order to evaluate its fitness for use, use cases had to be defined.

Use cases are part of requirements engineering and are a narrative description of user actions and expected outcomes [36]. They allow the derivation of a feature set that must be provided by a system to the user in order to be of use. The aforementioned domain experts from each user group were prompted to write an easy-to-understand narrative text that outlines an everyday interaction with an OCMIS from their point of view, including prerequisites such as prescriptions for patients, diagnoses for physicians, and principal active substances for pharmacists (Textbox 1). Suitable exemplary medical conditions for the use cases were consented by all 6 domain experts (Textbox 2).

During the course of the study, each participant solved the use case for their group with 3 group-specific scenarios given in consecutive order, based on the use cases defined above. All scenarios are equal in structure and involve finding a medication for a specific medical condition, which facilitated participant learning and familiarization with the OCMIS. Subsequently, we established how OCMIS usability would be evaluated.

Usability evaluations are critical for assuring user acceptability when designing applications [40]. Approaches from pragmatic and academic contexts are relevant when conducting usability studies [41]. International Organization for Standardization (ISO) guideline 9241-11 includes 3 dimensions for usability: effectiveness, efficiency, and user satisfaction [42]. Effectiveness is expressed as task success, efficiency is expressed as task completion time, and user satisfaction is captured in a scoring system (eg, using the SUS).

A sample size calculation was conducted. Literature suggests a sample size of 12-14 as sufficient to distinguish user satisfaction reliably between websites [47]. However, a sample size calculation based on a desired margin of error of 12 points in SUS score with SD of 21 and confidence level of 90%, as suggested by the literature [48], resulted in a minimum sample size of 15 participants for each platform. Internal consistency was measured using Cronbach alpha. The literature suggests acceptable values range from .70 to .90 [49,50].

Inclusion and exclusion parameters were defined prior the study. Physicians had to have completed medical school; in Chile, this includes 2 years of practical experience in the field. Pharmacists had to have at least 1 year of professional experience. Patients were only included if they had bought medication at least once in their life. Possible participants were contacted via email invitation among special interest groups (eg, for pharmacists, invitations were sent to members of the College of Pharmaceutical and Biochemical Chemists of Chile). The data collection phase lasted 3 months and was followed up by statistical data analysis.

Initially, group-wise statistical tests were conducted, comparing platforms in terms of task time, task success, and SUS score. If results were statistically significant, an adjusted pairwise examination was performed to identify the significantly different feature. SUS score and task time were compared between OCMIS using the Kruskal-Wallis test for independent samples to compare means. Task success was evaluated using the chi-square test in combination with a standardized Z-score residual post hoc test. The Pearson chi-square test evaluated how likely it is that any observed difference between the sets arose by chance. Its null hypothesis states that the frequency distribution of certain events observed in a sample is consistent with a particular theoretical distribution [51]. This study evaluated the usability for OCMIS as shown in Figure 2.

Study participants included 136 patients, 80 physicians, and 67 pharmacists. The overall response rate was 283 of 4849 contacted individuals (5.8%). Table 1 provides an overview of study participant demographics. The mean ages across the different groups ranged from 31 to 38 years. Of the 136 patient participants, 87 (64%) were female, as were 36 of the 80 physician participants (45%) and 30 of the 68 pharmacists (45%). Self-assessed health literacy (where 5 indicated optimal health literacy and 1-4 indicated limited health literacy) of the study population varied between 30% and 35% for patients and pharmacists and peaked at over 50% within the physician group. Of the 67 pharmacists surveyed, 56 (83%) had used OCMIS prior to participating in this study, compared with 62 of 80 physicians (77%) and only 80 of 136 patients (58%). All participants from all groups reported that they used the internet on a daily basis, therefore data collected on internet use was not included in the overview.

To reduce the out-of-pocket spending for patients, 68 of 80 physicians (85%) reported considering the patient’s health insurance when prescribing medications, and 57 of 80 (71%) reported considering the economic situation of the patient.

When asked whether generics are bioequivalent to their respective innovator medication, 33 of 80 of physician participants (41%) stated that they are equal. In contrast, only 30 of 67 pharmacists surveyed (45%) agreed that innovator drugs could be replaced with generics without concern. In addition, 24 of 67 pharmacists (36%) disagreed and 12 of 67 pharmacists (18%) stated some concerns about replacing innovator drugs with generics.

The second usability measure was task success (Figure 3). Patients’ task success levels were relatively consistent, independent of which OCMIS was used, ranging from 84% (Pharol) to 92% (TMED).

On the other hand, physicians’ success was heavily platform-dependent, reaching a completion rate of just 67% on MINSAL and a 100% task success rate using TMED. Pharmacists’ task success rates ranged from 50% on the CENABAST platform to 92% on the ISP platform. TMED performance was in the middle of the group, with 75% of participants successfully completing the tasks.

Median task completion time in seconds for each task is shown in Figure 4. As the 3 tasks had the same structure, we hypothesized that task times would follow a downward trend; this was confirmed overall, with the exception of Farmazon and MINSAL in the patient group, where completion times increased slightly for the second and third task. In the case of TMED, initial task times are higher than with the other systems but with later tasks, the task times approach those of other OCMIS. Physicians took the least amount of time to finish the given tasks. An aggregated comparison can be found in Table 2.

The third dimension of usability, user satisfaction, proved to have a very high overall internal consistency, as indicated by a Cronbach alpha value of .89 for SUS scores. With one exception each in the patient and pharmacist groups, median SUS scores were above the global average of 68 (SD 12.5) for SUS scores for websites (Figure 5) [40].

The null hypothesis was defined as not exhibiting any differences for any of the given aspects (task time, task success, SUS score), with α=.05. Due to data skewness, normality was not assumed and subsequently only nonparametrical tests were performed.

In addition to quantitative data, 76 of 136 patients (55%), 36 of 80 physicians (45%), and 31 of 67 pharmacists (46%) provided qualitative feedback about features that they considered desirable for OCMIS. Comments were analyzed for their content and tagged by keyword (Textbox 3).

An online usability study was conducted to evaluate OCMIS on the dimensions of task success (completion), task completion time, and user satisfaction.

The ongoing controversy of whether to prescribe innovator medications or use bioequivalent generic products is reflected within the study population. Generally, physicians are slightly more confident in using generic products than pharmacists.

For patients, online pharmacies (Farmazon and Pharol) seemed to be the most suited to their tasks as indicated by high user satisfaction scores. Task time was significantly lower for the OCMIS of traditional pharmacies when compared to online pharmacies. Task success rates indicated that all platforms seemed to be suited for the use case.

Physicians seemed to have difficulties completing their tasks when using the MINSAL platform, but not when using the OCMIS of traditional pharmacies (Salcobrand) or the reference implementation (TMED).

The user satisfaction scores of pharmacists identified both ISP and TMED as the most usable platforms, with no significant difference in user satisfaction between them. The platform of public medication supplier CENABAST received lower SUS scores and also had lower task success rates.

For the selection of OCMIS, a discussion with 2 professional representatives was conducted; this may not be representative of which OCMIS are used by health care professionals on a national level. However, more than half of the participants indicated an awareness of the OCMIS presented in this study, indicating that the selected OCMIS were relevant. Health literacy was not homogeneous among participants, indicating unequal starting conditions for each participant; however, this reflects reality. Participant recruitment was carried out by email distribution to special interest groups, which might introduce bias as these individuals may have a higher awareness of OCMIS.

Due to the design of online usability studies, a unique combination of advantages was achieved. The study was not moderated and no social desirability response bias [54] was introduced by this in vivo setting, assuring the most natural conditions for the user while they evaluated the OCMIS. The study design facilitated the automated collection of qualitative and quantitative data directly after the experience. In comparison to traditional usability studies, a higher number of participants was recruited in a shorter time frame, which contributed to the robustness of the results.

This study demonstrated that TMED is a promising approach and showed that interoperable, neutral information models can empower stakeholders in context-agnostic medication decisions. Although an independent group should verify these results to avoid any potential bias, TMED was statistically proven to not be inferior to other OCMIS in usability aspects, while offering flexible search and extension capabilities due to its underlying interoperable information model.

Based on the results and qualitative feedback on desired features provided by participants, improvements can be incorporated to alleviate information asymmetries and foster data democratization within the pharmaceutical sector even further by providing user-tailored information. The approach of personalized drug information provision is promising and can serve as a basis for other applications, such as electronic prescriptions, and enable research opportunities through its standardized approach.