Over the past decade, the internet has evolved into a primary source of knowledge for many individuals. The demand for purchasing things online, including medications, has experienced a parallel surge [1]. In the last few years, a discernible transformation has occurred within the pharmacy industry, whereby a considerable number of conventional pharmacies have adopted e-commerce platforms. These internet-based pharmacy firms engage in commercial activities through web-based platforms, enabling the immediate distribution of drugs and health care equipment to clients using postal services.

The field of online pharmacies has seen significant market value growth in the last few years. In the United States, online pharmacies have raised about $29.35 billion in 2014 and are projected to reach US $128 billion by 2023, reflecting a yearly increase of 17.7% [2]. The main factors influencing the customer preference for online pharmacies are convenience, affordability, and availability of drugs that may not normally be accessible or may be subject to cancellation [3,4]. The spread of mobile and wireless gadgets on a worldwide scale has generated an exceptional and incomparable potential for the provision of global medical care. According to data provided by the International Telecommunication Union, the global number of digital phone connections reached over 6 billion in 2012, representing an 86% market share globally. It is noteworthy that more than 70% of these customers were located in economically deprived nations such as Pakistan [5]. However, a significant number of customers residing in developing nations have shown reluctance toward accessing pharmacies on the internet. The reluctance to use such networks might be linked to multiple factors, which include a lack of awareness of their functionality, concerns regarding their reliability and authenticity, and limited accessibility. The main catalyst for the widespread use of e-pharmacies by most clients is expediency [6]. A survey reveals that over 80% of persons living in disadvantaged areas have use of mobile telephone connections. The geographical reach of these networks encompasses approximately 90% of the globe’s population [7]. Internet pharmacies are rapidly growing in Pakistan. Organizations want to build trust and establish a favorable bond with customers. Pakistan’s internet-based pharmacy business is expected to develop at a compound annual growth rate of 16.8% between 2020 and 2025 [8]. The growth might be ascribed to a significant shift in customer preferences, as they progressively favor the use of such services. An extensive examination is necessary to investigate the current condition and incidence of online shopping for medicines in Pakistan, including established sites like Dawaai.pk, DVAGO, and Tabiyat.pk, among others. The predominantly digital nature of web-based pharmacies, except for the combination model of DVAGO, has significantly influenced client preferences [9]. The industry ensures medicine legitimacy and follows regulations [10]. The use of health care apps and electronic gadgets is crucial for both persons who are using them for self-management and clinicians who are using them to enhance health care, as they have the ability to access metrics that allow them to evaluate the effectiveness and security of these innovations [11]. Potential risks to people’s safety and well-being may arise if health-related apps and gadgets are not subjected to thorough assessments of their suitability and credibility. The lack of reliability in corroborating information inside medical apps has been emphasized in multiple studies. The increasing concern over the possible risks linked to the use of medical apps has led to an upsurge in curiosity in the surveillance of the effectiveness and dependability of these apps. Although there are no regulations in Pakistan regarding the usage of mobile health (mHealth) apps, we used international guidelines as a reference. In 2013, the Food and Drug Administration issued a handbook with guidelines for evaluating the quality of mobile health apps. Another guideline on digital health was established by the World Health Organization in order to enhance patient outcomes and provide direction for the application of initiatives related to digital health [12,13]. In the last decade, a research team led by Stoyanov developed the Mobile App Rating Scale (MARS) [14]. The MARS constitutes a well-recognized, dependable evaluation instrument designed exclusively for the assessment of the capabilities of (mHealth) apps [14,15]. The method has repeatedly shown a high level of dependability and a broad range of applications after being used for comprehensive evaluation of smartphone app content. The MARS is composed of 23 discrete elements that are organized into several areas, including participation, utility, design, accuracy of data, and subjective assessment [15-26].

The aim of our research was to evaluate digital pharmacy apps in Pakistan that were available on the Google Play Store and the iOS App Store. To achieve the aforementioned goal, an evaluation of these apps was conducted using the MARS.

The investigation was done with scrupulous attention to ethical guidelines. The investigators acquired ethical permission from the Institutional Ethics Review Board committee of Jinnah University for Women. Their authorization was granted under the reference number JUW/IERB/PHARM-ERA-004/2023. This measure was used to enhance the dependability and credibility of the study while upholding the highest ethical standards.

A pair of assessors (pharmacists) was selected to evaluate the online pharmacy apps. The inclusion criteria required both assessors to have expertise in pharmacy and training in the MARS. The exclusion criterion was to eliminate individuals who were not trained in the MARS. Both assessors searched for web-based pharmacy apps on the official marketplaces of the 2 prominent operating systems, namely Android (Play Store) and iOS (App Store). To guarantee the comprehensiveness of the findings, a widely searched phrase, namely “online pharmacy apps in Pakistan,” was used in both stores.

The study’s sample selection criteria were used to determine which apps met the specific inclusion criteria and exclude those apps that did not meet the inclusion criteria. Inclusion criteria included apps solely originating from Pakistan, accessible from the Google Play Store and iOS App Store, and which involved online pharmacies. Financial or premium app-related biases were not taken into account, and only English-language apps were considered. Exclusion criteria encompassed the exclusion of paid apps, apps in languages other than English, apps not available for download on the Google Play Store and iOS App Store, and any duplicate apps.

The complete dataset of the apps included the input of crucial information. The collection of data encompassed several attributes of the apps, such as their title, platform (Android or iOS), costs, class (about health care, wellness, and physical activity), date of the latest update, the newest language, number of reviewers, score, creator, and number of downloads.

The 2 neutral assessors assessed the apps by name and downloaded them from the iOS App Store and Google Play Store. The chosen apps demonstrated probable suitability and were entered into a database. The apps that satisfied the established criteria were kept, but those that failed to meet the requirements were eliminated. In instances when uncertainty arose over the suitability of an app, a third assessor was engaged.

The 2 assessors received instructions regarding the use of the MARS via individually viewing tutorial videos accessible on YouTube. Subsequently, the users proceeded to independently download, use, and assess the quality of apps. To gather the data, the researchers used a collection form that encompassed various details such as the app’s creator, device, version, release year of the latest release, costs, the number of downloaded files, feedback from consumers, the existence of a statement of confidentiality, privacy-related technical factors, medical device data, and the elements evaluated through the MARS.

The assessment of the usability of the incorporated apps was carried out by using the MARS, a comprehensive tool including 23 criteria that are categorized into 5 distinct sections.

The engagement component of the assessment evaluates many factors including enjoyment, interest among users, personalization, interaction (including alerts, messages, signals, and comments), and suitability for the target demographic.

The functionality component encompasses 4 key aspects related to operational efficiency, including (1) utilization, (2) directions, (3) conceptual flow, and (4) gestural design. The part on aesthetics assesses the elements of graphic layout, visual attractiveness, chromatic palette, and aesthetic coherence. The next part examines the existence of quality data such as textual content, comments, evaluations, and citations derived from trustworthy sources.

The subjective quality component consists of 4 items. This section assesses the consumer’s level of enthusiasm for the app. Every single aspect was assessed using a rating system ranging from 1 (representing inadequacy) to 5 (representing excellence). The composite app grade was determined by computing the average score across parts A, B, C, and D. The app’s performance results are a subjective measure, which was derived separately by calculating the mean value of subsection E. Furthermore, subsection F included 6 app-specific questions that evaluated the perceived influence of the app on the user’s understanding, mindset, intention to make a change, and the probability of achieving successful modification of behavior concerning the targeted wellness behavior. To improve the objectivity of the MARS in evaluating app excellence, the inclusion of the subjective aspect of the quality part was omitted from the computation of the general average app performance grade. Moreover, the robust association shown in prior research between the MARS sum score and the individual’s star rating suggests its efficacy in reflecting the overall perceived excellence [14].

Each reviewer conducted an independent evaluation of all the elements under the criteria of MARS for each of the apps. The average rating for each item within each app was computed using the numbers supplied by both evaluators. The mean scores for every category and part of the MARS were computed by taking the average values obtained from all the apps. As a result, mean scores were calculated for each category, and the same method was used to get the overall score and its corresponding standard deviation for each app.

The mean score for each dimension of the MARS was determined by calculating the mean values as recommended by the raters. In addition, the mean and standard deviation were obtained and Pearson correlation analysis was conducted for all dimensions of the MARS. The interrater reliability estimate for the MARS tool was conducted using 3 statistical indicators (Cohen κ coefficient, Krippendorff α, and intraclass correlation coefficient [ICC]). The κ coefficient was evaluated by assigning quadratic weights to different values. The ICC was calculated using a 2-way random model to assess the degree of agreement. Weighted κ, Krippendorff α, and ICC were calculated for each dimension.

A preliminary search was conducted on the Google Play Store and iOS App Store to identify 50 online pharmacy apps accessible in Pakistan, as demonstrated in Figure 1. A total of 10 apps were eliminated from the study due to their failure to meet the predetermined inclusion criteria. Additionally, 10 apps were found to be in languages other than English, so they were not included in the analysis. Furthermore, 7 apps could not be downloaded. A total of 23 apps were chosen for inclusion in the study, as shown in Figure 1. The apps have been downloaded from the Google Play Store and the iOS App Store, and they were evaluated using the MARS.

Web-based pharmacy apps in Pakistan offer a wide range of health care and pharmaceutical products and services. The tables in this section provide detailed information on technical features, creator details, customer reviews, pricing, version specifications, affiliations, and other relevant details of apps. These tables can aid users in selecting suitable apps and provide valuable insights into Pakistani health care and pharmaceutical competition. Table 1 presents a complete overview of the technological attributes associated with various pharmaceutical apps. Every app has been assigned a distinct serial number that uniquely identifies it and this table provides a comprehensive overview of the apps’ special technological capabilities. These qualities encompass the ability to distribute material across multiple platforms and offer secure login capabilities, among other functionalities. The table is presented as a valuable resource for both customers and developers, facilitating their comprehension of the various functionalities and fundamental potentials of these apps. Table 2 presents a summary of the main domains of focus (ie, strategies for utilization and the affiliations with which they are connected). The analysis suggests that the major objective of these kinds of apps is to enhance the welfare of individuals, while concurrently implementing a business-oriented approach that involves severing connections between companies. This information facilitates users’ understanding of the underlying goals and target market of these apps. Table 3 provides a comprehensive analysis of the manufacturers of pharmacy apps, encompassing essential details such as the number of evaluations, feedback, the date of the most recent update, and the pricing model of the app (whether it is free or paid). The table presented herein can be used by users to evaluate the reliability of these programs by taking into account both the reputation of the creators and the feedback offered by consumers. Table 4 presents a comprehensive examination of several versions of pharmaceutical software including their download statistics, release dates, and the operating systems on which they may be accessed. The provided data hold significant value for users who are in search of compatibility with their devices and the most up-to-date versions of apps. Table 5 presents concise and informative descriptions of pharmacy apps, furnishing users with a concise summary of the features and distinctive value proposition of each app. This feature is especially advantageous for individuals who are in search of a suitable app that meets their health care and pharmaceutical needs.

The mean scores of the 23 apps ranged from 2.64 to 4.00. The mean scores of each dimension of the MARS are demonstrated in Figure 2. The individual scores of each app are presented in Figure 3.

The ICC coefficient was computed for the 5 aspects of the MARS (Table 6). The ICC coefficient for the variable “Engagement” was determined to be 0.764, with a 95% CI ranging from 0.521 to 0.893. This finding indicates a considerable degree of consensus among raters about the concept of participation. Similarly, the construct labeled “Functionality” demonstrated a notable ICC coefficient of 0.824 (95% CI 0.630-0.922), which signifies a robust consensus among raters on their evaluations of the construct’s functionality. The variable “Aesthetics” demonstrated the highest ICC coefficient, measuring 0.949 (95% CI 0.885-0.978), indicating a substantial level of consensus about aesthetic judgments. The dimension of “Information” demonstrated a moderate ICC cofficient of 0.702 (95% CI 0.601-0.875), and “Subjective quality” exhibited strong agreement, as indicated by an ICC coefficient of 0.871 (95% CI 0.720-0.943). The Krippendorff α coefficient was computed to evaluate the reliability of the MARS over the identical set of 5 dimensions. The Krippendorff α coefficients for the dimensions of “Engagement,” “Functionality,” “Aesthetics,” “Information,” and “Subjective quality” were 0.759 (95% CI 0.661-0.857), 0.825 (95% CI 0.727-0.923), 0.979 (95% CI 0.881-1.077), 0.705 (95% CI 0.689-0.885), and 0.755 (95% CI 0.657-0.853), respectively. The Krippendorff α values, accompanied by their corresponding CIs, serve to affirm the reliability and coherence of raters’ interpretations across the assessed factors, so enhancing the overall comprehension of the data’s integrity. Furthermore, the investigators used weighted κ statistics to evaluate the characteristics of the MARS. The weighted κ value for the parameter “Engagement” was calculated to be 0.764 (95% CI 0.350-0.722), indicating a reasonable degree of consistency that has been adjusted for chance. The assessment of “Functionality” resulted in a Cohen κ score of 0.658 (95% CI 0.510-0.805), showing a statistically significant degree of consensus among evaluators about the meaning of functionality. The variable “Aesthetics” demonstrated a statistically significant κ value of 0.907 (95% CI 0.819-0.994), showing a strong agreement in the evaluation of aesthetic attributes. The κ statistic for the value of the parameter “Information” was calculated to be 0.388 (95% CI 0.151-0.626). The study’s findings indicate a rather low degree of consensus between assessors about the dimension of Information. The subjective quality assessment resulted in a κ score of 0.670 (95% CI 0.488-0.851), showing a statistically significant level of agreement. The reliability and internal consistency of every element of the MARS questionnaire were assessed using Cronbach α. The rating of “Engagement” had a Cronbach α value of 0.867, indicating a high degree of internal consistency. The concept of “Functionality” demonstrated a significant degree of internal consistency, as indicated via a Cronbach α value of 0.904. The concept of “Aesthetics” had a notably high Cronbach α rating of 0.905, suggesting a commendable level of internal consistency. The Cronbach α value of 0.821 indicates that the “Information” parameter has a high level of internal coherence. The construct of “Subjective quality” demonstrated a Cronbach α coefficient of 0.911, indicating a substantial degree of internal consistency.

Table 7 presents a detailed overview of the relationship values, means, and standard deviations of various elements of the MARS. The dimension denoted as “Engagement” exhibited a notable favorable correlation of 0.949. The average score for this aspect was 3.324, accompanied by a rather small standard deviation of 0.560. The results of this study indicate a substantial degree of concurrence and uniformity among participants in their assessments of engagement. The parameter “Functionality” demonstrated a significant positive relationship of 0.923. The data exhibited a marginally elevated mean rating of 3.520, accompanied by an acceptable standard deviation of 0.491. These findings indicate consistent perceptions of the functionality of the subject matter. The aspect referred to as “Aesthetic” demonstrated a considerable positive association of 0.798. Additionally, it had a mean value of 3.626 and a standard deviation of 0.504. The aforementioned figures indicate a substantial agreement among participants on their assessments of beauty. The aspect labeled as “Information” exhibited a noteworthy positive relationship of 0.763. The average grade for this aspect was 3.044, suggesting a consistent perspective. Moreover, the observed standard deviation of 0.359 indicates a rather small range of variation in individuals’ judgments regarding the information aspect. The study’s findings provide valuable insights into the prevailing viewpoints of the participants and the level of agreement seen across multiple aspects. These observations enhance our understanding of the information quality derived from the MARS.

Online pharmacies not only offer medication selling, purchasing, and delivery to patients but they also provide medication details, which include brand, dosage form, indication, side effects, warning, and price. Thus, the critical first step in the treatment process is to evaluate the standards of online pharmacies. This investigation represents a significant achievement as it constitutes the inaugural evaluation of the quality of online pharmacy apps in Pakistan through the utilization of a mobile app grading scale developed by Stoyanov et al [14]. In this comprehensive study, a meticulous analysis was undertaken of the pharmacy apps available in Pakistan; as Amor-García et al [12] did with genitourinary tumor apps, we chose and downloaded our apps using both the Google Play Store and the iOS App Store. The sample size of 23 apps was comparable to the sample size in research carried out by Kim et al [26]. These apps met the established criteria. As previously reported in other research, we discovered that the Google Play Store had a wider variety of apps than the iOS App Store. Nine apps were downloaded from the iOS App Store, while 14 came from the Google Play Store [21,27].

The MARS is an innovative and effective tool that is used for evaluating mobile health apps. The mean score of all apps ranged from 2.64 to 3.92, which was similar to scores reported by previous studies [17,28-30]. The subjective quality component of the MARS was omitted from the collective mean score [24,31]. Furthermore, we determined the average scores for several dimensions of “Engagement” (mean 3.32, SD 0.560), “Functionality” (mean 3.52, SD 0.49), “Aesthetics” (mean 3.62, SD 0.50), and “Information quality” (mean 3.04, SD 0.35). Several methodologies have been developed to evaluate the efficacy of mHealth apps. Nevertheless, the absence of a universally accepted method and the limited participation of scientific professionals have made it impractical to regularly put them into practice [32]. The MARS approach, as documented by Stoyanov et al [14], is regarded as a user-friendly, straightforward, and coherent instrument that is highly regarded for its reliability due to its endorsement by skilled technicians and health care experts. This scale enables a comprehensive analysis that encompasses several disciplines and is intended for the assessment of health care apps. It consists of 5 domains that encompass the key factors necessary for a thorough evaluation. In their study, Stoyanov et al [14] found that the MARS assessment had excellent internal consistency (Cronbach α=0.9) and interrater reliability (2-way mixed ICC coefficient=0.79, 95% CI 0.75‐0.83) when used to assess 50 mental health apps.

Our findings were similar to both the original MARS and other translated versions, according to reliability and validity studies [33-36]. As per globally recognized quality standards, all sections’ internal reliability and Cronbach α as well as the overall scores were found to be good [37]. Subsequent translation studies and the original MARS study also noted this high level of internal consistency. In contrast, our research revealed a good degree of interrater reliability, with an ICC ranging from 0.702 to 0.913. To ensure construct validity, we opted for a multitrait scale evaluation with the item-subscale association over a factor analysis, as this approach has proven effective in prior research [26]. Our findings met the predetermined success level and demonstrated good parallel and divergent reliability, except for the “Information” subscale because many parameters were not measurable and less information on content related to medications. Nonetheless, we discovered a low ICC for the “Information” subscale. Prior research has similarly documented the diminished outcomes associated with the information variable [17].

The mobile apps provided did not consistently cite the authors or websites that provided information, and there was no assurance of the accuracy and timeliness of the material.

The diminished quality of data has been linked to a variety of dangers for mobile app users, primarily due to the potential for misinformation leading to inaccurate self-diagnoses and unfavorable health-related decisions in prevention, wellness, and therapy [38,39]. The majority of apps were simple to download and log-in friendly. Of the 23 apps, we only encountered login issues with 2 or 3. In contrast to other aspects of the MARS, the information dimension was reduced because, of the 23 apps, some apps provided information about the specifics of medications, such as dosage forms, indications, side effects, precautions, categories, and alternative brands, and many apps only presented brand names with the cost and category of drugs. The medication information is useful for patients to have in addition to the drugs they choose or for other purposes relevant to medicines or managing their treatment.

As a result of transportation and public gathering limitations brought on by the COVID-19 epidemic that struck the nation in February 2020, the selling of medicines online expanded significantly in the years that followed [40]. Many people started to order medications from online pharmacies even if the quality of the products was still questionable according to medical professionals. Pharmacists argued that online pharmacies put their businesses at risk and allow the free selling of potentially harmful drugs [41-46]. The actual number of online pharmacies in Pakistan is unknown, but according to authorities, most of them are located in the country’s 2 biggest cities, Karachi and Lahore. They claim that, in specific locations, clients will get prescription drugs and other medical supplies within 4 hours of placing their order, or, in other cities, within 48 hours via a shipping service. In addition, many online pharmacies provide free shipping to client residences and discounts of as much as 20%. Online pharmacies help people get medicines in just a few clicks. Online pharmacies are not only a source of medicine delivery. Online pharmacy apps provide other services including prescription refills, information about the availability of drugs, and information about drugs. Pharmacies are the main pillar for this treatment method [47,48]. However, there have been no studies performed on this topic. Our study provides information about these pharmacy services, including engagement, functions, layout, and basic information.

The scope of our research was mostly limited to a single geographic region, namely Pakistan, and specifically focused on apps accessible in the English language. Additionally, the limitations of our study stem from the narrow focus on free apps. We did not conduct a thorough literature analysis and our technique was primarily focused on app stores. As a result, it might be possible that we did not fully explore the entire collection of available data. The evaluation was done by just 2 assessors. There were more assessors in previous studies that used the MARS.

To bolster the reliability of online pharmacy apps in Pakistan, it is advisable to enforce more stringent laws and guidelines, regularly assess the content of these apps through audits, fortify data security measures, and provide standardized and evidence-based medical information. Subsequent investigations may focus on the exploration of client observations and opinions about these apps, with a particular emphasis on the dynamics of user-platform interaction and the establishment of trustworthiness.

The MARS is a trustworthy and adaptable tool for assessing the reliability of mobile apps. The evaluation of online pharmacy apps of Pakistan using the MARS provided helpful data on the reliability and efficacy of these apps. The results showed the trustworthiness and dependability of web-based pharmacy apps. Many of the evaluated online pharmacy apps exhibited high reliability and credibility in all dimensions of the MARS.