According to the 2018-2019 China Mobile App Store Market Annual Monitoring Report from iiMedia Research, 360 Mobile Assistant was the most popular Android app market in China [17]. Hence, we selected 360 Mobile Assistant as the typical Chinese Android app market and the App Store as the typical iOS app market to retrieve virtual hospital apps.

On November 18, 2019, we crawled mobile apps in 360 Mobile Assistant (Android) and App Store (iOS). We used the following 7 Chinese keywords to conduct a preliminary search in app markets: see a doctor, medication guide, consultation, hospital registration, doctor, medication, medical treatment. We selected the apps that appeared in both Android markets and iOS markets for in-depth analysis. We found that many apps were short-lived, similar to mental health apps [18,19]. More than half of the studied apps were revised roughly every four months [18]. Due to these observations, we excluded the new apps that were first released after July 2019. For these apps, the information was insufficient and unstable for objective analysis. Three investigators screened the names and descriptions of these apps and then tested functionalities of the apps to distinguish between virtual hospital–related apps and unrelated apps. We excluded the apps that more than one investigator labeled as not relevant to the theme. In this way, we identified the included virtual hospital apps. We downloaded and crawled the qualified apps with the researchers’ related information.

All of the apps were sorted by number of downloads. For each included app, extracted data consisted of (1) flat user score, reflecting perceived satisfaction of a user; (2) the number of downloads, showing the measure of usage frequency and intention; (3) the number of positive, negative, and neutral ratings; (4) content of reviews; and (5) the last updated date of an app. Critically, due to the fact that Apple’s App Store requires a minimum number of reviews before releasing average ratings (ie, small numbers are suppressed), we think that the score on the App Store is more referential. Thus, we used average ratings in App Store. While App Store does not provide the number of downloads, we used the number of downloads from 360 Mobile Assistant for analysis. We divided apps into three categories according to the number of downloads: (1) frequently used apps, (2) occasionally used apps, and (3) seldom-used apps. First, apps with fewer than the average number of downloads of all determined apps were defined as seldom-used apps. Second, apps with fewer than the average number of downloads of the remaining apps were defined as occasionally used apps. Other apps were defined as frequently used apps. Procedures are shown in Figure 1.

According to the proposal put forward by Mendiola et al [19] and review data, we sum up the following four aspects that concern users: (1) app functionalities, including online doctor consultation, in-app purchases, tailored education, and community forums; (2) security and privacy, including user data management and user privacy; (3) health management, including health tracking, reminders, and notifications; and (4) technical aspects, including user interface and equipment connection.

For qualified apps, we selected and reviewed user-generated reviews after the last update of the 7 most popular apps. We found 4630 reviews. Before aspect-based content analysis, we filtered poor-quality or fake reviews from our data set based on the following rules: (1) review expresses only positive or negative emotions and without mentioning any specific aspects, such as “good!” or ”very bad!“ (n=1902); and (2) review is an apparent case of human manipulation, such as advertising reviews that have nothing to do with the corresponding app (n=42).

After screening, we got a clean review data set (n=2686), where each review was authentic and specific. On the clean review data set, we asked 3 investigators to use a binary system to assign ”1“ to a particular aspect in a review in which the user was satisfied or ”0“ to indicate that the user was dissatisfied with a particular aspect. Investigators independently conducted quality assessments and the final results were decided by majority vote. Examples are shown in Textbox 1 and Figure 2. Note that most of the reviews only included one or two aspects (2570 reviews with one aspect each, 116 reviews with two aspects each).

Table 1 summarizes the characteristics of included virtual hospital apps, including the statistical information of downloads and rating scores. According to the search strategy and data filtering, we took a total of 239 apps for analysis. The average rating score was 3.8 out of 5 and the average number of downloads was 36,464. We included a total of 92,366 rating scores, including 90,106 positive scores, 1512 negative scores, and 748 neutral scores. Note that people can give rating scores without reviews for apps. Thus, there was not a one-to-one correspondence between content of reviews and the numbers of ratings. In addition, apps had an average of 386 scores, including 377 positive scores, 3 neutral scores, and 6 negative scores per app.

The results in Table 1 show that there was no linear relationship between the number of rating scores and the number of downloads. Frequently used apps only accounted for 2.9% (7/239) of the number of virtual hospital apps but accounted for 63.06% (5,496,000/8,715,000) of all downloads. There was not a big difference in average rating scores between the frequently used apps and the occasionally used apps, but the seldom-used apps scored notably lower than the other two. For frequently used apps, negative rating scores accounted for the highest proportion of the total comparison theory, followed by seldom-used apps, and lastly occasionally used apps. People set higher requirements for popular virtual hospital apps (ie, frequently used apps).

We ranked apps according to their average rating scores and the number of downloads in the frequently used apps, occasionally used apps, and seldom-used apps categories. We took the top 3 and the last 3 apps in each category for in-depth analysis.

The results are shown in Table 2. For frequently used apps, the difference between average rating scores was not large, but the number of downloads and the number of positive, neutral, and negative ratings varied greatly. Although Dr Chunyu had the most downloads (17.98 million), it also had the most neutral rating scores and received a low average ranking score in this category. The majority of apps with more downloads had more positive reviews, neutral reviews, and negative reviews. There are also good virtual hospital apps in the occasionally used and seldom-used app categories, for example Good Doctor for People and Dr Ruyi, with average rating scores of 4.9. For seldom-used apps, apps with higher average rating scores had more downloads than apps with lower scores, except Dr Thumb. We observed that some apps in the seldom-used apps category specialized in types of virtual hospitals, such as Dr Doodle for pregnant women and newborns and Dr Warmth for video consultation.

We analyzed authentic and specific reviews, defined in the section “Aspect-Based Content Labeling of Valuable User Reviews.” The 4 themes were (1) app functionalities, (2) security and privacy, (3) health management, and (4) technical aspects. The themes and associated aspects are summarized in Table 3.

From Table 3, we observe statistical information regarding user concerns about virtual hospital apps. First, we compared the total number of reviews for each aspect. The majority of reviews expressed opinions about online doctor consultation, health tracking, and user data management for virtual hospital apps in China, while fewer reviews discussed community forums, user privacy, and tailored education. Second, we determined the ratio of positive and negative evaluations. Many reviews praised online health services (948/600, 91.3% for online doctor consultation; 148/152, 97.4% for reminders and notifications). About 80% (32/40) of reviews showed positive opinions about community forums within virtual hospital apps, indicating that users like understanding, sharing, and discussing health-related issues in apps. However, it is imperative to improve security and privacy in virtual hospital apps, especially for user data management, which 86.0% (394/458) of reviews complained about. Details of typical user opinions about each aspect are described below.

This study explored user preferences and requirements for virtual hospital apps to predict and analyze current models and future directions of the medical service market with mobile internet. First, we provided a statistical description of full data to compare distributions of rating scores and number of downloads with 3 different categories. Second, we conducted in-depth aspect-based content analysis to discover opinion details about interest in and barriers to participation. In this study, the app and review selection strategy benefited research, with better generalization and time-efficiency. Despite many works that have discussed virtual hospitals from the perspective of economics [21] or doctors [9], this paper is the first to provide an in-depth report and highlight situations from the users’ perspective.

End user engagement and follow-up are prerequisites for sustained promotion and effectiveness. In the field of virtual hospitals, frequently used apps with the most downloads received the highest average rating scores (4.8/5), while there was great variation in the average rating scores among seldom-used apps. Interestingly, the ratios between positive and negative reviews were similar in frequently used, occasionally used, and seldom-used apps, although the Matthew effect was clearly seen in the number of downloads. We noticed that frequently used apps suffered from more negative reviews (1162/61,426, 1.89%) with detailed user experience. Users tending to give high rating scores for those popular apps had higher expectations for conversational features, personalization, performance, and high-quality content and preferred putting forward the related proposals. Frequently used apps should take responsibility for possible updates and improvements.

We noticed that high-probability access to individualized expert consultation was the main reason for both new and old user engagement. First, health care service apps can greatly relieve unequally distributed medical resources in China. Medical resources are geographically unbalanced, especially for rural areas and for persons with a disability [2]. For virtual hospitals in apps, users can access a virtual hospital regardless of time or location. All that is needed is an internet-enabled phone. In positive reviews, users said that they liked to try virtual hospital apps because they were a considerable distance away from good medical resources. When patients reported that they were cured by a doctor's treatment in an app, they were more likely to pay for in-app purchases for further health services [14]. Users felt comfortable with famous, professional doctors who also worked for offline hospitals rather than those who were from unknown or ambiguous sources. Our findings suggest that virtual hospital apps should rely on actual medical conditions and qualified physicians and medicines. With the support of government policies [4,5], users in China were willing to trust that apps could invite high-quality doctors to trade their leisure time for online consultation. Second, most users were satisfied with the prompt response, consulting attitude, and advanced knowledge of doctors in apps. Users saved waiting time, reduced transportation costs, and shared fair doctor navigation. Third, a multichannel communication strategy for points of patient engagement was welcomed. Besides video chatting, which is similar to an offline service, new consultation options in apps, such as reservation calls, voice messages, graphic messages, video messages, or plain text, could relieve patients’ tension and give patients thinking time.

Compared with online doctor consultations, users were wary of health products in in-app purchases, such as medicines, supplements, probiotic food, and home health care devices. About 47.5% (170/358) of users raised questions because distribution channels for medicine and medical devices lacked trusted verification. It was hard for customers to protect their rights with in-app purchases. Stakeholders need to pay more attention to consumers’ rights protection in the selling of health-related products.

Users cared about their health-related data. On the one hand, users sought convenient and flexible data usage, including data association, data synchronization, data transmission, and data sharing across multiple apps and external devices. Specifically, inferred from patients’ comments, we found that some users wanted to switch apps and share their data with other communities or doctors. However, that is a big challenge now [22]. An agreement is needed to put forward sharing and canonical management of health data in China. On the other hand, users asked for precision and integrity for data management. Accurate and complete health consultation records are often a great help to detect or review changes in physical and psychological indicators. Users wanted to learn from their health data history.

With regard to interactivity and visual settings, users preferred broad individualization. Personalization can enhance convenience and user-friendliness. For tailored education, relevant and meaningful individualized health knowledge provides engagement, serendipity, and high value to users, while irrelevant or unindividualized health care information displays make users feel unengaged. This type of individualization has been shown to be vital for sustained engagement [23]. For user interface, about 66.4% (190/286) of comments showed positive attitudes to the current design. The others, including foreign app users, elderly persons, and visually impaired users, asked for alternative customization view modes for promoting large-scale, easy usage. The design of indicative icons and feedback functions required more consideration. For reminders and notifications, most users found such functions helpful for consultation appointments, abnormal account access, or medicine plans. Notifications allow users to tackle health problems and to schedule time for maintaining health [24,25]. Personalized reminders can better reinforce behavioral changes in app users.

Community forums based on peer support, education for health practice, health promotion, and emotional support can serve as an effective supplement to professional doctor consultation. It can relieve China’s insufficient medical resources. A total of 1.48% (40/2686) of reviews mentioned that social health education is a powerful force in human health. People share their health experiences, knowledge, and practical support, and others learn from that. Such client-to-client communication and question answering could meet the needs of finding people who have similar health issues. In reviews, users provided tips to promote peer support, for example, placing forum joining in a prominent position during patients’ in-app journeys.

Equipment connection and user privacy remain a considerable challenge in virtual hospital apps. Satisfaction of these two aspects was relatively low, with a satisfaction rate of only 4.8% (10/208) and 3.0% (2/66), respectively. With fast growth of wearable devices [20], inexpensive health-related wearable devices provide everyone the possibility of constantly tracking personal health data. However, due to inaccurate data estimation and poor connection of devices, the user experience was negative. Inaccurate data estimation limits data usage, such as using data as an accurate and valid reference for doctor consultation. Data show that user privacy protection was not paid enough attention. Almost all comments about this issue were negative. Apps lacked explicit features to protect user accounts, especially for abnormal log-in attempts. Users suggested that email or a message alert could be an easy way to improve security. App makers need to consider how to balance user security and privacy with flexible data access.

Virtual hospital apps are convenient and suitable for two types of users: (1) users who are far away from good medical resources in time or space, especially for rural users and users with disabilities, and (2) users who require medical support and group support but are uncomfortable with face-to-face communication. However, users who demand high security and privacy need to practice caution during app selection and usage.

A practical finding from our analysis is that the Chinese market has begun to pay attention to privacy, security, personalization, and socialization in virtual hospital apps. We estimate that these will provide better solutions in future development.

This research has some limitations. First, we explored only 2 app markets, although they are the most representative ones for Android and iOS operating systems in China. Second, we were not sure about the recall ratio for the whole mobile virtual hospital market because there were other popular ways to access virtual hospital services with mobile internet, such as temporal mobile online meetings, Wechat applets, and online chat groups. Except for platform operators, data of these tools were incomplete and inaccessible in most situations. Thus, we mainly focused on public app markets. Third, we limited content analysis on frequently used apps, as we thought that other data may suffer from severe bias due to the small number of helpful reviews and downloads. However, it is known that this can be a risk for sampling strategies. Finally, this research did not include opinions or usage experiences of medical workers, such as doctors and nurses.

Virtual hospital apps are growing in popularity with the high pace of mobile internet in China. User evaluation provides a valuable insight for app development and online medical service reform. This study provides mobile internet–based statistical data for medical service providers, domain policy makers, and potential related stakeholders, and it provides aspect-based content analysis to support a better understanding of virtual hospital app users.