Patient choice has come to prominence in the United Kingdom with the advent of the National Health Service (NHS) Choose and Book and representation in key health policies such as Choice Matters [1] and High Quality Care for All: NHS Next Stage Review [2]. Patients in many countries (including the United Kingdom, Netherlands, and the United States) can now vote with their feet and choose health providers that fit best with their preferences and needs [3]. Providing choice is thought to be important in promoting competition between providers, with the goal of improving both the quality and efficiency of care [4]. While the actual evidence supporting the contribution of patient choice to cost control and quality of care is mixed [5], improving patient choice and shared decision making remains a stated objective of different health systems worldwide.

Service users trying to select between different health providers can use information from a variety of sources. Increasingly, patients are using websites that provide information about the comparative quality of health care from different providers [6]. For example, patients can compare hospitals using a wide range of quality and performance indicators such as waiting times, staffing, and patient safety on the website of the Care Quality Commission, the independent regulator of health care in England [7]. Service users are also turning to information based on the experiences of other patients when making a choice of provider. While patients have long used the experiences of friends and family in choosing hospitals, service users can now more systematically access collected information on patient experience (eg, Consumer Assessment of Health care Providers and Systems reports in the United States [8] and NHS Choices in the United Kingdom [9]). There is evidence that information based on patient experience is considered at least as important by service users choosing between different providers as different performance indicators provided by hospitals or reporting bodies [10].

An interesting recent development is the advent of patient rating websites such as PatientOpinion [11] and IWantGreatCare [12] (United Kingdom) and Rate MDs [13] (United States and Canada), where patients express views about the care they have received in much the same way as they might rate a hotel on a travel website. Users rating and commenting on the health care they receive is only set to increase with growing access to the Internet (particularly through increasingly ubiquitous mobile phone and tablet devices) [14]. Our research team has previously shown that online discretionary patient ratings can be useful in providing reliable information about health care quality [15,16].

This paper describes the process that was undertaken in the development of a hospital ratings platform for a consumer health care app. The aim to incorporate the best available evidence lies in sharp contrast to the majority of health related apps [17]. By working with patients and members of the public, we also sought to meet end-user needs often overlooked [18]. To develop the hospital ratings service, research was undertaken to determine which factors were considered important to individuals when choosing a nonemergency health care provider. This paper describes how established research methods can be used in the development of new mHealth apps.

The aim of this study was to generate a list of quality indicators from the general public that were deemed important when choosing nonemergency secondary health care services along with the rationales for these choices, with the intention of using the findings in a new mHealth hospital ratings platform. Further, we aimed to illustrate the importance of rigorous research methodologies to underpin the development of mHealth technologies. The study was considered as part of a service evaluation, and ethics approval was not required. The study was conducted in London between November 2011 and June 2013.

As the first part of a mixed methods approach to identify which factors were considered most important to people when choosing nonemergency secondary health care providers in the United Kingdom, a review of existing literature (both academic and grey literature) was conducted. Publications were included only if they described patient choice in the United Kingdom health system so as to avoid confounding factors in the context of other health systems. For example, although there are many relevant articles from the United States, the differences between the largely privately funded United States and publicly funded United Kingdom systems may influence what users consider important when choosing a hospital. Patient choice did not feature prominently in national health policy until more recently [1,2]. A pilot literature review revealed a dearth of high quality research formally investigating patient choice in secondary health care prior to 2005. The patient choice agenda was first investigated when Lewis began studying patients’ attitudes towards choice of hospital in the context of waiting times for cardiothoracic surgery [19]. Studies were therefore excluded if they had been carried out prior to 2005.

The aim of the questionnaire was to formulate a ranked list of quality indicators. The survey was completed by participants with a member of the research team at hand to explain any terms or answer questions. Care was taken to ensure that facilitators did not directly ask questions to avoid leading or influencing participants choices. A power calculation [20] for a study comparing the attitudes regarding choosing secondary health care between two groups (general public and out-patients attending clinic appointments at the hospital) determined a target total sample size of 400 (population of London, 2001: 7,172,091 [21]; standard of error 0.05), thus two groups of approximately 200 participants. Members of the general public were recruited (n=201), and the data are presented here. Data collected from the second group (patients at the hospital) are beyond the scope of this manuscript. An initial pilot questionnaire was conducted on 20 individuals prior to its wider use to identify and correct any unforeseen problems. Quota sampling was used to estimate size of target groups to ensure accurate representation of ages and genders [22]. Four age categories were formed by combining existing categories from 2001 United Kingdom national census data [21]; target proportions for each age category were based on urban population proportions from the same census. Due to time and resource constraints, convenience sampling was then used to collect data, with checks to ensure collected data approximated the census age proportions. A higher proportion of people aged 18 to 35 years were included compared to people 60 years and older to account for lower usage of mHealth apps in the latter age group as described by previous investigators [23].

Inclusion criteria were English-speaking adults (18 years and older) UK residents. In order to ensure data collection was feasible under time and resource constraints, convenience sampling was then employed to recruit participants in six separate locations in central and greater London to provide greater geographical spread and wider generalizability of the data. Questionnaire collection ceased once the number of participants in each demographic group approached the estimated targets.

Informed consent was obtained for each participant completing the questionnaire. Participants were asked to provide demographic information and rank a predetermined list of quality indicators in order of importance. An ordinal scale was used, where respondents were asked to rank the factors in order of importance, first within their categories and then the categories themselves. This allowed us to assess the relative importance of the factors, as opposed to absolute importance [24]. Data were collated using Excel (Microsoft Corporation); SPSS (IBM Corporation) was used to undertake statistical analysis. Differences in mean ranks within categories were determined using the Friedman test, with P<.05 considered to be statistically significant.

Focus groups were used to discuss the rationales behind the quality indicators considered to be important. Convenience sampling was used to recruit participants due to time and cost constraints, and referrals from initial recruits were used for further recruiting. We sought to recruit an equal representation of genders and ages in order to increase the generalizability of the results. Due to resource and time constraints it was not possible to match the age stratification of focus group participants with questionnaire respondents. The median age of participants at pilot focus groups was 40 years; therefore, participants were stratified by age and gender using this as a marker of division (males under 40 years, females under 40 years, males over 40 years, females over 40 years) to enable timely data collection and efficient analyses. Four focus groups were conducted, each comprising 6 individuals, with a gradual shift from broad open questions to narrow, focused questions [25]. Written consent was gained from each participant in advance. A final script of questions for the focus groups was confirmed following a restructuring of a preliminary script that had been piloted. Each focus group lasted for approximately 90 minutes and was led by one researcher acting as an impartial facilitator and one as an assistant moderator. All recordings were transcribed verbatim, and thematic analysis [26] was used to identify common themes.

Service users across many health systems are now offered a wider choice of health care providers and increasingly have at their disposal a wide variety of factors to consider when making these decisions. The rapid adoption of mobile phones and tablet devices has enhanced access to information about different hospitals by making it possible for patients to view and share this information at any time and while on the move [31].

This study collates the existing literature regarding which factors are considered important for consumers in this context, contributes a categorized and ranked list of quality indicators, and reconciles the rationales underpinning these decisions. Furthermore, this study demonstrates how this information can be harnessed in the context of developing a robust user-generated ratings platform for use on mobile communication technologies.

Although mobile technologies are frequently put forward as a solution to challenges in health informatics, there is often a lack of rigorous research underpinning their development and evaluation. This project illustrates the importance of sound research methodology when developing these strategies by employing a mixed methods approach to reconfigure the ratings service based on factors that the public held to be important in choosing nonemergency health providers.

Findings included that staff competence was the most important factor within the hospital staff category, with participants asserting that they would travel further than their nearest hospital to secure treatment under a doctor they perceived to be more competent. However, the qualitative analyses revealed that there is no single measure by which competence could be judged; rather it was a compound of many factors including amount of experience, qualifications, place of education, and interpersonal skills.

Cleanliness and modernity of equipment stood out as the two most important hospital facilities. This is concurrent with previous reports that people consider information about cleanliness when researching a hospital [32,33]. Qualitative analyses discovered that while this was a factor deemed to be highly important, it was not widely sought after. Participants suggested this might be due to a commonly held assumption that hospitals are of equal cleanliness, therefore only those hospitals with a remarkably poor reputation for cleanliness may be of note. This was also the case for modernity of equipment. Future mHealth developers should reflect on this subtlety in order to include factors that are not only important but also highly sought after to avoid information overload for users.

Participants could not differentiate level of importance between various types of hospital statistics. Hospital-wide statistics may be of limited use to users who would be more interested in department-specific statistics. Moreover, users appreciate that overall hospital statistics may not be an accurate representation of an their department of interest due to interdepartmental variation. Conversely it may be argued that an inability to compare the importance of statistical descriptors may reflect that they are poorly understood by users. This may explain equal significance attributed to individual statistics within this category and highlights the need for the careful inclusion of statistics that are relevant to the user’s individual health encounter in mHealth platforms (See Figure 3). Care must be taken by developers to ensure that presentation of statistics, including color coding or a glossary of terms, aids user interpretation. These subtleties may not be appreciated without formal research methods informing these strategies.

The fact that the categories of hospital staff, hospital facilities and hospital statistics were deemed equally important illustrates that users’ demands for information about hospitals are extensive and varied. mHealth developers should aim to provide information about these categories equally in order to reflect and satisfy these demands. Adequate provision of these varied factors requires an equally varied presentation of information. For example, participants asserted that graphs and percentages provided objective evidence of statistical measures, whereas past users’ reviews were more useful in capturing complex domains such as staff competence. Therefore we recommend that mHealth developers include a range of formats as this study illustrates that each caters to different, and equally important, categories of quality indicators.

An important limitation of the study is that the questions were asked outside the context of mobile phones and mHealth. This was a purposeful decision as it was felt that doing so may lead to reduced applicability of this research. Further research is required specifically investigating whether the information consumers want in the context of an mHealth app is any different from the factors that are important when choosing secondary health care in general. The original study was adequately powered for a comparison between patient groups and the general public; the analyses included here may therefore by underpowered due to resource constraints. This study was unable to match age stratification between quantitative and qualitative stages. We recommend that future investigators attempt to do so to allow closer mapping of the two datasets.

The huge interest in developing apps for mobile phone and tablet platforms to enhance health outcomes and service delivery—widely termed mHealth—has led to an “enthusiastic proliferation of untested methods” [17]. An evidence base needs to be developed to make this field credible and address the needs of the end-user. More attention needs to be paid to structuring app development in theory or best practice [34].

This study used a mixed methods approach to find that information about hospital staff, hospital facilities and hospital statistics are equally important to people when choosing a hospital. Information about getting to the hospital is least important. Staff competence is most important regarding hospital staff, which is a multifactorial domain best captured by past users’ reviews; cleanliness and modernity of equipment are most important regarding hospital facilities but are not actively sought after. People find it difficult to compare relative importance between various hospital statistics. Barriers to understanding statistics may be removed by use of graphs and percentages.

Users of health care demand a wide and varied range of information about hospitals. mHealth developers must determine which information is most relevant to their users’ needs and provide this in an accessible format. Less important information must be identified and removed to avoid information overload. A sophisticated appreciation of the complex needs of mHealth users is possible when these strategies are underpinned by rigorous research methods. This study demonstrates how a mixed methods approach can enhance mHealth solutions.