Identifying Quality Indicators Used by Patients to Choose Secondary Health Care Providers: A Mixed Methods Approach

Introduction

Background

Patient choice has come to prominence in the United Kingdom with the advent of the National Health Service (NHS) Choose and Book and representation in key health policies such as Choice Matters [1] and High Quality Care for All: NHS Next Stage Review [2]. Patients in many countries (including the United Kingdom, Netherlands, and the United States) can now vote with their feet and choose health providers that fit best with their preferences and needs [3]. Providing choice is thought to be important in promoting competition between providers, with the goal of improving both the quality and efficiency of care [4]. While the actual evidence supporting the contribution of patient choice to cost control and quality of care is mixed [5], improving patient choice and shared decision making remains a stated objective of different health systems worldwide.

Service users trying to select between different health providers can use information from a variety of sources. Increasingly, patients are using websites that provide information about the comparative quality of health care from different providers [6]. For example, patients can compare hospitals using a wide range of quality and performance indicators such as waiting times, staffing, and patient safety on the website of the Care Quality Commission, the independent regulator of health care in England [7]. Service users are also turning to information based on the experiences of other patients when making a choice of provider. While patients have long used the experiences of friends and family in choosing hospitals, service users can now more systematically access collected information on patient experience (eg, Consumer Assessment of Health care Providers and Systems reports in the United States [8] and NHS Choices in the United Kingdom [9]). There is evidence that information based on patient experience is considered at least as important by service users choosing between different providers as different performance indicators provided by hospitals or reporting bodies [10].

An interesting recent development is the advent of patient rating websites such as PatientOpinion [11] and IWantGreatCare [12] (United Kingdom) and Rate MDs [13] (United States and Canada), where patients express views about the care they have received in much the same way as they might rate a hotel on a travel website. Users rating and commenting on the health care they receive is only set to increase with growing access to the Internet (particularly through increasingly ubiquitous mobile phone and tablet devices) [14]. Our research team has previously shown that online discretionary patient ratings can be useful in providing reliable information about health care quality [15,16].

This paper describes the process that was undertaken in the development of a hospital ratings platform for a consumer health care app. The aim to incorporate the best available evidence lies in sharp contrast to the majority of health related apps [17]. By working with patients and members of the public, we also sought to meet end-user needs often overlooked [18]. To develop the hospital ratings service, research was undertaken to determine which factors were considered important to individuals when choosing a nonemergency health care provider. This paper describes how established research methods can be used in the development of new mHealth apps.

Objectives

The aim of this study was to generate a list of quality indicators from the general public that were deemed important when choosing nonemergency secondary health care services along with the rationales for these choices, with the intention of using the findings in a new mHealth hospital ratings platform. Further, we aimed to illustrate the importance of rigorous research methodologies to underpin the development of mHealth technologies. The study was considered as part of a service evaluation, and ethics approval was not required. The study was conducted in London between November 2011 and June 2013.

Methods

Literature Review

As the first part of a mixed methods approach to identify which factors were considered most important to people when choosing nonemergency secondary health care providers in the United Kingdom, a review of existing literature (both academic and grey literature) was conducted. Publications were included only if they described patient choice in the United Kingdom health system so as to avoid confounding factors in the context of other health systems. For example, although there are many relevant articles from the United States, the differences between the largely privately funded United States and publicly funded United Kingdom systems may influence what users consider important when choosing a hospital. Patient choice did not feature prominently in national health policy until more recently [1,2]. A pilot literature review revealed a dearth of high quality research formally investigating patient choice in secondary health care prior to 2005. The patient choice agenda was first investigated when Lewis began studying patients’ attitudes towards choice of hospital in the context of waiting times for cardiothoracic surgery [19]. Studies were therefore excluded if they had been carried out prior to 2005.

Survey

The aim of the questionnaire was to formulate a ranked list of quality indicators. The survey was completed by participants with a member of the research team at hand to explain any terms or answer questions. Care was taken to ensure that facilitators did not directly ask questions to avoid leading or influencing participants choices. A power calculation [20] for a study comparing the attitudes regarding choosing secondary health care between two groups (general public and out-patients attending clinic appointments at the hospital) determined a target total sample size of 400 (population of London, 2001: 7,172,091 [21]; standard of error 0.05), thus two groups of approximately 200 participants. Members of the general public were recruited (n=201), and the data are presented here. Data collected from the second group (patients at the hospital) are beyond the scope of this manuscript. An initial pilot questionnaire was conducted on 20 individuals prior to its wider use to identify and correct any unforeseen problems. Quota sampling was used to estimate size of target groups to ensure accurate representation of ages and genders [22]. Four age categories were formed by combining existing categories from 2001 United Kingdom national census data [21]; target proportions for each age category were based on urban population proportions from the same census. Due to time and resource constraints, convenience sampling was then used to collect data, with checks to ensure collected data approximated the census age proportions. A higher proportion of people aged 18 to 35 years were included compared to people 60 years and older to account for lower usage of mHealth apps in the latter age group as described by previous investigators [23].

Inclusion criteria were English-speaking adults (18 years and older) UK residents. In order to ensure data collection was feasible under time and resource constraints, convenience sampling was then employed to recruit participants in six separate locations in central and greater London to provide greater geographical spread and wider generalizability of the data. Questionnaire collection ceased once the number of participants in each demographic group approached the estimated targets.

Informed consent was obtained for each participant completing the questionnaire. Participants were asked to provide demographic information and rank a predetermined list of quality indicators in order of importance. An ordinal scale was used, where respondents were asked to rank the factors in order of importance, first within their categories and then the categories themselves. This allowed us to assess the relative importance of the factors, as opposed to absolute importance [24]. Data were collated using Excel (Microsoft Corporation); SPSS (IBM Corporation) was used to undertake statistical analysis. Differences in mean ranks within categories were determined using the Friedman test, with P<.05 considered to be statistically significant.

Focus Groups

Focus groups were used to discuss the rationales behind the quality indicators considered to be important. Convenience sampling was used to recruit participants due to time and cost constraints, and referrals from initial recruits were used for further recruiting. We sought to recruit an equal representation of genders and ages in order to increase the generalizability of the results. Due to resource and time constraints it was not possible to match the age stratification of focus group participants with questionnaire respondents. The median age of participants at pilot focus groups was 40 years; therefore, participants were stratified by age and gender using this as a marker of division (males under 40 years, females under 40 years, males over 40 years, females over 40 years) to enable timely data collection and efficient analyses. Four focus groups were conducted, each comprising 6 individuals, with a gradual shift from broad open questions to narrow, focused questions [25]. Written consent was gained from each participant in advance. A final script of questions for the focus groups was confirmed following a restructuring of a preliminary script that had been piloted. Each focus group lasted for approximately 90 minutes and was led by one researcher acting as an impartial facilitator and one as an assistant moderator. All recordings were transcribed verbatim, and thematic analysis [26] was used to identify common themes.

Results

Characteristics of Quality Indicators

Literature Review

The full findings of the literature review are beyond the scope of this manuscript but we include key details pertinent to subsequent survey and focus group development. Searches of the grey literature were included due to a paucity of peer-reviewed publications. Five publications were identified for critical review [24,27-30]. Regular surveys commissioned by the UK Department of Health regarding the subject of choice in health care were also examined. A summary of included literature is presented in Textbox 1. From these, a list of choice factors important to patients selecting a health provider was devised. The factors identified in the literature review were separated into four categories of quality indicators: hospital statistics, hospital staff, hospital facilities, and hospital access (Textbox 2). These categories formed the questionnaire and informed the discussion topics for focus groups.

Questionnaire

Members of the general public completed the questionnaire (93 male, 108 female, n=201). The age spread of the sample compared to 2001 population proportions can be seen in Figure 1. Respondents ranked quality indicators in order of importance within their specified categories (ie, 1 through 5 with 1 being the most important). Based on mean rankings, the quality indicators were arranged in order of preference within respective categories (Table 1). Similarly respondents were asked to rank the overall categories (statistics, staff, facilities, and access) (Table 2). The Friedman test was used to determine the statistical significance of the differences between the mean ranks obtained for the quality indicators within and between categories to determine the true order (Multimedia appendix 1). Final ranked order of quality indicators and categories was determined after statistical analyses (Table 3).

While three of the categories (statistics, staff, facilities) were deemed equally important, quality indicators under the category of access were considered to be of less importance. Within each group some indicators were seen as being more important than others. Regarding staff, competence was seen as being significantly more important than friendliness and respectfulness. In terms of facilities, up-to-date facilities and the cleanliness of the premises were seen as equally important but more so than the other factors. In the category of statistics, infection rates, mortality rates, complication rates, and waiting times were of equal importance; statistics regarding readmission rates were seen as less important. Regarding access, ease of travel was more important that the cost and availability of car parking.

‎View this figure

Focus Groups

Four focus groups were used to explore the rationales behind rankings formulated from the questionnaire. Thematic analysis was conducted by performing manual coding [26], from which a collective list of codes was assembled. Overarching subthemes, and subsequently themes, were identified and reviewed. The themes, subthemes, and codes for different preferences established during the analysis are presented in Table 4 and are visually represented in Figure 2. The findings from both the quantitative and qualitative methodologies were mapped to each other under the four categories identified by the systematic literature review.

‎View this figure

Interpretation of Data

Hospital Statistics

Qualitative analyses of the focus groups showed that people consider a wide range of statistical descriptors. Participants reasoned that information regarding infection rates was an important consideration, mainly due to the extensive media coverage: this may account for why infection rates were ranked joint highest in this category during the quantitative analyses. Elsewhere, the qualitative data disagreed with the survey findings, asserting that readmission rates were also considered to be important by the focus group participants because people felt the rates reflected the success of a specific treatment or condition. Furthermore, unlike the survey respondents, the focus group participants did not feel that mortality rates were a valuable quality indicator because it was reasoned that a hospital may undertake more challenging cases, which could inflate mortality rates despite the hospital faring well on other indicators such as staff competence. Participants also speculated that in many cases departmental statistics may be more meaningful than those describing the hospital overall due to the variation between departments within a hospital, leading to potential misrepresentation of the overall hospital statistics. This interesting finding was not captured by the quantitative analyses and highlights a potential avenue for future research.

Hospital Staff

Staff competence was identified to be the most important factor by the quantitative analyses, followed by friendliness and respectfulness equally. Qualitative analyses were in agreement, with participants confirming the importance of receiving treatment by competent staff, with some even stating that they would travel further than their nearest hospital in order to secure treatment by a competent doctor. However, the focus groups identified that there is no single measure by which the public could judge competence. Rather competence was defined as a compound of experience, qualifications, place of education, or even the possession of excellent interpersonal skills.

Hospital Facilities

The quantitative analysis identifies cleanliness and modernity of equipment as the two highest ranked indicators. Rationales elicited from the focus groups shed light on why this may be the case. Members of the focus groups felt cleanliness to be very important in hospitals but did not necessarily seek out data about it when making a choice of hospital. It was suggested that this was a consequence of the assumption that cleanliness is the same in all hospitals. There was less consensus regarding the importance of modern equipment, although some certainly felt access to the latest technologies to be important.

Hospital Access

The quantitative analyses showed that this category was less important than the other three. Within this category, ease of traveling to the hospital was significantly more important than parking and cost of travel. Focus groups revealed that the proximity of the hospital to home or work was an important consideration. However, this is very much dependent on the severity of illness and the availability of treatment, with participants expressing that they may be willing to travel beyond their most proximal hospital in order to benefit from a higher quality of care. Therefore, it appears that the importance of this indicator may depend on the context of the decision.

Discussion

Key Findings and Recommendations

Service users across many health systems are now offered a wider choice of health care providers and increasingly have at their disposal a wide variety of factors to consider when making these decisions. The rapid adoption of mobile phones and tablet devices has enhanced access to information about different hospitals by making it possible for patients to view and share this information at any time and while on the move [31].

This study collates the existing literature regarding which factors are considered important for consumers in this context, contributes a categorized and ranked list of quality indicators, and reconciles the rationales underpinning these decisions. Furthermore, this study demonstrates how this information can be harnessed in the context of developing a robust user-generated ratings platform for use on mobile communication technologies.

Although mobile technologies are frequently put forward as a solution to challenges in health informatics, there is often a lack of rigorous research underpinning their development and evaluation. This project illustrates the importance of sound research methodology when developing these strategies by employing a mixed methods approach to reconfigure the ratings service based on factors that the public held to be important in choosing nonemergency health providers.

Findings included that staff competence was the most important factor within the hospital staff category, with participants asserting that they would travel further than their nearest hospital to secure treatment under a doctor they perceived to be more competent. However, the qualitative analyses revealed that there is no single measure by which competence could be judged; rather it was a compound of many factors including amount of experience, qualifications, place of education, and interpersonal skills.

Cleanliness and modernity of equipment stood out as the two most important hospital facilities. This is concurrent with previous reports that people consider information about cleanliness when researching a hospital [32,33]. Qualitative analyses discovered that while this was a factor deemed to be highly important, it was not widely sought after. Participants suggested this might be due to a commonly held assumption that hospitals are of equal cleanliness, therefore only those hospitals with a remarkably poor reputation for cleanliness may be of note. This was also the case for modernity of equipment. Future mHealth developers should reflect on this subtlety in order to include factors that are not only important but also highly sought after to avoid information overload for users.

Participants could not differentiate level of importance between various types of hospital statistics. Hospital-wide statistics may be of limited use to users who would be more interested in department-specific statistics. Moreover, users appreciate that overall hospital statistics may not be an accurate representation of an their department of interest due to interdepartmental variation. Conversely it may be argued that an inability to compare the importance of statistical descriptors may reflect that they are poorly understood by users. This may explain equal significance attributed to individual statistics within this category and highlights the need for the careful inclusion of statistics that are relevant to the user’s individual health encounter in mHealth platforms (See Figure 3). Care must be taken by developers to ensure that presentation of statistics, including color coding or a glossary of terms, aids user interpretation. These subtleties may not be appreciated without formal research methods informing these strategies.

The fact that the categories of hospital staff, hospital facilities and hospital statistics were deemed equally important illustrates that users’ demands for information about hospitals are extensive and varied. mHealth developers should aim to provide information about these categories equally in order to reflect and satisfy these demands. Adequate provision of these varied factors requires an equally varied presentation of information. For example, participants asserted that graphs and percentages provided objective evidence of statistical measures, whereas past users’ reviews were more useful in capturing complex domains such as staff competence. Therefore we recommend that mHealth developers include a range of formats as this study illustrates that each caters to different, and equally important, categories of quality indicators.

‎View this figure

Limitations

An important limitation of the study is that the questions were asked outside the context of mobile phones and mHealth. This was a purposeful decision as it was felt that doing so may lead to reduced applicability of this research. Further research is required specifically investigating whether the information consumers want in the context of an mHealth app is any different from the factors that are important when choosing secondary health care in general. The original study was adequately powered for a comparison between patient groups and the general public; the analyses included here may therefore by underpowered due to resource constraints. This study was unable to match age stratification between quantitative and qualitative stages. We recommend that future investigators attempt to do so to allow closer mapping of the two datasets.

Conclusion

The huge interest in developing apps for mobile phone and tablet platforms to enhance health outcomes and service delivery—widely termed mHealth—has led to an “enthusiastic proliferation of untested methods” [17]. An evidence base needs to be developed to make this field credible and address the needs of the end-user. More attention needs to be paid to structuring app development in theory or best practice [34].

This study used a mixed methods approach to find that information about hospital staff, hospital facilities and hospital statistics are equally important to people when choosing a hospital. Information about getting to the hospital is least important. Staff competence is most important regarding hospital staff, which is a multifactorial domain best captured by past users’ reviews; cleanliness and modernity of equipment are most important regarding hospital facilities but are not actively sought after. People find it difficult to compare relative importance between various hospital statistics. Barriers to understanding statistics may be removed by use of graphs and percentages.

Users of health care demand a wide and varied range of information about hospitals. mHealth developers must determine which information is most relevant to their users’ needs and provide this in an accessible format. Less important information must be identified and removed to avoid information overload. A sophisticated appreciation of the complex needs of mHealth users is possible when these strategies are underpinned by rigorous research methods. This study demonstrates how a mixed methods approach can enhance mHealth solutions.