Published on 10.02.15 in Vol 3, No 1 (2015): Jan-Mar
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/3509, first published May 02, 2014.
Using a Collaborative Research Approach to Develop an Interdisciplinary Research Agenda for the Study of Mobile Health Interventions for Older Adults
Background: Seniors with chronic diseases are often called on to self-manage their conditions. Mobile health (mHealth) tools may be a useful strategy to help seniors access health information at the point of decision-making, receive real-time feedback and coaching, and monitor health conditions. However, developing successful mHealth interventions for seniors presents many challenges. One of the key challenges is to ensure the scope of possible research questions includes the diverse views of seniors, experts and the stakeholder groups who support seniors as they manage chronic disease.
Objective: Our primary objective was to present a case-study of a collaborative research approach to the development of an interdisciplinary research agenda. Our secondary objectives were to report on the results of a nominal group technique (NGT) approach used generate research questions and to assess the success of including non-academic researchers to enrich the scope, priority, and total number of possible research questions.
Methods: We invited researchers and stakeholders to participate in a full day meeting that included rapid-style presentations by researchers, health care professionals, technology experts, patients and community groups followed by group discussions. An NGT was used to establish group consensus on the following question: In your opinion, what research needs to be done to better understand the effectiveness, usability and design of mobile health apps and devices for older adults?
Results: Overall, the collaborative approach was a very successful strategy to bring together a diverse group of participants with the same end goal. The 32 participants generated 119 items in total. The top three research questions that emerged from the NGT were related to adoption, the need for high quality tools and the digital divide. Strong sub-themes included privacy and security, engagement and design. The NGT also helped us include the perspectives information from non-academic researchers that would not have been captured if the process had been limited to the research team.
Conclusions: Developing ways for patients and other stakeholders to have a voice when it comes to developing patient awareness as related to mHealth may guide future research into engagement, ownership, usability and design. It is our intention that our paper be used and adapted by other researchers to engage small or vulnerable populations often excluded from mHealth research and design.
JMIR mHealth uHealth 2015;3(1):e11
- mobile health;
- nominal group technique;
- participatory research: collaborative research;
- older adults;
mHealth and Older Adults
Mobile technologies (mHealth) are emerging as a way to engage the population in health care. mHealth refers to "...the provision of health services and medical and public health practice via mobile devices, mobile phones, patient monitoring devices, personal digital assistants, and other wireless devices" . For patients and providers alike, mHealth can improve access to information and can also cultivate communication on healthy living and disease management [ ]. In 2011, the United States (US) Secretary of Health and Human Services, Kathleen Sebelius described the mHealth innovation as “the biggest technology breakthrough of our time” that will “address our greatest national challenge” [ ]. The World Health Organization sees mHealth as a means to improve access to care and reduce professional isolation, especially with isolated populations or rural communities [ ].
There is a perception that older adults are late adopters of new technology. However, as of 2013, 59% of American adults aged 65 and over were online, 77% owned a mobile phone, 27% owned a tablet or e-reader, and 18% owned a smartphone . Similar adoption rates have been seen in Canada [ ], Britain [ ], and Australia [ ]. Recent surveys have also found older adults to be particularly interested in mobile tools to help prevent and manage disease [ , ].
Older adults are also high impact users of health care. In Canada, per capita health care spending is five times higher for seniors than for younger adults and this number is growing . In the US, seniors make up 12% of the population and account for a third of all health care spending [ ]. For our health care systems to be both efficient and effective in the long-term, our highest impact users must be able to receive care and then implement the recommended treatments in their own lives. And yet, at the moment, a mere half of us are willing or able to adhere to recommended treatments, with the oldest, sickest, poorest, and least literate struggling the most [ - ].
For mHealth to be effective, we need it to be accessible for older adults. Most mHealth research has focused on younger people who provide a poor proxy for the older user . In the face of what we know about the current digital divide, the concentration on existing users rather than high-impact users is also alarming. For example, three in four seniors report needing some help to learn to use a mobile device [ ]. As with people who tend to adhere to treatment, those who are online or who own a digital device are far more likely to be physically abled, healthy, educated, and wealthy [ , ] It is not age that prevents adoption so much as the age-related physical and cognitive changes that make it hard or frustrating to use a digital device designed for younger users [ , ]. And yet, despite the difficulty, 79% of seniors who are online feel that “people without Internet access are at a real disadvantage because of all the information they might be missing” [ ].
Group Consensus in Collaborative Research
mHealth research is multidisciplinary by necessity. The design, implementation, and evaluation of mHealth tools require expertise in health care, systems design, programming, and business. With so many stakeholders at the table, we need strategies to build consensus across diverse groups. Consensus building-strategies are often used to help group opinions converge . Two popular methods include the Delphi Technique and the Nominal Group Technique (NGT). If done well, both the Delphi and NGT strategies can help groups reach consensus while avoiding the common pitfalls of group dynamics, such as having an ‘expert’ take over or having a small number of participants dominate the discussion.
The Delphi technique is often used to develop clinical practice guidelines. It is particularly suited to helping large, diverse panels of experts reach consensus on the priorities and recommendations while minimizing the influence of individual panellists and the contact among panellists [- ]. It is a group method that is administered by a leader who assembles a panel of experts, asks questions, synthesizes feedback, and guides the group towards consensus [ ]. Unlike traditional survey methods, where the goal is to make generalizations across a population, the Delphi is an iterative process, more like a series of focus groups that leads the participants to a consensus. The goal is to reach an agreement in an area where none previously existed [ ]. The Delphi involves several rounds of surveys to gather feedback and interpret expert opinion. It continues until opinions converge. Because the Delphi technique is used to organize conflicting judgements, consensus may not be possible [ ].
The NGT is a qualitative method also used for consensus building[- ]. The NGT can be particularly useful for exploring new and emerging ideas in health care when results need to be prioritized [ , ]. It is an exploratory tool that helps a group generate ideas where the evidence base is limited. As ideas are shared, they are clarified by the group and then ranked. Unlike the Delphi technique, a key feature of the NGT is a face-to-face meeting that gives each participant equal voice in the creation and ranking of ideas [ ]. It is well suited to small groups, which need to quickly develop and agree on a list of ideas that can be ranked in order of importance or need. The NGT is also structured enough that it ensures that no single participant dominates the discussion.
When our research team began working together to establish an interdisciplinary research agenda for studying the intersections of mHealth and aging, it was clear that the topic was very complex and evolving. As a team, one of our goals was to understand the perceptions and judgements of both the ‘experts’ and the end users most affected by our research. We chose the NGT over the Delphi method because it allowed us to involve stakeholders who were deeply invested in health care but who had little interest or expertise in mHealth. In particular, we wanted to have a clear perspective on the needs of high impact users, which often include late and non-adopters of mHealth technologies. Thus, the objective of this paper is to share our experiences using an NGT with interdisciplinary researchers and health care stakeholders to develop an interdisciplinary research agenda for mHealth.
To engage our community, we organized an mHealth research event at the University of Waterloo School of Pharmacy that had three purposes: (1) to facilitate trans-disciplinary knowledge exchange among researchers and knowledge users dedicated to mHealth development for older adults; (2) include older users and their community supports in the discussion; and (3) identify research questions using the NGT.
Our sample size was chosen with purpose. It needed to be large enough to cover most opinions and perceptions but not so large as to lose focus on the purpose. We chose our representatives with this same purpose. Achieving full saturation was not the goal of this study; our intention was to get an initial broad sample of the ideas that people who would be developing, using or promoting mHealth apps . Although certain ideas were consistent among the groups, each group identified entirely new suggestions, indicating that a saturation of ideas could not be not reached. Considering that many participants voted for ideas generated by others, it may be worth repeating the event with similar groups in other communities, and to develop a forced-choice inventory based on the most consistently high-ranking ideas that could then be used by larger groups of participants [ ].
In planning the research day, we received ethics approval from the University of Waterloo Research Ethics Board (Office of Research Ethics #19064). Following the research day, the researchers reconvened and reviewed the results of the NGT activity and used the NGT methodology to develop the research agenda.
Participants and Recruitment
Meeting participants were recruited from Southwestern Ontario. Our recruitment goal was to include groups that often work on solutions in their own specific context, and rarely collaborate when developing creative solutions. To solicit a broad spectrum of opinions, we recruited participants who either had relationships with the 50-plus community or who were involved in the development of mobile apps. Specifically, we invited researchers in systems design and/or aging, health care providers for older adults, mobile technology professionals, community members living with chronic illness, and disease-specific advocacy organizations. Participants were not required to be experts in mHealth.
We mailed written invitations to 30 potential stakeholder organizations and individuals in May 2013. We followed up with up to three phone calls and recruited 32 participants who represented 18 organizations (). The participating health professionals were from geriatrics, nursing, physiotherapy, pharmacy, family medicine, and homecare. The patient and advocates represented patients living with diabetes, arthritis, and dementia. The technology professionals included programmers, developers, designers, and trainees.
To reach the goal of effective stakeholder collaboration, we used three strategies: rapid-style presentations, group discussion, and an NGT (Agenda,). The meeting began with three sets of 5-in-5 presentations where attendees gave 5 successive presentations lasting 5 minutes each followed by a break. All participants were invited to present their perspective on mHealth including challenges they were facing and questions they had. We limited presentations to 5 minutes to encourage participants to share a high-level overview not an in-depth analysis. Presenters were advised to follow either the Ignite event model and prepare 5 one-minute slides [ ] or the Pecha Kucha model and prepare 15-20 slides where each last 15-20 seconds [ ]. A 10-minute break was taken after each series of 5 presentations to allow all participants to share any thoughts, ideas, and questions that arose during the presentations. Following the presentations and discussion, we provided a one-hour lunch to encourage networking.
|10:00am – 10:30am||Welcome and 5x5 Minute Rapid Presentations (Researchers)|
|10:30am – 10:40am||Group discussion|
|10:40am - 11:10am||5x5 Minute Rapid Presentations (Health Providers)|
|11:10am – 11:20am||Group discussion|
|11:20am – 11:50 m||5x5 Minute Rapid Presentations (Patients, Technology Professionals)|
|11:50am – 12:00pm||Group discussion|
|12:00pm – 1:00pm||Networking Lunch|
|1:00pm – 2:00pm||Guided Group Discussions (Nominal Group Technique)|
Nominal Group Technique
For the second half of the workshop, the NGT method was used to identify potential research questions. Participants were split into three focus groups of 4 to 8 participants according to their background: health care providers, technology professionals, and community stakeholders. Each group followed the same NGT process and was facilitated by a member of the research team. Facilitators began by asking the following question: “In your opinion, what research needs to be done to better understand the effectiveness, usability, and design of mobile health apps and devices for older adults?”
Each facilitator followed a script adapted from a briefing on conducting NGT from the US Centers for Disease Control . We led groups through the following steps: (1) introducing/clarifying the research task; (2) generating ideas silently as individuals and then as a group (on a flip chart); (3) adding, merging, or removing ideas; (4) individually ranking the five most important ideas; (5) reviewing the aggregated rankings as a group; and (6) closing the session [ ].
After the initial NGT sessions, we determined that it would be beneficial to hold an additional session with the research team. Researchers across our institution were represented and included the disciplines of engineering, arts, sciences, business, and health. We began the researcher-focused NGT with a brief overview of what had happened during the original NGT sessions, and a very high level discussion of the results of the original NGT sessions.
Data Collection and Analysis
The ideas generated were intended to represent the distinct perspectives of stakeholders and researchers. The purpose of coding was to explore emerging themes, while being mindful that we were examining feedback collected from multiple participants [- ]. As described by Braun, our thematic analysis included a series of six interconnected stages (familiarization, generating initial codes, searching for themes among codes, defining, naming, and interpreting themes) that enabled us to move back and forth within the data to develop a coherent account of the phenomenon [ ]. Afterwards, two researchers (KM, KG) independently coded the generated (original and unedited) ideas and disparities were resolved by discussion. The codes were categorized into themes by one researcher (KM) and verified by a second researcher (KG).
The health professionals identified 32 phrases, the technology professionals identified 29 phrases, community stakeholders identified 19 phrases, and the researchers identified 39 phrases. The top five ranked phrases for each group are listed in. The themes that emerged as key priorities across all groups included adoption/motivation, privacy/security, need for high quality tools and the digital divide ( ).
Adoption and Motivation referred to the need to design mHealth tools that older adults can use, afford, and adapt for long-term use. It includes designing for people who have age-related impairments and having support in place to help older users learn to use new mHealth tools. Privacy/security refers to creating tools that protect confidentiality across multiple devices and systems while being transparent about who owns and accesses patient data. High quality tools refer to the need for evidence-based systems that provide accurate health information or advice and that are designed to effectively change behaviour. Finally, the digital divide refers to the need for systems that includes patients who would traditionally struggle including those with disabilities, lower income, lower literacy, and limited experience with technology.
As a group, the health care professionals focused on how they could support patients and were the only group that did not rank ‘the digital divide’ in their top themes. The technology professionals focused more on defining the end-user and on promoting usability for diverse groups. The community stakeholders focused on ways to be inclusive by promoting adoption. Finally, like the community stakeholders, the researcher group was most focused on inclusiveness but also noted the importance of having a quantifiable end-result.
At many points, participants repeatedly emphasized that they were not experts in mHealth and were unsure if their opinions were valid. Many were concerned that their personal experiences with technology did not represent all stakeholders. That said, at the conclusion of the meeting, many participants were deeply interested in the topic and requested online information sources and newsletters. Several participants also noted that mobile and online technologies were at the top of the list for future strategic planning in their organizations.
|Group||Top 5 Phrases by Theme||Votes|
|Health care providers||Patient generated data: Who will own the data? What should be shared? How should we share? When should we share? What will the patient be willing to share and receive? How do we convert patient data into intelligence for use by both patient/provider?||24|
|Flexibility in design: What is the \'right design for long-term use\'? Should it be tailored, have task-specific design, use plug and play, have multiple interfaces, use clear, non-technical language? What are the reasons for the failure of current designs? Usability? What are barriers to long-term use?||16|
|Support for patients: Should we provide patients with a support system or coach? Is that sustainable? Who receives the data and acts on them?||11|
|Check and balance system: How should we provide feedback to patients?||9|
|Affordability: How do we design frugally for the lowest common denominator?||7|
|Technology professionals||Target audience: What is/are the technology usage patterns, gender, ethnicity and personal models of our end users?||23|
|Devices: How to provide secure access for multiple devices?||12|
|Ease of use: What types of gestures are hard or easy for users?||10|
|Digital divide: Who has barriers to mobile technology (e.g., income)?||9|
|Archetypal problems in mobile health: Need to be defined||9|
|Community stakeholders||Motivators: What are the motivators for patients to use or keep using health technology?||9|
|Digital divide: Who is left behind by financial and physical limitations?||9|
|Teaching new users: How do we best teach new users while avoiding assumptions based on knowledge or education?||8|
|Assumptions: How do we avoid making assumptions about users?||7|
|Confidentiality & Security: How do we combat fear?||7|
|Researcher||Digital divide: How do you design motivation for older, less tech savvy populations?||14|
|Engagement: How can we employ our knowledge of behavioral psychology to improve user engagement?||12|
|Behaviour change: Can mobile devices change behavior (in older adults or health care providers)?||11|
|Robust design: How do we track technology development, big data development, and delivery system information to accelerate mHealth opportunities?||9|
|Evaluation: What metrics should be used instead of waiting for a longitudinal study?||9|
To better understand the effectiveness, usability, and design of mHealth tools for older adults, research should focus on adoption/motivation, privacy/security, defining “quality” and accessibility. While each group had a different perspective, all stakeholders were ultimately focused on end-user engagement and usability. Based on our NGT, the key to mHealth development for older users are to build tools that patients and health care providers can trust (evidence-based, secure) and that are accessible to end users (adaptable, affordable, easy to use).
We found that rapid-style presentations, group discussion and an NGT were simple and useful approaches to identifying a collaborative research agenda. It was also useful for educating and engaging stakeholders who would have been reluctant to partner in mHealth research in the past. Other researchers have had similar experiences using the NGT for consensus building. Giangregorio et al successfully used the NGT approach to identify future research priorities for osteoporosis . Chasens and Olshanskyn used the NGT to prioritize the problems experienced by patients with type 2 diabetes and found that it was a very useful tool for providing a voice to all participants [ ]. Further, Carney et al found the NGT helped bridge the gap between researchers and clinicians when looking at the needs of community nurses [ ].
We purposely laid out our meeting to give all participants a base of understanding for the topic through the initial discussion in the morning before we ran the NGT sessions. Everyone was given equal time and attention, and we organized the presentations in a way that gave no group authority over another group. In the planning phase, we told every participant they were welcome to create a presentation, but it was not mandatory. Following this, after the planned presentations at the morning meeting, we offered those who did not previously present the opportunity to present, or add their comments to the discussion, and nearly all participants did. All of our participants were very clear to emphasize what they did not know, and we found in every case their contributions to the discussion and the NGT sessions to be invaluable.
One observation that emerged during the NGT activities was that each group was very clear on what they did not know. For example, patients and advocates would repeat the statement "I'm not a programmer or researcher" in different ways throughout the dialogue. A benefit of the NGT is that it minimizes power differentials. In our case, separating our stakeholders into groups gave participants the opportunity to go through the NGT process among a group of people whom they felt they were on equal footing with. This was particularly evident in the patient group, where after one participant made the comment that they did not know about technology, the other participants said they did not know much either. It was also emphasized that we did not want or need them to be experts; rather the purpose was for us to get their ideas and input into future research directions.
Recently, Richardson and Reid outlined several barriers to engaging patients with mHealth that mirrored the concerns of our participants - the most significant being that there is a strong tendency in mHealth apps not to accommodate functional abilities of seniors . The best people to convey the abilities of seniors are seniors themselves. Participants’ comments about their discomfort with technology were one of the most valuable pieces of information we gathered. One of the greatest benefits to using the NGT method was that it gave participants the opportunity to offer us valuable insights that they were not aware they had, through providing support for them to give their perspectives and ideas. It also confirmed our idea that discomfort with the topic of technology was a barrier to engagement, which is important to guide both future research directions, as well as awareness to have while building collaborative partnerships between the various groups involved in these discussions.
The Dangers of Focusing on the Divide
The Digital Divide not only refers to the economic ability people have to purchase the technology but also to their ability to understand and use the technology. In our NGT, each group had their own concerns about how the digital divide affected both uptake and continued use of mHealth apps. Regardless of financial or educational constraints, each group at some point thought about and discussed how to best develop an app or tool that could be most accessible to the broadest group of people who may benefit from using it.
In publicly funded health systems such as those in Canada, Britain and Australia, equitable access to health care is considered a human right. Some imagine that the digital divide is a temporary problem that will vanish as physical technologies become cheaper [, ]. Others caution that unequal access leads not only to political and economic exclusion, but also to social exclusion [ ]. However, the risk of linking digital inclusion with social inclusion, and by extension, linking technological progress with social progress, is that we disregard people who have no interest in adopting a new technology, even with technological or financial capacity [ , ]. Wyatt et al suggest that the decision to not use the Internet, and by extension technologies, is a choice, and does not always reflect a disadvantaged position and group non-users in the following way [ , ]: (1) The resisters who have never used the Internet because they do not want to; (2) The rejecters who have stopped using the Internet voluntarily, perhaps because they found it boring or expensive; (3) The excluded who have never had access but would like it; and (4) The expelled who have lost access involuntarily.
In the above list, the first two categories are individuals who have shown agency in their decision whereas the latter two are limited by their own situation. While much of the literature on the digital divide focuses on the generic or ideal user, it will become increasingly important to examine the everyday practices of older adults and how their practices change with the inclusion, or exclusion, of mHealth.
Adoption and Ownership
It is generally accepted that we need to attain positive user attitudes to impact behavior and influence acceptance [, ]. It is also fairly well accepted that positive user attitudes are important predictors of systems usage, and by extension, success [ , ]. Interestingly, during our research day, neither the health care professional group nor the researcher group touched on the question of who exactly patients considered authorities in terms of information about technology adoption. Van Alstyne et al have noted that “ownership is critical to the success of information systems projects” and related it to “self-interest; owners have a greater vested interest in system's success than nonowners” [ ]. This attitude may lead to systems built to support the builders rather than the users [ ]. There is little literature on adoption and ownership of mHealth, and the majority of the available literature centers on health technology adoption in organizations. Going outside of the health sphere for information is essentially mandatory, and even then, there is little information available outside of the context of user acceptance.
Strengths and Limitations
Our overall goal was to bring the participants in at the pre-planning stage to ensure that their input informs the entire research cycle. We were also aware that the process may not end up being a fully participatory model, but we wanted to ensure that our research team made a deliberate effort to consider the needs of a very diverse stakeholder group that historically has not worked together collaboratively.
We acknowledge that at the last moment, six attendees from our patients/advocates group had to drop out due to health concerns. Considering that we were inviting adults with complex or serious health conditions, we anticipated that this might be a challenge. While we tried to compensate by inviting a higher proportion of patient groups, it likely limited the patient voices in the NGT. As a result, we made a greater effort to ensure that each patient and advocate in attendance had their opinions heard, and that the patients and advocates completed the NGT in their own group.
The NGT is a strategic and effective means of increasing productivity using focus group methods . Keeping in mind the warning from Delbecq et al that broadly stated or unfocused NGT questions are likely to elicit a variety of responses from persons who have had varied experiences [ ], we knew that our questions needed to be framed in a way that helped participants generate information to sufficiently convey their understanding and experiences. Before conducting the NGT, we paid careful attention to the question we would pose to the group. Furthermore, through using the NGT process, we found its depersonalized and highly structured format to be ideal for promoting a respectful, creative and meaningful discussion about personally important issues that we think would have been difficult to achieve with other focus group frameworks. It appeared to be particularly effective at minimizing the perceived hierarchical structures or power differentials among participants because the process allows for individual idea-gathering and generating, and give participants equal voice in the presentation of ideas.
While determining our criteria for recruitment, we particularly wanted to include the patient and patient-advocate voice because they are participants with a perspective that is both broad and specific to older adults. Our targeted community groups reflected this. As patients and patient representatives, they provided us with an opportunity to examine the specific perspectives of users with challenging needs, discuss mHealth efforts already in development by stakeholder groups and their successes, and derive insight into how users were engaging with existing mHealth technologies.
To augment the identification of relevant research questions, health care professionals were also targeted in order to probe the perspective of those providing care to potential users. Integrating an app into daily life requires both the help of the user’s health care support system and also an understanding of the additional demands on all of the users to gain buy-in of health support teams. Our perspective was that the patient is not the only knowledge user. mHealth tools that allow users to access health information at the point of decision-making, to receive real-time feedback and coaching and to monitor health conditions, must be equally beneficial or relevant to health care providers.
Although the structured format of the NGT is intended to maximize the greatest number of group responses, certain members may intentionally or unintentionally influence this process by their own agenda. A key goal of all three facilitators was to actively redirect the process toward the defined tasks of the NGT process. However, even when working within the boundaries of the NGT method, we faced challenges facilitating communication among multiple stakeholders. In most groups, one or two participants tried to consistently bring the discussion back to their own perspective or discuss a particular idea at length. Facilitation was necessary to ensure the groups remained on topic and avoided focusing intensely on a single point, particularly when the topic at hand was of personal importance to a participant. It is important to train facilitators properly in NGT, because a level of confidence is required to steer the discussion back to the boundaries of the NGT, particularly when working with invested individuals.
Ultimately, our hope is that this kind of collaborative approach - the Nominal Group Technique - can be used and adapted by other researchers to engage small or vulnerable populations often excluded from mHealth research and design. We believe that the case study experience presented here is transferable for researchers, community organizations, and others with a vested interest in promoting and encouraging mHealth advancements for seniors.
More work is clearly warranted to gather the perspectives of individuals and additional community groups. If we all believe that seniors are active users of mobile technologies and desire to be engaged in their health care programmes, then health care practitioners, technology developers, and other professionals have an obligation to involve them in both decisions about their care and their access to it via mHealth technologies.
This study was funded through a grant from the University of Waterloo Chronic Disease Prevention Initiative.
Conflicts of Interest
- Whittaker R, Merry S, Dorey E, Maddison R. A development and evaluation process for mHealth interventions: examples from New Zealand. J Health Commun 2012;17 Suppl 1:11-21. [CrossRef] [Medline]
- Shields T, Chetley A, Davis J. InfoDev working paper series.: InfoDev working paper series; 2005. Improving Health, Connecting People: The role of ICT in the health sector in developing countries: Summary of online consultations URL: http://www.infodev.org/en/Document.486.pdf [accessed 2015-01-12] [WebCite Cache]
- Sibelius K. NCI Cancer Bulletin. 2011 Dec 01. mHealth Summit Keynote Address URL: http://www.cancer.gov/ncicancerbulletin/121311/page4 [accessed 2015-01-12] [WebCite Cache]
- Mechael PN, Sloninsky D. The World Health Organization.: World Health Organization; 2008. Towards the development of an mHealth strategy: a literature review URL: http://www.who.int/goe/mobile_health/mHealthReview_Aug09.pdf [accessed 2015-01-12] [WebCite Cache]
- Research Center P. Accessed April 4, at. 2014. Older Adults and Technology Use (April 2014) URL: http://www.pewInternet.org/2014/04/03/older-adults-and-technology-use/ [accessed 2015-01-13] [WebCite Cache]
- Statistics Canada. Individual Internet use and e-commerce.: The Daily; 2013 Aug 28. URL: http://www.statcan.gc.ca/daily-quotidien/131028/dq131028a-eng.pdf [accessed 2015-01-13] [WebCite Cache]
- Office for National Statistics.: Office for National Statistics; 2013. Internet access quarterly update, Q4 URL: http://www.ons.gov.uk/ons/dcp171778_353031.pdf [accessed 2015-01-13] [WebCite Cache]
- Australia Bureau of Statistics.: Australia Bureau of Statistics; 2014. Household use of information technology, Australia 2012-13 URL: http://www.abs.gov.au/ausstats/abs@.nsf/Lookup/8146.0Chapter32012-13 [accessed 2015-01-13] [WebCite Cache]
- Accenture.: Accenture; 2013. Insight Driven Health: Silver Surfers are Catching the eHealth Wave URL: http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-Silver-Surfers-Are-Catching-The-eHealth-Wave.pdf [accessed 2015-01-13] [WebCite Cache]
- Barrett, L. AARP Research and Strategic Analysis.: AARP Research and Strategic Analysis Health and Caregiving among the 50+: Ownership, Use and Interest in Mobile Technology URL: http://assets.aarp.org/rgcenter/general/health-caregiving-mobile-technology.pdf [accessed 2015-01-13] [WebCite Cache]
- Canadian Institute for Health Information. Ottawa, Canada: Canadian Institute for Health Information; 2011. National Health Expenditure Trends 1975 to 2011 URL: https://secure.cihi.ca/free_products/nhex_trends_report_2011_en.pdf [accessed 2015-01-13] [WebCite Cache]
- Will You Still Treat Me When I'm 65? In: Alliance for Aging Research. Washington, DC: Alliance for Aging Research; 1996.
- George J, Shalansky SJ. Predictors of refill non-adherence in patients with heart failure. Br J Clin Pharmacol 2007 Apr;63(4):488-493. [CrossRef] [Medline]
- Ratanawongsa N, Karter AJ, Parker MM, Lyles CR, Heisler M, Moffet HH, et al. Communication and medication refill adherence: the Diabetes Study of Northern California. JAMA Intern Med 2013 Feb 11;173(3):210-218 [FREE Full text] [CrossRef] [Medline]
- Hansen RA, Maciejewski M, Yu-Isenberg K, Farley JF. Adherence to antipsychotics and cardiometabolic medication: association with health care utilization and costs. Psychiatr Serv 2012 Sep 1;63(9):920-928. [CrossRef] [Medline]
- Fitzgerald AA, Powers JD, Ho PM, Maddox TM, Peterson PN, Allen LA, et al. Impact of medication nonadherence on hospitalizations and mortality in heart failure. J Card Fail 2011 Aug;17(8):664-669. [CrossRef] [Medline]
- Task Force for Compliance. Noncompliance with Medications: An Economic Tragedy with Important Implications for Health Care Reform. Washington, DC: Task Force for Compliance; 1994. URL: http://www.npcnow.org/publication/noncompliance-medications-economic-tragedy-important-implications-health-care-reform [accessed 2014-01-03] [WebCite Cache]
- Osterberg L, Blaschke T. Adherence to medication. N Engl J Med 2005 Aug 4;353(5):487-497. [CrossRef] [Medline]
- Sabaté E. Adherence to Long-Term Therapies: Evidence for Action. In: World Health Organization. Geneva, Switzerland: World Health Organization; 2003.
- Ho PM, Spertus JA, Masoudi FA, Reid KJ, Peterson ED, Magid DJ, et al. Impact of medication therapy discontinuation on mortality after myocardial infarction. Arch Intern Med 2006 Sep 25;166(17):1842-1847. [CrossRef] [Medline]
- Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, et al. The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 2013;10(1):e1001362 [FREE Full text] [CrossRef] [Medline]
- Sciadas G. Catalogue no. 56F0009Xie. Ottawa, Canada: Statistics Canada; 2014. The digital divide in Canada URL: http://publications.gc.ca/Collection/Statcan/56F0009X/56F0009XIE2002001.pdf [accessed 2015-01-13] [WebCite Cache]
- Czaja SJ, Lee CC. Information technology and older adults. In: Sears A, Jacko J, editors. The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. New York, NY: Lawrence Erlbaum and Associates; 2007:777-792.
- Tang HH, Kao SA. Understanding the user model of the elderly people while using mobile phones. 2005 Jul 22 Presented at: HCII ’05 (Human Computer Interaction International); 22-27 July 2005; Cesars Palace, Las Vegas, Nevada, USA p. 21-28 URL: http://drhhtang.files.wordpress.com/2008/09/2005tangkao.pdf
- Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health 1984 Sep;74(9):979-983. [Medline]
- Davies S, Romano P, Schmidt E, Schultz E, Geppert J, McDonald K. Assessment of a novel Hybrid Delphi and nominal groups technique to evaluate quality indicators. Health Serv Res 2011;46:2005-2018.
- Delbecq AL, Van de Ven AH, Gustafson DH. Group Techniques for Program Planning a Guide to Nominal GroupDelphi Processes. Dallas, TX. USA: Scott-Foresman; 1975.
- Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: a research tool for general practice? Fam Pract 1993 Mar;10(1):76-81. [Medline]
- Delbecq AL. The nominal group as a technique for understanding the qualitative dimensions of client needs. In: Bell RA, editor. Assessing Health and Human Service Needs. New York: Human Sciences Press; 1983:210-218.
- Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995 Aug 5;311(7001):376-380 [FREE Full text] [Medline]
- de Ruyter K. Focus versus nominal group interviews: a comparative analysis. Mrkting Intelligence & Plan 1996 Nov 20;14(6):44-50. [CrossRef]
- Sim J. Collecting and analysing qualitative data: issues raised by the focus group. J Adv Nurs 1998 Aug;28(2):345-352. [Medline]
- Stolper E, van Leeuwen Y, van Royen P, van de Wiel M, van Bokhoven M, Houben P, et al. Establishing a European research agenda on 'gut feelings' in general practice. A qualitative study using the nominal group technique. Eur J Gen Pract 2010 Jun;16(2):75-79. [CrossRef] [Medline]
- Centres for Disease ControlPrevention. Department of Health and Human Services.: Centres for Disease Control; 2006. Gaining Consensus among stakeholders through the Nominal Group Technique URL: http://www.cdc.gov/healthyyouth/evaluation/pdf/brief7.pdf [accessed 2015-01-13] [WebCite Cache]
- Hutchings A, Raine R. A systematic review of factors affecting the judgments produced by formal consensus development methods in health care. J Health Serv Res Policy 2006 Jul;11(3):172-179. [CrossRef] [Medline]
- Mason M. Sample size and saturation in PhD Studies using qualitative interviews. Forum: Qualitative Social Research 2010 Sep;11(3).
- Bostwick D, Foss G. Obtaining consumer input: two strategies for identifying and ranking the problems of mentally retarded adults. Education & Training of the Mentally Retarded 1981;16:207-212.
- Ignite Events. How To URL: http://igniteshow.com/howto [accessed 2015-01-13] [WebCite Cache]
- Pecha Kucha (website).: Pecha Kucha FAQ URL: http://www.pechakucha.org/faq [accessed 2015-01-13] [WebCite Cache]
- Delbecq AL, Van der Ven AH. A group process model for problem identification and programme planning. J Appl Behav Sci 1971;7:466-492.
- Frohna A, Stern D. The nature of qualitative comments in evaluating professionalism. Med Educ 2005 Aug;39(8):763-768. [CrossRef] [Medline]
- Kluger AN, DeNisi A. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 1996;119(2):254-284.
- Shute VJ. Focus on Formative Feedback. Review of Educational Research 2008 Mar 01;78(1):153-189. [CrossRef]
- Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006 Jan;3(2):77-101. [CrossRef]
- Giangregorio L, MacIntyre N, Heinonen A, Cheung A, Wark JD, Shipp K, et al. Too Fit To Fracture: a consensus on future research priorities in osteoporosis and exercise. Osteoporos Int 2014 May;25(5):1465-1472. [CrossRef] [Medline]
- Chasens ER, Olshansky E. Daytime sleepiness, diabetes, and psychological well-being. Issues Ment Health Nurs 2008 Oct;29(10):1134-1150. [CrossRef] [Medline]
- Carney O, McIntosh J, Worth A. The use of the Nominal Group Technique in research with community nurses. J Adv Nurs 1996 May;23(5):1024-1029. [CrossRef]
- Richardson JE, Reid MC. The promises and pitfalls of leveraging mobile health technology for pain care. Pain Med 2013 Nov;14(11):1621-1626 [FREE Full text] [CrossRef] [Medline]
- Castells M. The Internet Galaxy. Oxford: Oxford University Press; 2001.
- Compaine B, editor. The Digital Divide, Facing a Crisis or Creating a Myth?. Cambridge, MA: MIT Press; 2001.
- Cammaerts B. Beyond the Digital Divide: Reducing Exclusion, Fostering Inclusion. Brussles: VUB Brussels University Press; 2003.
- Henwood F, Wyatt N. Critical perspectives on technologies, in/equalities and the information society. In: Henwood F, Wyatt S, Miller N, Senker P, editors. Technology and in/equality: Questioning the information society. London, England: Routlage; 2000:1-20.
- Mansell R. From Digital Divides to Digital Entitlements in Knowledge Societies. Current Sociology 2002 May 01;50(3):407-426. [CrossRef]
- Wyatt S, Thomas G, Terrenova T. They Came, They Surfed, They Went Back to the Beach, Conceptualising Use and Non-use of the Internet’. In: Woolgar S, editor. Virtual Society? Technology, Cyberbole, Reality. Oxford: Oxford University Press; 2005:199-218.
- Wyatt S. The digital divide, health information and everyday life. New Media & Society 2005 Apr 01;7(2):199-218. [CrossRef]
- Joshi K. An Investigation of Equity as a Determinant of User Information Satisfaction. Decision Sciences 1990 Dec;21(4):786-807. [CrossRef]
- Lucas HC. Empirical evidence for a descriptive model of implementation. MIS Quarterly 1978;2:27-41.
- Davis F. User acceptance of information technology: System characteristics, user perceptions and behavioural impacts. International Journal of Man-Machine Studies 1993;38:475-487.
- Grantham CE, Vaske JJ. Predicting the usage of an advanced communication technology. Behaviour and Information Technology 1985;4(4):327-335.
- VanAlstyne M, Brynjolfsson E, Madnick S. Why not one big database? Principles for data ownership. Decision Support Systems 1995;15:267-284.
- Clegg C, Axtell C, Damodaran L, Farbey B, Hull R, Lloyd-jones R, et al. Information technology: a study of performance and the role of human and organizational factors. Ergonomics 1997 Sep;40(9):851-871. [CrossRef]
- Forsyth DR. Group dynamics (2nd ed.). Belmont, CA: Brooks/Cole; 1990.
- Delbecq A, Van de Ven A, Gustafson D. Group techniques for program planning: a guide to nominal group and delphi processes. Middleton, Wis: Green Briar Press; 1986.
|NGT: nominal group technique|
Edited by G Eysenbach; submitted 02.05.14; peer-reviewed by M Mishkind, L Klein; comments to author 16.09.14; revised version received 30.10.14; accepted 06.11.14; published 10.02.15
©Kathryn Mercer, Neill Baskerville, Catherine M Burns, Feng Chang, Lora Giangregorio, Jill Tomasson Goodwin, Leila Sadat Rezai, Kelly Grindrod. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 10.02.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.