Published on in Vol 9, No 6 (2021): June

Preprints (earlier versions) of this paper are available at, first published .
Considerations for the Design and Implementation of COVID-19 Contact Tracing Apps: Scoping Review

Considerations for the Design and Implementation of COVID-19 Contact Tracing Apps: Scoping Review

Considerations for the Design and Implementation of COVID-19 Contact Tracing Apps: Scoping Review


1Research Institute of the McGill University Health Centre, Montreal, QC, Canada

2Division of Pediatric Emergency Medicine, Department of Pediatrics, Montreal Children’s Hospital, McGill University Health Centre, Montreal, QC, Canada

3Research Centre of the University of Montreal Hospital Centre, Montreal, QC, Canada

4Centre d’excellence sur le partenariat avec les patients et le public, Montreal, QC, Canada

5Schulich Library of Physical Sciences, Life Sciences, and Engineering, McGill University, Montreal, QC, Canada

6Department of Health Policy, Management and Evaluation, School of Public Health, University of Montreal, Montreal, QC, Canada

Corresponding Author:

Esli Osmanlliu, MD

Division of Pediatric Emergency Medicine, Department of Pediatrics

Montreal Children’s Hospital

McGill University Health Centre

MUHC Glen Site

1001 Décarie Blvd

Montreal, QC, H4A 3J1


Phone: 1 514 412 4499

Fax:1 514 412 4217


Background: Given the magnitude and speed of SARS-CoV-2 transmission, achieving timely and effective manual contact tracing has been a challenging task. Early in the pandemic, contact tracing apps generated substantial enthusiasm due to their potential for automating tracing and reducing transmission rates while enabling targeted confinement strategies. However, although surveys demonstrate public interest in using such apps, their actual uptake remains limited. Their social acceptability is challenged by issues around privacy, fairness, and effectiveness, among other concerns.

Objective: This study aims to examine the extent to which design and implementation considerations for contact tracing apps are detailed in the available literature, focusing on aspects related to participatory and responsible eHealth innovation, and synthesize recommendations that support the development of successful COVID-19 contact tracing apps and related eHealth technologies.

Methods: Searches were performed on five databases, and articles were selected based on eligibility criteria. Papers pertaining to the design, implementation, or acceptability of contact tracing apps were included. Articles published since 2019, written in English or French, and for which the full articles were available were considered eligible for analysis. To assess the scope of the knowledge found in the current literature, we used three complementary frameworks: (1) the Holistic Framework to Improve the Uptake and Impact of eHealth Technologies, (2) the Montreal model, and (3) the Responsible Innovation in Health Assessment Tool.

Results: A total of 63 articles qualified for the final analysis. Less than half of the selected articles cited the need for a participatory process (n=25, 40%), which nonetheless was the most frequently referenced item of the Framework to Improve the Uptake and Impact of eHealth Technologies. Regarding the Montreal model, stakeholder consultation was the most frequently described level of engagement in the development of contact tracing apps (n=24, 38%), while collaboration and partnership were cited the least (n=2, 3%). As for the Responsible Innovation in Health framework, all the articles (n=63, 100%) addressed population health, whereas only 2% (n=1) covered environmental considerations.

Conclusions: Most studies lacked fundamental aspects of eHealth development and implementation. Our results demonstrate that stakeholders of COVID-19 contact tracing apps lack important information to be able to critically appraise this eHealth innovation. This may have contributed to the modest uptake of contact tracing apps worldwide. We make evidence-informed recommendations regarding data management, communication, stakeholder engagement, user experience, and implementation strategies for the successful and responsible development of contact tracing apps.

JMIR Mhealth Uhealth 2021;9(6):e27102




As the global battle against the COVID-19 pandemic continues, the SARS-CoV-2 virus has infected over 160 million people and claimed over 3.3 million lives by May 2021 [1]. Even as vaccination programs expand their reach, nonpharmaceutical interventions such as social distancing, isolation, and quarantining remain essential for reducing viral transmission. Many will not have access to the vaccine until later in 2021, as limited production and delivery capacity warrants prioritizing high-risk groups. Many countries risk even longer delays in gaining access, despite efforts to promote a more equitable global distribution of the vaccine [2]. Moreover, long-term immunity from vaccination is not assured, and new virus variants have been associated with increased contagiosity [3]. Nonpharmaceutical interventions must therefore continue to support the fight against COVID-19, in complement to the global ramping up of vaccination.

Contact tracing is a fundamental containment strategy in response to emerging outbreaks. Public health agencies aim to rapidly identify individuals who may have been exposed to a person who is infected to recommend the most appropriate course of action (eg, self-isolation, symptom recording, and testing). The incubation period of this virus can last up to 14 days, during which infected individuals can unsuspectingly contaminate others [4,5]. Presymptomatic transmission along with other epidemiological, social, economic, and political challenges [6], coupled with chronic underfunding of public health systems [7], have undermined the reach of manual contact tracing during the COVID-19 pandemic.

Contact Tracing Apps as eHealth Solutions During the COVID-19 Pandemic

A variety of eHealth solutions, which leverage information and communication technologies for the betterment of health and health care services [8], have been proposed in response to the pandemic [9]. Early in the pandemic, digital contact tracing rapidly emerged as a promising tool to support manual tracers [10,11] and enable a more selective approach to regional lockdowns [12]. Contact tracing apps constitute an example of eHealth aimed at supporting standard nonpharmaceutical interventions. Generally, the intended use is to digitally collect information within their network of users to reduce pathogen transmission. A recent review of international technological innovations developed in response to the pandemic listed almost 100 tracing applications at different stages of development, most of which were smartphone-based [13]. These can be further categorized as position tracking applications, which aim to enforce the quarantine of infected individuals, and the more commonly deployed contact tracing applications, which continuously measure distances between users to rapidly notify the high-risk contacts of an individual with a confirmed SARS-CoV-2 infection. The latter type predominates internationally, as it is more compatible with civil liberties. Although the term “contact tracing app” is typically used in academic texts and the general press, the primary purpose of such tools is to rapidly identify and notify individuals who have had a high-risk exposure [10]. Terminology matters as it may erode public trust in tools that are perceived to enable state surveillance of individual mobility (tracing, tracking). In this context, the contact tracing technology codeveloped by Apple and Google refers to an “Exposure Notification System” [14]. Similarly, the government of Canada encourages the public to use COVID Alert, an “exposure notification app,” which it describes as “an additional tool to protect yourself and your loved ones” [15].

Contact tracing apps can differ according to eight fundamental characteristics. First, their installation can be voluntary or compulsory. Second, the extent of informed consent varies between apps. Third, some apps use a decentralized data management strategy, while others enable linkages with governmental agencies. Fourth, their ability to detect contact between users can rely on technologies such as GPS, Bluetooth, or Quick Response codes. Fifth, the specific algorithms deployed in the back end of these apps will determine their output (eg, the calculation of a risk of infection or the tracing of potential contacts). Sixth, they require varying levels of human oversight, if at all present. Seventh, the degree of interaction with users regarding recommended actions (eg, testing and isolation recommendations) and the extent of interaction with public health agencies can differ. Last, safety protocols for data privacy may also vary [13,16,17]. Contact tracing apps thus refer to a heterogeneous cluster of eHealth tools that likely differ in effectiveness and uptake depending on their respective design characteristics.

App Effectiveness, Barriers to Adoption, and Facilitators

Given their fundamental mechanism, the effectiveness of contact tracing apps depends in part on the level of uptake and ongoing use [10,18] by patients who have contracted the SARS-CoV-2 virus and other citizens. An influential modeling study published by a team of researchers at the University of Oxford originally suggested that a 60% adoption rate should be targeted for effective virus transmission reduction, although any level of uptake may help lower disease transmission [19,20]. Even though numerous public consultations have suggested a general willingness to use contact tracing apps during the pandemic [21-28], the available data suggests low rates of continuous use in practice [27,29,30]. For instance, app penetration rates as of March 2021 were as low as 3.6% in France, 6.1% in Japan, 14% in Canada, and 28.5% in the United Kingdom. On the other hand, other countries such as Iceland and Finland have seen higher rates of adoption, at 38.5% and 45.3%, respectively [31,32]. The significant disparity in adoption rates worldwide highlights the fact that certain approaches could be better than others for successful implementation of this emerging technology.

As contact tracing apps encompass many underlying principles and disciplines, multiple aspects can facilitate or hinder their adoption. One of the main caveats in their implementation is concerns over data security and management. Societies are rightfully preoccupied with the challenges in reconciling civil liberties with public health imperatives in a pandemic context [33]. Data privacy, breaches in confidentiality, and the fear of mass surveillance are among the main concerns raised in surveys on user perspectives [21-24,27,28]. Moreover, as with other health informatics interventions, contact tracing apps may generate or exacerbate inequalities if they are not deployed carefully [34,35]. There is a risk of discrimination, repression, and systematic exclusion, especially among communities of color and marginalized groups, which are disproportionately affected by COVID-19 due to structural economic, political, and social vulnerabilities [35-37].

Key factors have been suggested as drivers for widespread success of contact tracing apps: integration with local health policy, adaptable workflows in an ever-evolving context, rapid notification systems, the ability to evaluate the effectiveness of the app transparently, and clear communications addressing privacy concerns [38].

New eHealth initiatives have emerged at an accelerating pace in the last decade; some have seen widespread adoption, whereas others have failed to provide sustained value. These failings can be attributed to design and implementation efforts that were initiated without a good understanding of the interdependencies between technology, societal values, and user experience in a health care setting. Many conceptual frameworks based on implementation science have been developed to evaluate and orient eHealth delivery. These frameworks highlight key factors that predict successful and sustainable eHealth technologies. The urgency of the ongoing public health crisis stimulated the rapid development of contact tracing apps and other eHealth innovations, and this generated a substantial number of related publications. Their coverage of essential design and implementation characteristics for eHealth innovation remains underinvestigated.

Objectives and Research Questions

The primary objective of this review is to map and analyze the literature on the design and implementation of COVID-19 contact tracing apps. This was achieved through three distinct questions: (1) to what extent does the available literature discuss features that promote the use of contact tracing apps by interested parties? (2) how have patients and citizens been engaged in the design and implementation of these apps? and (3) does the development of these apps correspond to principles of responsible research and innovation?

Through these questions, we studied how the development of contact tracing apps has taken into account considerations related to the uptake and impact of the innovation, the engagement of end users, and the responsible development of eHealth technologies. We ultimately identified the components required for successful and responsible eHealth development as described in the available literature on contact tracing apps and those that are lacking.

This study is reported according to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines [39] (Multimedia Appendix 1). The scoping review was conducted in accordance with the multistage framework outlined by Arksey and O’Malley [40], as detailed in the following sections.

Theoretical Frameworks

Three complementary frameworks were selected to address each of the research questions. To answer the first research question on factors that promote uptake of contact tracing apps, we selected the Framework to Improve the Uptake and Impact of eHealth Technologies proposed by van Gemert-Pijnen et al [41]. This framework advocates for a holistic approach comprising six guiding principles and requirements for successful eHealth technology development: (1) participatory processes (eg, upstream involvement of citizens in the selection of app features), (2) continuous evaluation cycles (eg, evaluation of the use and effect of the app on an ongoing basis to update risk assessments), (3) specific actions for implementation (eg, postdesign activities to promote or maintain app uptake), (4) foresight of changes in the organization of health care (eg, interaction of contact tracing apps with traditional public health processes), (5) persuasive design techniques (eg, technology-based suggestions to stimulate the uptake of contact tracing apps), and (6) advanced methods to assess impact (eg, key performance indicators to assess the impact of contact tracing apps). Given the novel application of contact tracing apps in the context of the COVID-19 pandemic, information regarding the latter stages of the framework may understandably be theoretical or perhaps even lacking.

For the second research question, which addresses the engagement of users in the development of contact tracing apps, we selected the Montreal model established by Pomey et al [42], as it focuses on the experiences and knowledge of patients and citizens regarding their health, trajectory of care, and related services [43]. It draws on the Patient and Family Engagement Framework proposed by Carman et al [44] to include a multidimensional assessment of patient engagement in health innovation [45]. For this study, we defined patients as individuals who contracted the SARS-CoV-2 virus. This framework proposes four distinct levels along a patient or citizen engagement continuum, in ascending order of engagement: (1) information (eg, end users have the option of viewing an explanatory video on the proposed app), (2) consultation (eg, a survey to evaluate propensity for app acceptance), (3) collaboration (eg, app adaptation based on recommendations from the public), and (4) coconstruction (eg, governments engage stakeholders to co-design the app). Selected studies that referred to a participatory process were therefore further characterized according to the level of engagement they described.

Finally, for the third research question relating to responsible health innovation, we selected the Responsible Innovation in Health Assessment Tool by Pacifico Silva et al [46]. This framework comes from the field of Responsible Research and Innovation and helps assess health innovations by addressing challenges such as sustainability and equity. The framework elucidates five value domains that need to be considered for responsible innovation: (1) the population health domain, which includes the subvalues of health relevance, ethical, legal and social issues, and health equity (eg, how the app can be made accessible to vulnerable groups); (2) the health system value domain, which includes subsections on inclusiveness, level of care, and responsiveness (eg, contact tracing apps reduce labor from manual contact tracing); (3) the economic domain (eg, cost-effectiveness of contact tracing apps as compared to other public health interventions); (4) the organizational domain (eg, business strategies to increase app value); and (5) the environmental domain (eg, how app architecture can help reduce the carbon footprint).

Searching for Relevant Studies

A systematic literature search was performed by one author (GG) in five databases (PubMed, Scopus, IEEE Xplore, AMC Digital Library, and Europe PMC) using terms specifically related to the research question. Textbox 1 describes the search strategy used on PubMed. Additional database search strategies are available in Multimedia Appendix 2. The search was first performed on August 26, 2020, and produced 829 results. A second iteration performed on November 6, 2020, generated 1130 results, for a total of 1959 articles.

Search strategy used on PubMed ([mesh] stands for Medical Subject Headings and indicates that the subject is indexed in the literature; [tw] indicates searches in the title and abstract fields).

((“contact tracing” [mesh] OR “epidemiological monitoring” [mesh]) AND (“mobile applications” [mesh] OR “algorithms” [mesh] OR “computer security” [mesh] OR “big data” [mesh] OR “computer simulation” [mesh] OR “geographic mapping” [mesh] OR “geographic information systems” [mesh] OR “microcomputers” [mesh] OR “software” [mesh]))

OR (tracing [tw] AND (app [tw] OR apps [tw] OR proximity [tw]))

OR ((contact [tw] OR exposure [tw]) AND notification* [tw] AND (app [tw] OR apps [tw] OR application* [tw]))

OR ((digital* [tw] OR mobile [tw] OR ehealth [tw] OR “eHealth” [tw] OR mhealth [tw] OR “m-health” [tw] OR app [tw] OR apps [tw] OR application* [tw] OR “geolocation*” [tw] OR “location service*” [tw] OR “location system*” [tw] OR “location information” [tw] OR gps [tw] OR big data [tw] OR ((geographic [tw] OR geographical [tw]) AND tracking [tw])) AND (“contact tracing” [tw] OR “contact tracking” [tw] OR “digital epidemiology” [tw]))

Textbox 1. Search strategy used on PubMed ([mesh] stands for Medical Subject Headings and indicates that the subject is indexed in the literature; [tw] indicates searches in the title and abstract fields).

Selecting Studies

The study eligibility criteria (Textbox 2) were informed by a priori knowledge and by the review process itself, in keeping with the scoping review methodology. After removing duplicates, the inclusion and exclusion criteria were applied to the remaining articles.

Each of the 1959 articles was screened by one of two authors (ER and JP), initially excluding those with titles and abstracts unrelated to the topic of study. They then read the remaining articles to determine their relevance to the research question with respect to the inclusion criteria. The senior authors (EO and MPP) screened articles with uncertain relevance for final inclusion or exclusion. Discrepant decisions were resolved through team discussion and consensus. This process resulted in the inclusion of 63 articles, following the search iterations previously described.

Eligibility criteria.

Inclusion criteria

  • Related to design considerations of contact tracing apps for COVID-19 (eg, privacy considerations, citizen inclusion and participatory approach, or incentivization)
  • Related to user experience or implementation approaches of contact tracing apps for COVID-19 (eg, surveys or focus groups, strategies to mitigate social vulnerabilities, or recommendations for governments communication with the public)

Exclusion criteria

  • Published before 2019
  • Published in a language other than French or English
  • Pertaining exclusively to aspects of computer science or technical developments of contact tracing apps
  • Pertaining exclusively to epidemiological feasibility or efficacy of contact tracing apps
  • Authors could not obtain access to the full article
Textbox 2. Eligibility criteria.

Charting the Data

Data from the included studies was charted on an extraction grid (Multimedia Appendix 3) according to the following categories: authors, title, date of publication, publication stage, aims of the study, and key findings. Articles were also classified into six categories: (1) proposal; (2) comment, editorial, or opinion piece; (3) survey or focus group; (4) case study; (5) review; and (6) essay. A description of each article type is provided in Textbox 3.

Description of the types of articles.


Considers the problems of a particular situation and offers a corresponding solution (eg, proposal to incentivize a contact tracing app [47])

Comment, editorial, or opinion piece

Reflects the author’s or journal’s opinion about a subject (eg, contact tracing app effectiveness and data security [48])

Survey or focus group

Concentrates on survey or focus group methods for data collection (eg, user acceptability of a contact tracing app [49])

Case study

Studies a particular case in depth (eg, development of the Trace Together app [50])


Examines what has already been discovered about a subject (eg, systematic evaluation of content and features of a contact tracing app [51])


Discusses ideas from the literature in a support of arguments about a specific subject (eg, discussion of an intervention to introduce contact tracing technology [52])

Textbox 3. Description of the types of articles.

Articles that outlined theoretical recommendations or criticism regarding the overarching concept of contact tracing apps were categorized as being based on theory (eg, ethical considerations of instantaneous contact tracing). In contrast, articles that factually described specific use cases or empirical studies were categorized as being based on practice (eg, description of technical features of a given app and their potential impact on implementation or surveys on user acceptability of a contact tracing app).

Furthermore, the extraction grid included the components of the Gemert-Pijnen et al [41], Pomey et al [42], and Pacifico Silva et al [46] frameworks, as previously discussed. One author (JP) coded each component to determine whether a given article addressed a framework (0 if not present, 1 if present) and extracted the supporting sentences, where applicable. A second author (ER) reviewed each component attribution and supporting sentences, as well as the theory or practice categorization, and reviewed the text for missing information. The principal authors then reviewed the analysis before reporting the results.

Collating, Summarizing, and Reporting the Results

Figures were produced using codes in accordance with the three theoretical frameworks used and other relevant information such as the date of publication, the type of article, and the theory or practice classification. Visualization of these elements shed light on changes in the available information and on gaps in the underreported domains of design and implementation of contact tracing apps. Critical appraisals of the included articles were beyond the scope of this study.

This scoping review generated a total of 1959 records. Following the removal of duplicates and the application of the selection criteria, 63 articles were included in the analysis (Figure 1).

Figure 1. Flowchart of the study selection process.
View this figure

Characteristics of Included Studies

The included studies (N=63) were published between April 16 and November 6, 2020, in 40 different journals or preprint databases. The majority (n=48, 76%) of included studies were published articles or e-prints, and 24% (n=15) were preprints. The number of published articles peaked in August (n=16, 25%; Figure 2). Theory-based studies were predominant from April to June, whereas the proportion of practice-based articles increased thereafter.

Figure 3 illustrates the types of articles by month of publication. Among the 63 included studies, most were surveys (n=16, 25%), followed by proposals and opinion pieces, editorials, or commentaries (n=13, 21%); reviews (n=10, 16%); essays (n=8, 13%); and case studies (n=3, 5%). Opinion pieces, editorials, and commentaries (in blue) were the predominant category before July 2020, followed thereafter by more diversity in publication types. Only surveys (n=1) were illustrated in our analysis in November, but this could be explained by the fact that our second and last iteration was carried out on November 6, 2020.

Figure 2. Number of monthly publications/preprints, according to theory- and practice-based categorization.
View this figure
Figure 3. Types of article by month of publication.
View this figure

Main Findings

To What Extent Does the Available Literature Discuss Features That Promote the Use of Contact Tracing Apps by Interested Parties?

Overall, all of the six principles of the Framework to Improve the Uptake and Impact of eHealth Technologies [41] were covered in less than half of the selected articles. Among these, mentions of either use cases or recommendations for a participatory process in the development of contact tracing apps appeared in 40% (n=25) of the 63 articles, which made it the most frequently discussed component (Figure 4). Conversely, only 16% (n=10) of the studies discussed continuous evaluation cycles of contact tracing apps. Persuasive design techniques were mentioned in 37% (n=23) of the articles, followed by implementation considerations (n=16, 25%), advanced methods to assess impacts (n=14, 22%), and foresight of changes in the organization of health care (n=12, 19%).

Figure 4. Proportion of articles integrating holistic eHealth development principles.
View this figure
How Have Patients and Citizens Been Engaged in the Design and Implementation of These Apps?

None of the selected articles referred to documentation given to citizens or patients on COVID-19 contact tracing apps, leaving the Montreal model “information” component [42] unaddressed (Figure 5). On the other hand, stakeholder consultation was the most frequently described level of engagement (n=24, 38%). It is worth noting that only 1 study related to patients and only 1 mentioned the creation of a focus group (in contrast to surveys of the public and reviews of public commentaries on the internet). One article briefly mentioned collaboration, as the responses to the conducted surveys “likely prompted the pivot to wearable tech due to its observation of the public’s reluctance to use their mobile phones for contact-tracing” [50]. Only 3% (2/63) mentioned the importance of partnership in the development and implementation of contact tracing apps, but none of the included articles described an app development process occurring in partnership with citizens or patients.

Figure 5. Proportion of articles integrating each level along the patient or citizen engagement continuum.
View this figure
Does the Development of These Apps Correspond to Responsible Research and Innovation?

Among the five domains of the Responsible Innovation in Health Assessment Tool [46], population health appeared in all of the selected studies, constituting the most frequently addressed component (Figure 6). In contrast, only 2% (n=1) of the 63 studies discussed the environmental component. A discussion of the health system was included in 79% (n=50) of the articles, while the economic and organizational dimensions were present in 19% (n=12) and 10% (n=6), respectively.

Figure 6. Proportion of articles integrating Responsible Innovation in Health domains.
View this figure
Design Considerations and Implementation Recommendations

The principal considerations and recommendations regarding contact tracing apps in the COVID-19 pandemic context were drawn from the key findings of analyzed articles and are summarized in Table 1. Our results show that the main themes of the 162 theoretical considerations and practical recommendations include data management (n=58, 35.8%), user experience (n=48, 29.6%), communication (n=23, 14.2%), research and implementation methodology (n=18, 11.1%), and the engagement of stakeholders (n=15, 9.3%). Here, user experience refers to the main reactions or perceived barriers that may affect uptake (which mainly comes from survey results), incentivization, app functionality, and other design considerations. Data management designates app architecture, data protection, privacy, monitoring of data, laws governing the use of data, and ethical consideration of data use. Communication covers all recommendations that communication from governments, health authorities, or app developers must be clear and transparent or other measures that may increase public trust in those institutions, such as rebranding and optimizing already existing apps, and reframing the terminology used. The engagement of stakeholders refers to consultation with citizens or patients, strategies to reach vulnerable populations, and theoretical considerations about equity in health access. Finally, research and implementation methodology includes the frameworks proposed to guide future research, general considerations of how to improve uptake, and theoretical considerations of what constitutes ethical app use.

Table 1. Themes of considerations and recommendations regarding app design and implementation.
ThemesConsiderations and recommendations found in the literature (N=162), n (%)
Data management (eg, the use of blockchain [53-56] and sunset clause [57])58 (35.8)
User experience (eg, rewards to app users [58] and simple design and user interface for interaction [59])48 (29.6)
Communication (eg, clarifying false beliefs about the app [29] and prevention campaign about individual risks [60])23 (14.2)
Research and implementation methodology (eg, NPTa framework to guide development and evaluation of complex DPTb interventions [61], apps must be necessary, proportional, scientifically valid, and time bound [62])18 (11.1)
Engagement of stakeholders (eg, low-cost wristband in low-socioeconomic areas [54] or alignment of the app with local culture and vulnerable populations [63])15 (9.3)

aNPT: Normalization Process Theory.

bDPT: digital proximity tracing.

Principal Findings

This study demonstrates that the volume of articles on design and implementation considerations for COVID-19 contact tracing apps grew rapidly in the first months of the pandemic and peaked in August 2020. As the evidence base increased, the proportion of opinion pieces decreased in comparison to surveys and review articles. This is not surprising given the novelty of this technology and the urgency around technological developments triggered by the pandemic. Accordingly, earlier articles mostly discussed principles related to the development and implementation of contact tracing apps mostly in theoretical terms, while the proportion of empirical articles increased starting in July. Despite the growing number of publications, their scope remained limited with respect to design and implementation considerations. Our findings demonstrate that critical design and implementation considerations were lacking in the early academic literature on contact tracing apps.

In fact, less than half of the selected articles cited the need for a participatory process, which nonetheless constituted the most frequently referenced item of the Framework to Improve the Uptake and Impact of eHealth Technologies. Only 40% (25/63) of the articles presented evidence of public or patient engagement in the contact tracing app development process. This stands in stark contrast with the ideal of “co-creation from ideation to operationalization” in eHealth technologies, as described in this framework [41]. Other principles from the same framework were infrequently addressed, such as the need for continuous evaluation cycles, the creation of new processes for health care delivery, the need to assess impact, and the proposing of specific actions for implementation.

We further analyzed the level of citizen and patient engagement in the development of contact tracing apps by using the Montreal model [42]. We found that, among the 28 studies that addressed this component, nearly all (n=24, 86%) consisted of public consultations. Although essential to inform design and implementation of eHealth technologies, consultation ranks lower on the patient engagement continuum as compared to collaboration and partnership. This imbalance may have resulted from the urgency of the public health crisis combined with the perception that citizen and patient engagement is time-consuming [64-66]. However, the upstream efforts deployed to promote collaboration and partnership with users provided key insights at early stages of development [67,68]. Ignoring user engagement or relying solely on downstream consultation may in fact delay implementation by triggering a need to redesign certain features or even contribute to ultimately unsuccessful ventures [68,69].

Moreover, when we assessed these studies according to the Responsible Innovation in Health framework [46], we found that most of the 63 articles described the impact of contact tracing apps at the population level and on health systems (n=63, 100% and n=50, 79%, respectively), mainly in theoretical terms. However, only a few acknowledged the environmental, organizational, and economic value domains (each domain was found in less than 20% of the included studies). Their near absence from the literature may indicate that these domains are not perceived as being relevant to the development of COVID-19 contact tracing apps.

The incompleteness of the academic literature on the design and implementation characteristics of COVID-19 contact tracing apps stems in part from the novelty of the topic and the need for timely innovations to fight the pandemic. As such, we did not expect the literature to address all the domains of the selected frameworks. However, few academic publications highlighted the need to assess the impact of this intervention or consider organizational challenges related to its deployment, among other key elements to successful and responsible eHealth innovation. Likewise, the limited references to higher forms of upstream end user engagement stand in contrast to this technology’s inherent reliance on widespread adoption. It may be that app developers or researchers did not perceive some of these factors as bearing sufficient relevance for incorporation at the development stage or for later publication. Nonetheless, more than one year since the start of the pandemic, many of these elements are still lacking and were highlighted as essential to the success of COVID-19 contact tracing apps [38]. Considering these findings, we hypothesize that the sense of urgency instilled by the pandemic motivated shortcuts away from a full compliance with best practices in eHealth development. In turn, this may have contributed to the unconvincing implementation of contact tracing apps to date. Future research will help determine which factors are most associated with the development of sustainable, feasible, ethically acceptable, and socially desirable technological solutions in the context of a public health emergency. As specific evidence on this topic continues to accumulate, we draw upon the existing literature to make some practical recommendations for successful contact tracing apps and related eHealth innovations.

Recommendations Regarding the Design Considerations and Implementation of COVID-19 Contact Tracing Apps

Data Management

Data protection and the looming risk of mass surveillance have understandably dominated the debate on the ethical, legal, and social implications of contact tracing apps. Users must therefore be intelligibly informed of the steps taken to protect their privacy. One consideration is that the risk of the proposed contact tracing app should be compared with that of the frequently used apps that most people have on their cellphones. For example, the New York Times “Privacy Project” recently revealed the tracking of millions of unsuspecting Americans through location-sharing apps (eg, apps used to access directions, weather information, or local fidelity programs) [70]. The heightened public scrutiny of contact tracing apps may thus serve as an opportunity to promote digital literacy and reflect on the responsible use (or misuse) of digital tools.


Clear and transparent messaging developed with and for citizens can be promoted through simple and familiar means, such as concise information labels inspired by those of the food industry. This approach was used by members of the health care machine learning community to promote the transparent and responsible use of clinical decision support tools [71]. If they are shown to be effective, clear communication and efforts to improve health literacy will further promote stakeholders’ engagement [72,73] and may increase public trust in governments and health authorities. Rebranding or reframing such as changing the terminology from tracking apps to exposure notification apps may also be a solution that would improve both public trust and app uptake. Moreover, given the large number of contact tracing apps developed thus far, a consistent presentation of fundamental design components (see the Introduction section) would assist stakeholders in their evaluation of a given app. It may also promote efficient comparisons between different apps and perhaps limit the risk of unnecessary duplication.

Engaging Key Stakeholders

Key stakeholders of contact tracing apps include potential users, technology developers, policy makers, and funding agencies. As previously noted, the prevailing form of engagement has been through public consultations, which is recommended in the analyzed articles. We would therefore argue that app developers and the various institutions that implement such technologies would benefit from greater upstream collaboration and partnership with individuals from diverse backgrounds, including patients. Engaging key stakeholders early in the process will also help identify the right problem and constraints, eventually narrowing the range of most suitable technological tools. Indeed, even if a contact tracing app perfectly identified high-risk contacts, it would not achieve the desired outcome of reduced viral transmission if it required prohibitively expensive hardware or if it relied on massive viral testing in a strained health care system unable to provide a sufficient number of diagnostic tests.

As previously described, the potential risks of contact tracing apps may disproportionately affect minority and marginalized groups [35-37]. It is important to seek such participation in the development of these tools to mitigate and, ideally, eliminate the risk of increasing disparities through their use. This recommendation is also supported by the literature, which prescribes strategies for reaching low socioeconomic groups and older adults. Unfortunately, it may be difficult to engage with certain groups, such as for people who experience homelessness, (digital) illiteracy, living in remote settings, or underrepresentation within the typical channels of citizen engagement due to systemic racism or other structural vulnerabilities. In these cases, one practical option by which app developers could enhance the diversity of voices contributing to their product is to partner with community leaders and individuals who work closely with underrepresented groups. Although this will require an early investment of time and resources, the dividends in terms of greater inclusiveness, accurate problem identification, and a broader assessment of the impact on outcomes across different populations will strengthen a project’s chances of achieving responsible design and successful implementation. Public and private funders can also play an important role in ensuring that effective collaboration occurs early in the development of contact tracing apps by prioritizing proposals that promote a participatory process.

User Experience

A simple and intuitive user interface may not only enhance the user experience at an individual level, it may also improve uptake at the population level. Apps that focus on interactive design features can help users better understand how to use the app correctly and more effectively, whereas apps that are focused on information may reduce the assimilation of this information by users if they are not visually appealing [74]. Moreover, resolving in-app technical shortcomings will likely improve the user experience. In fact, an Australian survey noted that 24% of respondents had listed technical concerns as a reason not to download a contact tracing app during the COVID-19 pandemic [27]. An easy system for reporting technical difficulties and a minimally disruptive evaluation of other app functionalities may therefore increase use. Moreover, since one main barrier to app uptake is privacy and data security concerns [21-23,27,29,49,75,76], users can be empowered by designs that use customizable app functionalities. For example, app users could be offered choices in the technology used to collect the data (GPS or Bluetooth), how the data is stored [77], and the level of interaction with local health agencies.

Simple and useful apps will likely incentivize uptake and use. This can be supported by developing apps that require readily available personal technology, personalized updates on pertinent and accurate information aligned with local guidelines, minimal disruption to daily functioning (eg, minimizing battery use [78]), and facilitated testing in collaboration with local health care systems. Poor coordination with local health actors may therefore significantly compromise the potential user value of contact tracing apps. On the other hand, locally integrated apps may help streamline testing when it is indicated and provide current trustworthy recommendations. In many instances, COVID-19 contact tracing apps were designed to complement established traditional public health interventions. Apps that work in silos may therefore be less appealing to the public, as their perceived usefulness and effectiveness may be compromised [79]. Furthermore, implementation should aim for interoperability in the identification and notification of high-risk contacts in the greater interest of public health. This may be more attainable in jurisdictions that share compatible legal frameworks for data protection and privacy, such as the General Data Protection Regulation [80].

Financial incentives should also be considered. A study that focused on tracking the use of Germany’s official contact tracing app, Corona-Warn-App, found that app uptake is more prevalent among older populations, individuals with pre-existing conditions, and those with high levels of education and income. Additionally, the study reported that information interventions, in the form of short videos addressing privacy, effectiveness, and app functionality issues, were useful in increasing users’ knowledge about the app but were not effective in driving uptake. On the other hand, interventions that provide a monetary incentive (as low as €1 [US $1.22], €3 [US $3.67], or €5 [US $6.11]) upon installation were found to be useful in increasing uptake [81].

Research and Implementation Strategies

Empirical evidence is urgently needed to determine whether the benefits of a given contact tracing app significantly outweigh its risks. Although contact tracing technologies were used in prior Ebola and influenza outbreaks, there is limited empirical evidence on their effectiveness. As for their impact in reducing viral transmission during the COVID-19 pandemic, it is mainly based on mathematical simulation models that used varied assumptions and methodologies [18]. Real-life estimated treatment effects, beyond the current simulation models, are urgently needed, with precise descriptions of the context in which the tool was used. In addition to the estimated effect of contact tracing apps on public health outcomes (eg, the impact on the basic reproduction number R0), studies should also report on the input data and the corresponding outputs. At the least, information on app downloading and daily use must be made available, along with a description of key app features. Such information would enable the study of factors that enhance uptake and the relationship between app uptake and public health effectiveness. Observational studies on contact tracing app effectiveness during a period when large-scale interventions are continuously being proposed and deimplemented will certainly be limited by significant biases. They can nonetheless shed light on the role played by digital contact tracing during this pandemic and future infectious disease outbreaks. This observational evidence can then be compared to outcomes from simulation-based studies, including a recent modeling that suggests that a digital contact tracing and exposure notification system can support traditional public health interventions in reducing transmission, even at participation levels as low as 15% [82].

In addition, academic reports on contact tracing apps and related eHealth innovations must adhere to established reporting guidelines such as CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) for eHealth and mHealth interventions [83]. The global scale of contact tracing apps makes the incorporation of the key domains and subdomains proposed in the Responsible Innovation in Health framework even more crucial than in local innovations. eHealth proposals should therefore indicate their predicted or estimated impact on health inequalities, health system inclusiveness and responsiveness, frugality, business model, and eco-responsibility. Funding agencies and academic journals must request that submissions comprehensively address these issues.


The findings of this scoping review must be interpreted considering certain limitations. First, this study includes articles published up to November 6, 2020. As the volume of publications continues to grow, a systematic review focused on narrower questions related to contact tracing apps may become relevant. We attempted to maximize the reach of our search by including multiple databases and by developing a rigorous study selection process. Relevant articles only available in the gray literature or exclusively in governmental databases may nevertheless have been missed, although official apps developed by governments, such as the National Health Service contact tracing app, were included in several of the articles analyzed. Furthermore, given the purposefully broad question posed by this study, the reproducibility of data extraction and charting presented some challenges. We addressed them by relying on well-established and complementary frameworks that were particularly appropriate to the research question. Moreover, multiple authors reviewed the process and provided supportive statements when a particular component or domain was considered to be present in a given article.


The emerging academic literature on contact tracing apps reveals significant knowledge gaps regarding their design and implementation. Key stakeholders are thus limited in their ability to critically appraise this eHealth innovation. Most of the included studies lacked fundamental aspects of the successful eHealth development and implementation framework. Similarly, few articles described the impact of contact tracing apps on the environmental, organizational, and economic domains, which are essential to evaluate responsible innovation in health. Among the studies that described a form of public participation, nearly all of them relied on consultation as opposed to collaboration or partnership. These overlooked components of eHealth development and implementation may have contributed to the modest uptake of contact tracing tools worldwide. They suggest a critical gap between theory and practice, whereby numerous academic sources promote a holistic and participatory approach to eHealth innovation, but few products incorporate them. Partnerships between app developers, researchers, policy makers, and users early in the development process will narrow this gap. Transparent, systematic, and comprehensive reporting of COVID-19 contact tracing app outcomes will further enable their critical appraisal. The lessons learned about the social acceptability of contact tracing apps as they were deployed at an unprecedented pace and scale must serve in future iterations of this innovation and in the development of other eHealth technologies aimed at sustainably supporting public health. They must attest to the importance of stakeholder engagement, problem identification, minimal system disruptions, longitudinal outcome measurement, and use incentivization.


This research was supported by the Observatoire international sur les impacts sociétaux de l'IA et du numérique and the Fonds de recherche du Québec. MPP also receives funding from the Fonds de recherche du Québec pour le soutien à la bourse de chercheur senior. We would like to thank Eleonora Bogdanova for her invaluable help in the literature search and Olivier Demers-Payette, Nathalie de Marcellis-Warin, and Christophe Mondin for their technical and methodological insight.

Conflicts of Interest

None declared.

Multimedia Appendix 1

PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) Checklist.

PDF File (Adobe PDF File), 110 KB

Multimedia Appendix 2

Additional database search strategies.

PDF File (Adobe PDF File), 64 KB

Multimedia Appendix 3

Database of included articles and analysis.

XLSX File (Microsoft Excel File), 41 KB


  1. Dong E, Du H, Gardner L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect Dis 2020 May;20(5):533-534 [FREE Full text] [CrossRef] [Medline]
  2. COVAX: working for global equitable access to COVID-19 vaccines. World Health Organization.   URL: [accessed 2021-01-04]
  3. New COVID-19 variants. Centers for Disease Control and Prevention.   URL: [accessed 2021-01-04]
  4. Coronavirus disease 2019 (COVID-19)‎: situation report, 73. World Health Organization. 2020 Apr 02.   URL: [accessed 2020-12-02]
  5. Gandhi M, Yokoe DS, Havlir DV. Asymptomatic transmission, the Achilles' heel of current strategies to control Covid-19. N Engl J Med 2020 May 28;382(22):2158-2160 [FREE Full text] [CrossRef] [Medline]
  6. Tokar S. UCSF Team presents ethical framework for effective COVID-19 contact tracing. UC San Francisco.   URL: https:/​/www.​​news/​2020/​10/​418841/​ucsf-team-presents-ethical-framework-effective-covid-19-contact-tracing [accessed 2020-12-02]
  7. The impact of chronic underfunding on America’s public health system: trends, risks, and recommendations, 2020. Trust for America's Health. 2020.   URL: [accessed 2020-12-02]
  8. Echelard J, Méthot F, Nguyen H, Pomey M. Medical student training in eHealth: scoping review. JMIR Med Educ 2020 Sep 11;6(2):e20027 [FREE Full text] [CrossRef] [Medline]
  9. Whitelaw S, Mamas MA, Topol E, Van Spall HGC. Applications of digital technology in COVID-19 pandemic planning and response. Lancet 2020 Aug;2(8):e435-e440.
  10. Ferretti L, Wymant C, Kendall M, Zhao L, Nurtay A, Abeler-Dörner L, et al. Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing. Science 2020 May 08;368(6491):eabb6936 [FREE Full text] [CrossRef] [Medline]
  11. Yasaka TM, Lehrich BM, Sahyouni R. Peer-to-Peer Contact Tracing: Development of a Privacy-Preserving Smartphone App. JMIR Mhealth Uhealth 2020 Apr 07;8(4):e18936 [FREE Full text] [CrossRef] [Medline]
  12. Parker MJ, Fraser C, Abeler-Dörner L, Bonsall D. Ethics of instantaneous contact tracing using mobile phone apps in the control of the COVID-19 pandemic. J Med Ethics 2020 Jul;46(7):427-431 [FREE Full text] [CrossRef] [Medline]
  13. Mondin C, de Marcellis-Warin N. Recension des solutions technologiques développées dans le monde afin de limiter la propagation de la COVID-19 et typologie des applications de traçage. DocDroid. 2020 Oct.   URL: [accessed 2020-12-02]
  14. Exposure notifications: using technology to help public health authorities fight COVID‑19. Google.   URL: [accessed 2020-12-02]
  15. Download COVID Alert today. Government of Canada. 2020 Sep 15.   URL: https:/​/www.​​en/​public-health/​services/​diseases/​coronavirus-disease-covid-19/​covid-alert.​html [accessed 2020-12-09]
  16. Exit through the App Store? A rapid evidence review on the technical considerations and societal implications of using technology to transition from the COVID-19 crisis. Ada Lovelace Institute. 2020 Apr 20.   URL: [accessed 2020-12-09]
  17. Kahn J, editor. Digital Contact Tracing for Pandemic Response: Ethics and Governance Guidance. Baltimore, MD: Johns Hopkins University Press; 2020.
  18. Braithwaite I, Callender T, Bullock M, Aldridge RW. Automated and partly automated contact tracing: a systematic review to inform the control of COVID-19. Lancet 2020 Nov;2(11):e607-e621.
  19. Digital contact tracing can slow or even stop coronavirus transmission and ease us out of lockdown. University of Oxford: Our Research.   URL: https:/​/www.​​Article/​2020-04-16-digital-contact-tracing-can-slow-or-even-stop-coronavirus-transmission-and-ease-us-out-of-lockdown [accessed 2021-04-16]
  20. O'Neill PH. No, coronavirus apps don’t need 60% adoption to be effective. MIT Technology Review.   URL: https:/​/www.​​2020/​06/​05/​1002775/​covid-apps-effective-at-less-than-60-percent-download/​ [accessed 2021-04-16]
  21. Altmann S, Milsom L, Zillessen H, Blasone R, Gerdon F, Bach R, et al. Acceptability of app-based contact tracing for COVID-19: cross-country survey study. JMIR Mhealth Uhealth 2020 Aug 28;8(8):e19857 [FREE Full text] [CrossRef] [Medline]
  22. Bachtiger P, Adamson A, Quint JK, Peters NS. Belief of previous COVID-19 infection and unclear government policy are associated with reduced willingness to participate in app-based contact tracing: a UK-wide observational study of 13,000 patients. medRxiv. Preprint posted online on June 6, 2020. [CrossRef]
  23. Jansen-Kosterink S, Hurmuz M, den OM, van VL. Predictors to use mobile apps for monitoring COVID-19 symptoms and contact tracing: a survey among Dutch citizens. medRxiv. Preprint posted online on June 2, 2020. [CrossRef]
  24. Jonker M, de Bekker-Grob E, Veldwijk J, Goossens L, Bour S, Rutten-Van Mölken M. COVID-19 contact tracing apps: predicted uptake in the Netherlands based on a discrete choice experiment. JMIR Mhealth Uhealth 2020 Oct 09;8(10):e20741 [FREE Full text] [CrossRef] [Medline]
  25. Kaspar K. Motivations for social distancing and app use as complementary measures to combat the COVID-19 pandemic: quantitative survey study. J Med Internet Res 2020 Aug 27;22(8):e21613 [FREE Full text] [CrossRef] [Medline]
  26. O'Callaghan ME, Buckley J, Fitzgerald B, Johnson K, Laffey J, McNicholas B, et al. A national survey of attitudes to COVID-19 digital contact tracing in the Republic of Ireland. Ir J Med Sci 2020 Oct 16:1-25 [FREE Full text] [CrossRef] [Medline]
  27. Thomas R, Michaleff ZA, Greenwood H, Abukmail E, Glasziou P. More than privacy: Australians’ concerns and misconceptions about the COVIDSafe App. medRxiv. Preprint posted online on July 29, 2020. [CrossRef]
  28. Walrave M, Waeterloos C, Ponnet K. Adoption of a contact tracing app for containing COVID-19: a Health Belief Model approach. JMIR Public Health Surveill 2020 Sep 01;6(3):e20572 [FREE Full text] [CrossRef] [Medline]
  29. Montagni I, Roussel N, Thiébaut R, Tzourio C. The French Covid-19 contact tracing app: usage and opinions by students in the health domain. medRxiv. Preprint posted online on November 12, 2020. [CrossRef]
  30. von Wyl V, Höglinger M, Sieber C, Kaufmann M, Moser A, Serra-Burriel M, et al. Are COVID-19 proximity tracing apps working under real-world conditions? Indicator development and assessment of drivers for app (non-) use. medRxiv. Preprint posted online on September 2, 2020. [CrossRef]
  31. O'Neill PH, Ryan-Mosley T, Johnson B. A flood of coronavirus apps are tracking us: now it's time to keep track of them. MIT Technology Review.   URL: [accessed 2021-04-16]
  32. MIT Technology Review. Covid Tracing Tracker. Google Docs.   URL: https:/​/docs.​​spreadsheets/​d/​1ATalASO8KtZMx__zJREoOvFh0nmB-sAqJ1-CjVRSCOw/​edit#gid=1464910624 [accessed 2021-04-08]
  33. Flood CM, MacDonnell V, Thomas B, Wilson K. Reconciling civil liberties and public health in the response to COVID-19: an RSC policy briefing. The Royal Society of Canada. 2020 Sep.   URL: [accessed 2020-12-02]
  34. Veinot T, Mitchell H, Ancker J. Good intentions are not enough: how informatics interventions can worsen inequality. J Am Med Inform Assoc 2018 Aug 01;25(8):1080-1088 [FREE Full text] [CrossRef] [Medline]
  35. Gasser U, Ienca M, Scheibner J, Sleigh J, Vayena E. Digital tools against COVID-19: taxonomy, ethical challenges, and navigation aid. Lancet 2020 Aug;2(8):e425-e434.
  36. Hendl T, Chung R, Wild V. Pandemic surveillance and racialized subpopulations: mitigating vulnerabilities in COVID-19 apps. J Bioeth Inq 2020 Dec;17(4):829-834 [FREE Full text] [CrossRef] [Medline]
  37. Toh A, Brown D. How digital contact tracing for covid-19 could worsen inequality. Human Rights Watch. 2020 Jun 04.   URL: [accessed 2020-12-02]
  38. Colizza V, Grill E, Mikolajczyk R, Cattuto C, Kucharski A, Riley S, et al. Time to evaluate COVID-19 contact-tracing apps. Nat Med 2021 Mar;27(3):361-362. [CrossRef] [Medline]
  39. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018 Oct 02;169(7):467-473 [FREE Full text] [CrossRef] [Medline]
  40. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodology 2005 Feb;8(1):19-32. [CrossRef]
  41. van Gemert-Pijnen JEWC, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011 Dec 05;13(4):e111 [FREE Full text] [CrossRef] [Medline]
  42. Pomey M, Flora L, Karazivan P, Dumez V, Lebel P, Vanier MC, et al. [The Montreal model: the challenges of a partnership relationship between patients and healthcare professionals]. Sante Publique 2015;27(1 Suppl):S41-S50. [Medline]
  43. Guide d’implantation du partenariat de soins et de services: vers une pratique collaborative optimale entre intervenants et avec le patient. Formation RUISSS. 2014 Feb 10.   URL: [accessed 2020-12-02]
  44. Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff (Millwood) 2013 Feb;32(2):223-231. [CrossRef] [Medline]
  45. Vanier MC, Dumez V, Drouin E, Brault I, MacDonald SA, Boucher A, et al. Partners in interprofessional education: integrating patients-as-trainers. 2014 Presented at: Partnering with Patients, Families, and Communities to Link Interprofessional Practice and Education; April 2014; Arlington, VA   URL:
  46. Pacifico Silva H, Lehoux P, Miller FA, Denis J. Introducing responsible innovation in health: a policy-oriented framework. Health Res Policy Syst 2018 Sep 10;16(1):90 [FREE Full text] [CrossRef] [Medline]
  47. Loi M. How to fairly incentivise digital contact tracing. J Med Ethics 2020 Jul 09:medethics-2020-106388 [FREE Full text] [CrossRef] [Medline]
  48. Nature. Show evidence that apps for COVID-19 contact-tracing are secure and effective. Nature 2020 Apr;580(7805):563. [CrossRef] [Medline]
  49. Williams SN, Armitage C, Tampe T, Dienes K. Public attitudes towards COVID-19 contact tracing apps: a UK-based focus group study. PsyArXiv. Preprint posted online on May 15, 2020. [CrossRef]
  50. Lee T, Lee H. Tracing surveillance and auto-regulation in Singapore: ‘smart’ responses to COVID-19. Media Int Aust 2020 Aug 12;177(1):47-60. [CrossRef]
  51. Ming LC, Untong N, Aliudin NA, Osili N, Kifli N, Tan CS, et al. Mobile health apps on COVID-19 launched in the early days of the pandemic: content analysis and review. JMIR Mhealth Uhealth 2020 Sep 16;8(9):e19796 [FREE Full text] [CrossRef] [Medline]
  52. Nijsingh N, van Bergen A, Wild V. Applying a precautionary approach to mobile contact tracing for COVID-19: the value of reversibility. J Bioeth Inq 2020 Dec;17(4):823-827 [FREE Full text] [CrossRef] [Medline]
  53. Arifeen MM, Al Mamun A, Kaiser MS, Mahmud M. Blockchain-enable contact tracing for preserving user privacy during COVID-19 outbreak. Preprints. Preprint posted online on July 22, 2020. [CrossRef]
  54. Chowdhury MJM, Ferdous MS, Biswas K, Chowdhury N, Muthukkumarasamy V. COVID-19 contact tracing: challenges and future directions. Preprints. Preprint posted online on June 19, 2020. [CrossRef]
  55. Klaine PV, Zhang L, Zhou B, Sun Y, Xu H, Imran M. Privacy-preserving contact tracing and public risk assessment using blockchain for COVID-19 pandemic. IEEE Internet Things Magazine 2020 Sep;3(3):58-63. [CrossRef]
  56. Xu H, Zhang L, Onireti O, Fang Y, Buchanan WJ, Imran MA. BeepTrace: blockchain-enabled privacy-preserving contact tracing for COVID-19 pandemic and beyond. IEEE Internet Things J 2021 Mar 1;8(5):3915-3929. [CrossRef]
  57. Culnane C, Leins K, Rubinstein BI. Tracking, tracing, trust: contemplating mitigating the impact of COVID-19 through technological interventions. Med J Aust 2020 Jul;213(2):94-94.e1 [FREE Full text] [CrossRef] [Medline]
  58. Cruz MM, Oliveira RS, Beltrão APV, Lopes PHB, Viterbo J, Trevisan DG, et al. Assessing the level of acceptance of a crowdsourcing solution to monitor infectious diseases propagation. 2020 Presented at: 2020 IEEE International Smart Cities Conference; September 28-October 1, 2020; Piscataway, NJ. [CrossRef]
  59. Azad MA, Arshad J, Akmal SMA, Riaz F, Abdullah S, Imran M, et al. A first look at privacy analysis of COVID-19 contact tracing mobile applications. IEEE Internet Things J 2020:1-1. [CrossRef]
  60. Guillon M, Kergall P. Attitudes and opinions on quarantine and support for a contact-tracing application in France during the COVID-19 outbreak. Public Health 2020 Nov;188:21-31 [FREE Full text] [CrossRef] [Medline]
  61. von Wyl V. Challenges for nontechnical implementation of digital proximity tracing during the COVID-19 pandemic: media analysis of the SwissCovid app. JMIR Mhealth Uhealth 2021 Feb 26;9(2):e25345 [FREE Full text] [CrossRef] [Medline]
  62. Morley J, Cowls J, Taddeo M, Floridi L. Ethical guidelines for COVID-19 tracing apps. Nature 2020 Jun;582(7810):29-31. [CrossRef] [Medline]
  63. Vandamme A, Nguyen T. Belgium - concerns about coronavirus contact-tracing apps. Nature 2020 May;581(7809):384. [CrossRef] [Medline]
  64. Gross O. Faire de l'engagement des patients une réalité. In: L'engagement des patients au service du système de santé. London: Doin - John Libbey Eurotext; 2018.
  65. Martineau JT, Minyaoui A, Boivin A. Partnering with patients in healthcare research: a scoping review of ethical issues, challenges, and recommendations for practice. BMC Med Ethics 2020 May 11;21(1):34 [FREE Full text] [CrossRef] [Medline]
  66. Ellis LE, Kass NE. How are PCORI-funded researchers engaging patients in research and what are the ethical implications? AJOB Empir Bioeth 2017;8(1):1-10. [CrossRef] [Medline]
  67. Subramanyam R, Weisstein FL, Krishnan MS. User participation in software development projects. Commun ACM 2010 Mar 01;53(3):137-141. [CrossRef]
  68. Supple D, Roberts A, Hudson V, Masefield S, Fitch N, Rahmen M, U-BIOPRED PIP group. From tokenism to meaningful engagement: best practices in patient involvement in an EU project. Res Involv Engagem 2015;1:5 [FREE Full text] [CrossRef] [Medline]
  69. Boote J, Dalgleish M, Freeman J, Jones Z, Miles M, Rodgers H. 'But is it a question worth asking?' A reflective case study describing how public involvement can lead to researchers' ideas being abandoned. Health Expect 2014 Jun;17(3):440-451 [FREE Full text] [CrossRef] [Medline]
  70. Thompson SA, Warzel C. Twelve million phones, one dataset, zero privacy. The New York Times. 2019 Dec 19.   URL: [accessed 2021-01-11]
  71. Sendak MP, Gao M, Brajer N, Balu S. Presenting machine learning model information to clinical end users with model facts labels. NPJ Digit Med 2020;3:41. [CrossRef] [Medline]
  72. Maurer M, Dardess P, Carman KL, Frazier K, Smeeding L. Guide to patient and family engagement: environmental scan report. Agency for Healthcare Research and Quality. 2012 May.   URL: [accessed 2021-01-07]
  73. Kaisler RE, Missbach B. Co-creating a patient and public involvement and engagement 'how to' guide for researchers. Res Involv Engagem 2020;6:32 [FREE Full text] [CrossRef] [Medline]
  74. Davalbhakta S, Advani S, Kumar S, Agarwal V, Bhoyar S, Fedirko E, et al. A systematic review of smartphone applications available for corona virus disease 2019 (COVID19) and the assessment of their quality using the Mobile Application Rating Scale (MARS). J Med Syst 2020 Aug 10;44(9):164 [FREE Full text] [CrossRef] [Medline]
  75. Sharma S, Singh G, Sharma R, Jones P, Kraus S, Dwivedi YK. Digital health innovation: exploring adoption of COVID-19 digital contact tracing apps. IEEE Trans Eng Manage 2020:1-17. [CrossRef]
  76. Ahmed N, Michelin RA, Xue W, Ruj S, Malaney R, Kanhere SS, et al. A survey of COVID-19 contact tracing apps. IEEE Access 2020;8:134577-134601. [CrossRef]
  77. Klar R, Lanzerath D. The ethics of COVID-19 tracking apps – challenges and voluntariness. Res Ethics 2020 Aug 05;16(3-4):1-9. [CrossRef]
  78. Active Irish COVID-19 tracing app users drop on battery problem - HSE. Reuters. 2020 Aug 27.   URL: [accessed 2021-01-11]
  79. Carman KL, Workman TA. Engaging patients and consumers in research evidence: applying the conceptual model of patient and family engagement. Patient Educ Couns 2017 Jan;100(1):25-29 [FREE Full text] [CrossRef] [Medline]
  80. GDPR checklist for data controllers. General Data Protection Regulation.   URL: [accessed 2020-04-13]
  81. Munzert S, Selb P, Gohdes A, Stoetzer LF, Lowe W. Tracking and promoting the usage of a COVID-19 contact tracing app. Nat Hum Behav 2021 Feb;5(2):247-255. [CrossRef] [Medline]
  82. Abueg M, Hinch R, Wu N, Liu L, Probert W, Wu A, et al. Modeling the combined effect of digital exposure notification and non-pharmaceutical interventions on the COVID-19 epidemic in Washington state. medRxiv. Preprint posted online on September 2, 2020. [CrossRef]
  83. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011 Dec 31;13(4):e126 [FREE Full text] [CrossRef] [Medline]

CONSORT-EHEALTH: Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth
PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews

Edited by L Buis; submitted 14.01.21; peer-reviewed by W Wilkowska, D Leightley, K Eason; comments to author 24.03.21; revised version received 14.04.21; accepted 20.05.21; published 09.06.21


©Esli Osmanlliu, Edmond Rafie, Sylvain Bédard, Jesseca Paquette, Genevieve Gore, Marie-Pascale Pomey. Originally published in JMIR mHealth and uHealth (, 09.06.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.