Published on in Vol 9, No 3 (2021): March

Preprints (earlier versions) of this paper are available at, first published .
Analyzing the Essential Attributes of Nationally Issued COVID-19 Contact Tracing Apps: Open-Source Intelligence Approach and Content Analysis

Analyzing the Essential Attributes of Nationally Issued COVID-19 Contact Tracing Apps: Open-Source Intelligence Approach and Content Analysis

Analyzing the Essential Attributes of Nationally Issued COVID-19 Contact Tracing Apps: Open-Source Intelligence Approach and Content Analysis

Original Paper

Health Informatics Research Group, Faculty of Business Management and Social Sciences, University of Applied Sciences Osnabrueck, Osnabrück, Germany

Corresponding Author:

Ursula Hübner, PhD

Health Informatics Research Group

Faculty of Business Management and Social Sciences

University of Applied Sciences Osnabrueck

Caprivistraße 30a



Phone: 49 5419692012


Background: Contact tracing apps are potentially useful tools for supporting national COVID-19 containment strategies. Various national apps with different technical design features have been commissioned and issued by governments worldwide.

Objective: Our goal was to develop and propose an item set that was suitable for describing and monitoring nationally issued COVID-19 contact tracing apps. This item set could provide a framework for describing the key technical features of such apps and monitoring their use based on widely available information.

Methods: We used an open-source intelligence approach (OSINT) to access a multitude of publicly available sources and collect data and information regarding the development and use of contact tracing apps in different countries over several months (from June 2020 to January 2021). The collected documents were then iteratively analyzed via content analysis methods. During this process, an initial set of subject areas were refined into categories for evaluation (ie, coherent topics), which were then examined for individual features. These features were paraphrased as items in the form of questions and applied to information materials from a sample of countries (ie, Brazil, China, Finland, France, Germany, Italy, Singapore, South Korea, Spain, and the United Kingdom [England and Wales]). This sample was purposefully selected; our intention was to include the apps of different countries from around the world and to propose a valid item set that can be relatively easily applied by using an OSINT approach.

Results: Our OSINT approach and subsequent analysis of the collected documents resulted in the definition of the following five main categories and associated subcategories: (1) background information (open-source code, public information, and collaborators); (2) purpose and workflow (secondary data use and warning process design); (3) technical information (protocol, tracing technology, exposure notification system, and interoperability); (4) privacy protection (the entity of trust and anonymity); and (5) availability and use (release date and the number of downloads). Based on this structure, a set of items that constituted the evaluation framework were specified. The application of these items to the 10 selected countries revealed differences, especially with regard to the centralization of the entity of trust and the overall transparency of the apps’ technical makeup.

Conclusions: We provide a set of criteria for monitoring and evaluating COVID-19 tracing apps that can be easily applied to publicly issued information. The application of these criteria might help governments to identify design features that promote the successful, widespread adoption of COVID-19 tracing apps among target populations and across national boundaries.

JMIR Mhealth Uhealth 2021;9(3):e27232



The COVID-19 pandemic has drawn unprecedented attention to public health measures and exposed weaknesses in governmental pandemic management efforts throughout all nations. In particular, the evidence on presymptomatic virus transmissions [1] and the large variance in severity between asymptomatic disease progression and deadly disease progression [2] has delineated the limits of centrally coordinated and executed test and trace programs. This has led to increased attention and the development of COVID-19 tracing apps for smartphones, which have been deemed as feasible and potentially effective tools for pandemic management [3]. As a result, apps are now seen as potentially important tools for supporting pandemic management; they provide a promising opportunity to complement the tracing efforts of local health authorities.

The effectiveness of contact tracing apps however depends on how many people in a given population use the app [4], which poses a unique sociotechnical challenge. A study by Trang et al [5] suggests that the large-scale willingness to use such apps is closely tied to the design factors of the app itself. This is especially true for the following design factors: privacy preservation, transparency, and convenience. Furthermore, when expanding the scope of app use to an international level, demands for transparent and open app developments become even more pressing. Contact tracing apps should not only have the ability to function in one country, but also have the ability to be interoperable with the solutions used in other countries (or the same app should be used in multiple countries), as SARS-CoV-2 spreads around the globe with little regard for national borders. At the same time, app users must be assured that a given contact tracing app is not misused as a surveillance tool by the issuing government. Such misuse might compromise human rights, privacy, and the acceptance of app use. Thus, technical app designs, especially those that relate to privacy-preserving elements, have far-reaching consequences.

The challenges of effectively using contact tracing apps are fundamentally similar across governments. This calls for the close monitoring of the different approaches to guiding governments in choosing the most promising apps. Although several studies have investigated the acceptance factors of COVID-19 tracing apps [5-8], limited research has been conducted on cross-country comparisons. O’Neill et al [9] published an overview of several national COVID-19 tracing apps. However, their study was not conducted in a scientific context, and they used a rather small set of evaluation criteria [9]. Ming et al [10] assessed various COVID-19–related apps across several countries but only briefly touched on these apps’ tracing functionalities. Furthermore, checklists and item sets that pertain to the evaluation of general mobile health apps [11-15] do not seem particularly suited for evaluating contact tracing apps, as most criteria do not apply or are too unspecific. Therefore, our goal was to develop an item set that was suitable for describing and monitoring nationally coordinated COVID-19 contact tracing apps. This item set could provide a framework for describing the key technical features of such apps and monitoring their use based on widely available information.

In order to develop a suitable evaluation framework that accounts for feasibility in terms of the public availability of relevant information, we conducted an inductive procedure in which we screened the development and use of COVID-19 tracing apps in different countries over several months (from June 2020 to January 2021). The process of identifying and monitoring the apps was performed by using an open-source intelligence (OSINT) approach [16]. This procedure lends itself to the collection of data and information on new and emerging situations wherein there is only limited scientific knowledge available. Although this method involves drawing on scientific publications in journals, it goes beyond this and draws on a multitude of publicly available sources such as (1) grey literature (technical reports, preprints, white papers, and business documents); (2) government data (reports, press conferences, websites, and speeches); (3) conventional media (newspapers, magazines, radio, and television); and (4) the internet (web-based publications, blogs, and social media).

This approach enables the gathering of the most current information in a timely manner and helps to quickly assess the different cultural contexts of various countries. Textbox 1 provides an overview of the sources that were used in this study.

Two authors with backgrounds in medical informatics and information systems conducted parallel searches for publicly available information. Collected materials were reviewed by a third author who had a background in medical informatics. The third author also checked the collected information to determine whether the same information from different sources matched or contradicted each other. The search engines Google, DuckDuckGo, and Google Scholar were used for the search (search term combination format: “COVID-19 tracing app name” + “country name”). In addition, Wikipedia entries for COVID-19 contact tracing apps that were written in the native language of the apps’ respective countries were translated by using services that were provided by Google and Microsoft (for information that was hardcoded in pictures or fliers, the Google Translate app was used). This was done to find references for COVID-19 contact tracing apps, as Wikipedia pages that were written in the native languages of the apps’ respective countries were more up to date than those that were written in English and German (the two languages spoken by the authors). Our main information source was official information that was provided by the countries themselves (ie, technical reports from the apps’ developers or media statements that were provided by official representatives of the countries). Media and academic publications were used to verify the information that was provided by the official sources of the countries or to find information that was not provided by official sources.

Overview of the type of sources in this study.

Academic publications (9 documents) [5-8,17-21]

  • Peer-reviewed information from conferences or journals
  • Can also include nonpeer-reviewed sources like theses and dissertations

Public government data (28 documents) [22-48]

  • Official sources, such as public government reports, press releases, government websites, or development repositories (eg, GitLab and GitHub)

Grey literature (5 documents) [49-53]

  • Technical reports or preprints

Media (11 documents) [9,54-63]

  • News and articles from newspapers, magazines, radio, television, and podcasts (eg, British Broadcasting Corporation, The Guardian, and The New York Times), especially those on the internet
  • News and articles from social media websites (eg, Twitter, YouTube, etc)
Textbox 1. Overview of the type of sources in this study.

The initial entry points for researching relevant information revolved around subject areas that were reported to be important design factors in previous studies [5,18-21,53] on tracing apps, such as transparency, privacy protection, data processing techniques (the protocols used and the centralization of data processing), and app features. The documents and websites that were gathered from selected countries were then analyzed by using content analysis methods, which were applied throughout several iterations to inductively extract coherent, topical categories that emerged from the available information [64]. Each category was refined into subcategories and a set of related features. These were subsequently paraphrased as question items. The final item set was then reapplied to the selected countries.

The iteration process, its findings, and the final item set framework are presented in the Results section. Figure 1 summarizes our entire methodological approach. These methodological steps were applied to information from a sample of countries with contact tracing apps that were officially provided by the government to (1) automatically inform users about whether they have potentially been exposed to COVID-19 and (2) help public health officials with tracing and containing the spread of SARS-COV-2. The sample of national tracing apps was purposefully selected; our intention was to include apps from different countries from around the world. By looking at various countries, we aimed to explore the variety and variability in the technical attributes of COVID-19 tracing apps. The nationally commissioned apps from the following countries (in alphabetical order) constituted the purposeful sample: Brazil, China, Finland, France, Germany, Italy, Singapore, South Korea, Spain, and the United Kingdom (England and Wales). These countries reflect different regions that had different experiences with the COVID-19 outbreak and the containment of the pandemic. We also expected different technological approaches across the selected countries.

Figure 1. Sequence of methodological steps.
View this figure

Summary of Categories

After the initial screening of the sources, several clusters of information needs emerged. These were refined and summarized into five categories (Textbox 2). These categories were then used to derive the final item set.

In the following sections, the information we collected for each category is presented. These data are based on our review of the following apps: Corona Warn App (Germany), TousAntiCovid (France), Immuni (Italy), Radar COVID (Spain), Koronavilkku (Finland), NHS (National Health Service) COVID-19 (England and Wales), TraceTogether (Singapore), Self-Diagnosis app (South Korea), Self-Quarantine app (South Korea), and Coronavirus-SUS (Sistema Único de Saúde; Brazil). Of note, China has no single app and instead uses various other apps (eg, WeChat) that have integrated tracing solutions. The order of this list mirrors the availability of information (most to least available) in the 10 selected countries.

Categories and related information needs.

Background information

  • Open-source code
  • How transparent is the development process?
  • Is information regarding the development process and the app disseminated? If yes, how is it done?

Purpose and workflow

  • Is the exact position of users being tracked?
  • Can health authorities contact users via the app?
  • Is additional data gathered (eg, for epidemiological analysis)?

Technical information

  • Which protocol is implemented?
  • Which tracing technology is used?
  • Is the Google/Apple Exposure Notification System used?
  • Is it interoperable with other apps?

Privacy protection

  • Is a centralized or decentralized approach used?
  • Can the gathered data be deleted?
  • Does the user have to provide any personal information?
  • Is the use of the app mandatory (eg, access controls)?

Availability and use

  • When was the app released?
  • How often was the app downloaded?
Textbox 2. Categories and related information needs.

Background Information

After investigating the first category (ie, background information), we found that the provision of primary background information by the issuers of the apps (Table 1) was essential for assessing the issuers’ intentions for the app and the apps’ implications for data privacy. The apps of Germany [41], France [24], Italy [38], Spain [65], Finland [66], England and Wales [67], and Singapore [23] have open-source repositories of their code bases and websites [32,34-37,42,68,69], which act as dedicated information hubs for citizens and tourists. Except for Singapore’s and South Korea’s apps, all countries’ apps were commissioned by their respective health and digital ministries. However, the apps were mainly developed, deployed, and maintained by the private industry sector. Singapore and South Korea have dedicated ministries that are capable of developing contact tracing apps with minor support from the private industry sector. Only the German development team from SAP SE (Systeme, Andwendungen, Produkte in der Datenverarbeitung, Societas Europaea) provided some form of insight into their app development process via a wide range of documents, including the scoping document, technical documentation, and an issue tracker that can be found in their publicly accessible GitHub repository [41]. There was no documentation found for the apps of Brazil, China, and South Korea (eg, documents with information about open-source code or dedicated websites). As such, various publicly available sources, like statements from press conferences, government-issued guidelines, coverage by traditional media, and social media, had to be searched to obtain information about these countries’ apps [28,48,55,57,60,70-73].

Table 1. Background information provided by the issuers of the apps.
AppCountryOpen-source repositoryInformation websites
Corona Warn AppGermanyYesYes
Radar COVIDSpainYesYes
NHSa COVID-19England and WalesYesYes
Self-Diagnosis appSouth KoreaNoNo
Self-Quarantine appSouth KoreaNoNo
Various integrated tracing functions in apps (eg, WeChat)ChinaNoNo

aNHS: National Health Service.

bSUS: Sistema Único de Saúde.

Purpose and Workflow

Our analysis of the second category (ie, the purpose of the app) showed that all of the apps that were included in this study were developed to help public health officials with tracing and containing the spread of SARS-CoV-2, as stated in the materials that were publicly accessible at the time of our search [26-28,31,32,34-37,42,48,57,60,65,68,72,74,75]. Except for the apps of China and South Korea, all apps notify their users about whether they came into contact with another user who tested positive for COVID-19. China’s app informs its users via a 3-color code system (green, yellow, or red) and uses a quick response (QR) code that can be scanned by authorities and businesses. Green indicates a person who has unrestricted movement, yellow indicates a person who must undergo a 7-day home quarantine, and red indicates a patient with confirmed COVID-19 who needs to be quarantined for 14 days. In this system, it is not clear as to how these codes are assigned and removed [55,60,71]. South Korea uses a slightly different approach compared to those of the other countries. Most apps automatically match possible COVID-19 contact events. However, South Korea allows its users to self-report their health status with the Self-Diagnosis app. If the reported symptoms are indicative of a SARS-CoV-2 infection, health authorities can intervene. If a person tests positive for COVID-19, a case officer from the local government is assigned, and the person who tested positive must report their symptoms via another app called the “Self-Quarantine app.” If people with COVID-19 leave their designated quarantine areas, they and the case officers receive an alert [57].

Technical Information

With regard to the third category (ie, technical information), the Decentralized Privacy-Preserving Proximity Tracing (DP-3T) [49], Temporary Contact Numbers (TCN) [52], BlueTrace [22] and Pan-European Privacy-Preserving Proximity Tracing (PEPP-PT) [76] protocols were found to be used as reference implementations for the various apps (Table 2). DP-3T is used by almost all European countries except for France, which uses PEPP-PT with their own reference implementation for the Robust and Privacy-Preserving Proximity Tracing (ROBERT) protocol [51]. The protocols used for the apps of China and South Korea were not disclosed. The apps of Brazil, Finland, and England and Wales use the Google/Apple Exposure Notification System (GAEN) but do not provide any specific information about which reference protocol they used for app development. The DP-3T, TCN, BlueTrace, and ROBERT protocols all use Bluetooth-based proximity detection approaches. These Bluetooth-based proximity detection approaches use different mathematical models to calculate the likelihood of a potential exposure. The key parameters for these models are the time of being near other devices, the distance to other devices, and the results of an assessment for the contagiousness of a person with COVID-19. The sensitivity of these parameters can be adjusted by the issuer of the app, depending on the number of infections. South Korea relies on users’ self-reports, and when users are quarantined, the country relies on their GPS to alert health authorities about users leaving their quarantine zones [72]. China uses QR codes (to be able to access certain locations, these QR codes have to be scanned upon entry) and other undisclosed sensor data and metadata from smartphones [60,71,73]. Germany, Italy, Brazil, Spain, and England and Wales use the GAEN. The European Union has provided interoperability specifications for cross-border transmission chains between the approved apps that European Union states are planning to implement [29]. In October 19, 2020, the apps of Germany, Ireland, and Italy became interoperable with each other. This was made possible by the European interoperability gateway service, which will include other countries (eg, Finland) in the future [30,31]. Singapore and South Korea have dedicated ministries that are capable of developing contact tracing apps with only minor support from the private industry sector.

Table 2. Technical information of the apps.
AppCountryProtocolTracing technologyGAENaInteroperable with apps from other countries
Corona Warn AppGermanyDP-3Tb and TCNcBluetoothYesYes
Radar COVIDSpainDP-3TBluetoothYesYes
NHSf COVID-19England and WalesBluetoothYesYes
Self-Diagnosis appSouth KoreaReported dataNoNo
Self-Quarantine appSouth KoreaReported data and GPSNoNo
Various integrated tracing functions in apps (eg, WeChat)ChinaQuick response code, sensor data, and metadataNoNo

aGAEN: Google/Apple Exposure Notification System.

bDP-3T: Decentralized Privacy-Preserving Proximity Tracing.

cTCN: Temporary Contact Numbers.

dROBERT: Robust and Privacy-Preserving Proximity Tracing.

eNot available.

fNHS: National Health Service.

gSUS: Sistema Único de Saúde.

Privacy Protection

The different protocols have various implications with regard to the privacy of users (the fourth category). China and South Korea use fully centralized protocols and have no methods in place for anonymizing collected data (Table 3). The other apps from Germany, France, Italy, Spain, England and Wales, Singapore, and Brazil use different protocols but function similarly to each other. All countries’ apps use Bluetooth to perform tracing; they do not use any other methods of geolocation (eg, using GPS data). As such, the apps only record how long two devices (smartphones) come into contact but not where contacts occur. Singapore’s app requires mobile phone numbers to be registered. However, aside from this feature, none of the apps collect any user-identifying data and only exchange alternating IDs via Bluetooth. If the GAEN is used, the tracing and identification of contact events with people who test positive for COVID-19 are locally conducted on users’ smartphones in a decentralized manner. The apps of Singapore and France also trace contacts locally on smartphones via user logs. However, user logs must be uploaded to the centralized servers of health authorities for report processing. These servers identify contact events with people who test positive for COVID-19 and send warning messages to users. Additionally, Singapore has combined the function of its contact tracing app, TraceTogether, with the function of the SafeEntry check-in system. In this system, a QR code that contains users’ contact information must be scanned before entering a location. This information is then uploaded to a government server. England and Wales have also implemented a similar venue check-in function directly into their app.

In the decentralized approach of Germany, Italy, Spain, Finland, England and Wales, and Brazil, user logs never leave the smartphone. However, if app users from these countries test positive for COVID-19, they can enable their apps to upload a key to the central server of health authorities. This key is then sent to all devices, and health authorities can subsequently derive device IDs and check if they match one of the encounters in the user log. The use of contact tracing apps in Germany, France, Italy, Spain, Finland, England and Wales, Brazil, and Singapore is completely voluntary. Furthermore, the apps, along with their data, can be removed and deleted at any time.

Table 3. Privacy protection.
AppCountryContact tracing approachReport processing approachUser can opt outAnonymity (no user registration)
Corona Warn AppGermanyDecentralizedDecentralizedYesYes
Radar COVIDSpainDecentralizedDecentralizedYesa
NHSb COVID-19England and WalesDecentralizedDecentralized (tracing)/centralized (check-ins)YesYes
TraceTogetherSingaporeDecentralizedCentralizedYesPhone number
Self-Diagnosis appSouth KoreaCentralizedNoName, address, and phone number
Self-Quarantine appSouth KoreaCentralizedNoName, address, and phone number
Various integrated tracing functions in apps (eg, WeChat)ChinaCentralizedCentralizedNoNo

aNot available.

bNHS: National Health Service.

cSUS: Sistema Único de Saúde.

Availability and Use

With regard to the fifth category (ie, availability and use), the use of contact tracing apps first occurred in Asian countries, starting with China [70] (integrated tracing functions in existing apps like WeChat or Alipay) and South Korea [28] (Self-Diagnosis app and Self-Quarantine app) in February 2020 and Singapore [77] (TraceTogether) in March 2020. Italy [74] (Immuni), France [75] (their app originally launched as StopCovid and relaunched as TousAntiCovid in October 2020 [78]), and Germany [27] (Corona Warn App) all released their contact tracing apps in June 2020. Brazil expanded the function of its app [48] (Coronavirus-SUS) so that citizens could seek information about COVID-19 (eg, noticeable fake news and recent outbreak locations) and report possible COVID-19 symptoms to check for a potential SARS-CoV-2 infection; a tracing function was added in August 2020. Most of the autonomous communities in Spain adopted the Radar COVID app after COVID-19 testing ended in other regions at the end of August 2020 [79]. Finland also released its app at the end of August 2020 [33]. After a failed attempt at developing a contact tracing app (a result of privacy issues), England and Wales abandoned its first app development approach and released a new app in September 2020. This app also used the GAEN [25,26,54].

Although most countries in our sample regularly reported download numbers, Brazil did not provide regular reports on recent download numbers (Table 4). There are also no comparable download numbers available for China and South Korea, as China integrated their tracing functions into popular existing apps and South Korea used a different tracing system in which not all citizens continuously use one app. For South Korea however, download numbers are available from the Google Play Store; Apple does not report on the number of downloads.

Table 4. Availability and use.
AppCountryRelease dateDownloads, n (% of population)Date reportedSource type
Corona Warn AppGermanyJune 16, 202025.4 million (30.6%)February 11, 2021Official [40]
TousAntiCovidFranceJune 2, 202011 million (16.4%)December 8, 2020Official [43]
ImmuniItalyJune 15, 202010.3 million (17.06%)February 22, 2021Official [44]
Radar COVIDSpainAugust 20207.03 million (17%)February 14, 2021Official [45]
KoronavilkkuFinlandAugust 31, 2020>2.5 million (45.3%)November 5, 2021Official [46]
NHSa COVID-19England and WalesSeptember 24, 202021.8 million (36.7%)February 10, 2021Official [47]
TraceTogetherSingaporeMarch 20, 20204.2 million (73.7%)February 23, 2021Official [34]
Self-Diagnosis appSouth KoreaMarch 2020>500,000 (0.96%)February 23, 2021Google Play Store [61]
Self-Quarantine appSouth KoreaMarch 2020>500,000 (0.96%)February 23, 2021Google Play Store [62]
Coronavirus-SUSbBrazilAugust 1, 202010 million (4.77%)December 8, 2020Official [48] and press [63]
Various integrated tracing functions in apps (eg, WeChat)ChinaFebruary 2020c

aNHS: National Health Service.

bSUS: Sistema Único de Saúde.

cNot available.

Evaluation Framework

Conducting research on the collected materials and the five categories allowed us to create more specific subcategories and derive a potential item set that was suitable for describing and monitoring nationally coordinated tracing apps (Table 5). As more information was found through the OSINT-based search, the items were phrased in a more detailed manner. This proposed item set for a monitoring framework was applied to the apps of the 10 countries that were analyzed in this study (Multimedia Appendix 1). In total, 19 items were applied to the 11 apps of the 10 countries. This yielded 209 cells of information in the matrix. However, data for 22 cells were not publicly available. These 22 cells were mainly related to information from China, but they were also somewhat related to information from South Korea, Brazil, Spain, and England and Wales. The items that were missing information were primarily related to the protocols that countries used and data privacy issues, particularly those concerning conformance with data protection regulations and the secondary use of data (other purposes).

Table 5. Proposed item set for a monitoring framework.
Category, subcategory, and item numberItem
Background information


Q1.1What is the name of the app?

Open-source code

Q1.2Is there a publicly accessible repository of the source code?

Q1.3Is the published source code up to date?

Public information

Q1.4Is there some form of material (eg, website, guideline, or Frequently Asked Questions page) for informing the public about the app?


Q1.5Which institutions worked together to develop, host, and maintain the app?
Purpose and workflow

Warning process design

Q2.1What is the main mode of action for warning app users?

Secondary data use

Q2.2What other purposes are the data used for?
Technical information


Q3.1Which tracing protocols (eg, DP-3Ta, TCNb, ROBERTc, and BlueTrace) are implemented in the app?

Tracing technology

Q3.2Which tracing technology (eg, Bluetooth, GPS, and barcodes) is used by the app?

Exposure notification system

Q3.3Is the Google/Apple Exposure Notification System used?


Q3.4Is the app actively interoperable with apps from other countries?
Privacy protection

Entity of trust

Q4.1Is the process of contact tracing (eg, tracking each contact event) centralized or decentralized? Provide a short description of the workflow.

Q4.2Is the report processing approach (eg, matching contact events and informing the user) centralized or decentralized? Provide a short description of the workflow.

Q4.3Are app data processed as mandated by data protection regulations?

Q4.4Is the data automatically destroyed after a fixed period of time?

Q4.5Can the user opt out?


Q4.6Does the user have to register any information (eg, mobile phone number, name, address, or date of birth)?
Availability and use

Release date

Q5.1On which date could the app be downloaded by the public?

Number of downloads

Q5.2Are there officially reported download numbers?

aDP-3T: Decentralized Privacy-Preserving Proximity Tracing.

bTCN: Temporary Contact Numbers.

cROBERT: Robust and Privacy-Preserving Proximity Tracing.

Principal Results

COVID-19 tracing apps have become instruments in the arsenal of measures for fighting the spread of the disease. Many countries have adopted these instruments in their national pandemic management plans. This study therefore aimed to explore and identify the core features of COVID-19 tracing apps in order to provide a generic framework for conducting cross-country comparisons and monitoring apps over time. We found and screened publicly available information via an OSINT approach for analyzing selected subject areas, which served as initial entry points for collecting data. We then defined five categories to structure the evaluation framework. These categories were based on recurring information, and information for the framework was selected based on obtaining enough publicly available data. The categories were as follows: background information, purpose and workflow, technical information, privacy protection, and availability and use. Based on these categories, we constructed a set of specific items that could be used to evaluate the core features of COVID-19 tracing apps. In order to showcase the item set’s usefulness, it was applied to 10 countries that nationally commissioned COVID-19 tracing apps (ie, Brazil, China, France, Germany, Italy, Spain, Finland, Singapore, South Korea, and the United Kingdom [England and Wales]). Our comparison revealed differences among each countries’ apps, especially with regard to the centralization of the entity of trust and the overall transparency of the apps’ technical makeup. The proposed item set will help researchers evaluate the spread and use of contact tracing apps within and across countries in the future.

Key Characteristics for Evaluating COVID-19 Tracing Apps

The application of OSINT in the emerging field of COVID-19 tracing apps heavily depends on the public availability of essential and comprehensive background information. Ideally, such information can be retrieved from an official website that informs the public and provides technical background information. Information on app development is even more useful, as open-source projects provide detailed insights and involve unrestricted code audits. In open-source projects, the source code and the entire project description are often transparently accessible. Detailed insights on app protocols and their implementation are interesting from both a technical and economic point of view (eg, increasing public acceptance); transparency likely facilitates trust, which is associated with increased public acceptance [6,7]. Our analysis revealed varying degrees of transparency across countries, as some countries disclosed most of the abovementioned information (Germany, France, Italy, Spain, the United Kingdom, and Singapore), while others provided noticeably less information (Brazil, China and South Korea).

As the motivation to download and use an app is the ultimate predictor of an app’s success, information about download numbers, acceptance, and use are the most crucial for differentiating between successful and unsuccessful app designs. According to our initial findings, relative app download numbers were highest in Singapore, Finland, the United Kingdom, and Germany. Although download numbers are indicators of motivation, they do not necessarily reveal information about an app’s actual use [17]. App use can only be measured centrally by certain methods. For instance, Germany’s COVID-19 tracing app analyzes the total number of shared negative and positive test results [50]. However, the characteristics described in our framework could be used as possible predictors of app adoption rates in cross-country adoption research.

An app’s technical design is essential not only for assessing its usefulness, but also for preserving privacy. In particular, the decision to use Bluetooth, GPS, or other means (eg, QR code scanning with corresponding protocols) determines whether the app serves the purpose of tracing or location tracking. The latter could potentially result in greater privacy concerns. Thus, tracking is unlikely to be accepted in many countries. On the other hand, several COVID-19 tracing app users have expressed their interest in obtaining more detailed information about close encounters with people with COVID-19 after receiving rather superficial information from their Bluetooth-based tracing apps [59]. This shows that COVID-19 tracing app developers face the challenge of striking the right balance between maximizing privacy preservation and maximizing usefulness (ie, the range of functionalities). It has yet to be determined which composition of app traits works best in which cultures. The use of Bluetooth is also closely related to the use of the GAEN. The GEAN system seems to be indispensable, as tracing apps that do not use it cannot run in the background of Apple/Android phones. Furthermore, many countries have shifted toward using the GAEN [58]. However, in the future, it might be desirable to find technical solutions for overcoming public health agencies’ and governments’ dependency on Google and Apple.

Another important technical feature in our item set is interoperability, which refers to an app’s ability to operate in a synchronized manner with the COVID-19 tracing apps of other countries. This has become increasingly important, given the likely resurgence of international travel when the pandemic starts to recede. Du and colleagues [80] have stated that the risk of creating a useless Tower of Babel of contact tracing apps is very real, as contact tracing apps’ inability to work across different countries renders them ineffective. Researchers should therefore identify which apps allow for cross-country use. This is particularly interesting to people who live in regions that neighbor other countries, which is quite common in Europe.

The acceptance of COVID-19 tracing apps hinges on both its perceived benefits and perceived barriers (eg, privacy concerns) [8]. The basic and technical information that are outlined in this study lay the foundation for privacy preservation. However, such information must be supplemented with data on a range of additional factors that relate to the entity of trust and anonymity. With regard to the entity of trust, we included indicators that can be used to determine whether a central or decentral data processing method is used. We found that this distinction could and should be determined for contact tracing approaches and report processing approaches. Centrality is a particularly critical factor, as the providers of COVID-19 tracing apps are often governmental agencies, and many users might be reluctant to entrust their government with the exclusive handling of health-related data. In our sample, South Korea and China stood out because they opted for approaches with higher degrees of centrality. Furthermore, the aspects of centrality and a range of other well-known criteria for privacy preservation need to be considered. Therefore, we included automatic data deletion after a fixed period of time, the possibility to opt out, and the need to refrain from storing additional personal data (eg, mobile phone numbers) into our item set.


Several limitations apply to our study. First and foremost, the OSINT approach depends on the public availability of reliable and trustworthy data and information. Second, the great majority of items could be answered for most, but not all, countries. This could have been due to missing information or language issues. Incorporating the insights of experts from the studied countries could have provided us with additional information and a more nuanced picture of app use/development. However, the approach we chose is more feasible, and its pragmatic nature allows for the flexible incorporation of various information sources in a quickly evolving field. Third, we developed the item set by analyzing a limited number of countries, and we cannot exclude the possibility that additional criteria would have emerged if we included additional countries in the analysis.

Conclusion and Outlook

Ours is one of the first studies to provide a set of criteria for evaluating nationally commissioned COVID-19 tracing apps and to apply such criteria to 10 industrialized countries. Although cross-country comparisons have been previously conducted [6,9], our study provides a more comprehensive yet relatively easy-to-apply evaluation framework that uses various technical factors from publicly available sources as potential determinants of app adoption.

The evaluation of COVID-19 tracing apps is crucial for the assessment of factors that facilitate these apps’ widespread acceptance and usefulness within and across countries. The more people who accept and use the app, the better the virus can be contained. This has been demonstrated by simulation studies [4]. Given that transparency and privacy protection are crucial for building people’s trust in apps, the technical features that are proposed in our framework might play an important role in promoting widespread app adoption. The initial application of our framework to the 10 countries in this study revealed differences in countries’ app adoption rates. Based on the app download numbers, Finland, Germany, and the United Kingdom were more successful in deploying their apps because they chose transparent and decentralized contact tracing and report processing approaches. However, Singapore was the only city-state in our sample, and it had the highest app adoption rates.

The associations between technical features and app success must be addressed more thoroughly in future research. This can be done by analyzing a greater number of countries, and our framework provides the groundwork to do so. We made the item set very concise so that it can be easily shared with scientists in other countries via a collaborative approach or used to survey people in other countries via a crowdsourcing platform that is similar to that of Trang et al [5]. At the end of the pandemic, it will be interesting to see the role that apps actually played in the fight against the pandemic across different countries (ie, outside of those included in this study). Our framework will help studies that aim to evaluate apps’ contributions to the overall management of pandemics, by providing core descriptors of COVID-19 warning apps that can be used as predictors.

Insights on the success factors of tracing apps might prove useful for designing general national health apps. Over time, tracing apps might be combined with health record apps, as there have already been discussions and instances of developers expanding their apps’ features to include contact and symptom diaries, vaccination certificates, and educational resources for public health messaging.


This study was funded by the State of Lower Saxony, Germany (grant number: ZN 3062).

Authors' Contributions

UH initiated the study. JPW specified the theoretical framework, was supported by UH and ME, and conceptualized the research design. JPW collected and prepared the data with the support of ME, under the supervision of UH. All authors prepared the manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Applied monitoring framework.

XLSX File (Microsoft Excel File), 20 KB

  1. Furukawa NW, Brooks JT, Sobel J. Evidence supporting transmission of severe acute respiratory syndrome coronavirus 2 while presymptomatic or asymptomatic. Emerg Infect Dis 2020 Jul;26(7):e201595 [FREE Full text] [CrossRef] [Medline]
  2. Cen Y, Chen X, Shen Y, Zhang XH, Lei Y, Xu C, et al. Risk factors for disease progression in patients with mild to moderate coronavirus disease 2019-a multi-centre observational study. Clin Microbiol Infect 2020 Sep;26(9):1242-1247 [FREE Full text] [CrossRef] [Medline]
  3. Ferretti L, Wymant C, Kendall M, Zhao L, Nurtay A, Abeler-Dörner L, et al. Quantifying SARS-CoV-2 transmission suggests epidemic control with digital contact tracing. Science 2020 May 08;368(6491):eabb6936 [FREE Full text] [CrossRef] [Medline]
  4. Hinch R, Probert W, Nurtay A, Kendall M, Wymant C, Hall M, et al. Effective configurations of a digital contact tracing app: A report to NHSX. en. GitHub. 2020.   URL: [accessed 2021-03-22]
  5. Trang S, Trenz M, Weiger WH, Tarafdar M, Cheung CMK. One app to trace them all? Examining app specifications for mass acceptance of contact-tracing apps. Eur J Inf Syst 2020 Jul 27;29(4):415-428 [FREE Full text] [CrossRef]
  6. Altmann S, Milsom L, Zillessen H, Blasone R, Gerdon F, Bach R, et al. Acceptability of app-based contact tracing for COVID-19: Cross-country survey study. JMIR Mhealth Uhealth 2020 Aug 28;8(8):e19857 [FREE Full text] [CrossRef] [Medline]
  7. von Wyl V, Höglinger M, Sieber C, Kaufmann M, Moser A, Serra-Burriel M, et al. Drivers of acceptance of COVID-19 proximity tracing apps in Switzerland: panel survey analysis. JMIR Public Health Surveill 2021 Jan 06;7(1):e25701 [FREE Full text] [CrossRef] [Medline]
  8. Walrave M, Waeterloos C, Ponnet K. Adoption of a contact tracing app for containing COVID-19: A health belief model approach. JMIR Public Health Surveill 2020 Sep 01;6(3):e20572 [FREE Full text] [CrossRef] [Medline]
  9. O'Neill PH, Ryan-Mosley T, Johnson B. A flood of coronavirus apps are tracking us. Now it’s time to keep track of them. MIT Technology Review. 2020 May 07.   URL: [accessed 2021-02-12]
  10. Ming LC, Untong N, Aliudin NA, Osili N, Kifli N, Tan CS, et al. Mobile health apps on COVID-19 launched in the early days of the pandemic: Content analysis and review. JMIR Mhealth Uhealth 2020 Sep 16;8(9):e19796 [FREE Full text] [CrossRef] [Medline]
  11. Levine DM, Co Z, Newmark LP, Groisser AR, Holmgren AJ, Haas JS, et al. Design and testing of a mobile health application rating tool. NPJ Digit Med 2020 May 21;3:74 [FREE Full text] [CrossRef] [Medline]
  12. Nouri R, Kalhori SRN, Ghazisaeedi M, Marchand G, Yasini M. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inform Assoc 2018 Aug 01;25(8):1089-1098 [FREE Full text] [CrossRef] [Medline]
  13. Llorens-Vernet P, Miró J. Standards for mobile health-related apps: Systematic review and development of a guide. JMIR Mhealth Uhealth 2020 Mar 03;8(3):e13057 [FREE Full text] [CrossRef] [Medline]
  14. Khowaja K, Al-Thani D. New checklist for the heuristic evaluation of mHealth apps (HE4EH): Development and usability study. JMIR Mhealth Uhealth 2020 Oct 28;8(10):e20353 [FREE Full text] [CrossRef] [Medline]
  15. van Haasteren A, Vayena E, Powell J. The mobile health app trustworthiness checklist: Usability assessment. JMIR Mhealth Uhealth 2020 Jul 21;8(7):e16844 [FREE Full text] [CrossRef] [Medline]
  16. Lee S, Shon T. Open source intelligence base cyber threat inspection framework for critical infrastructures. In: 2016 Future Technologies Conference (FTC). 2016 Presented at: 2016 Future Technologies Conference (FTC); December 6-7, 2016; San Francisco, CA, USA p. 1030-1033. [CrossRef]
  17. Riillo CAF, Peroni C, Sarracino F. Determinants of acceptability of contact tracing apps for COVID-19: Initial results from Luxembourg. Économie et statistiques: Working papers du STATEC. 2020 Dec.   URL: [accessed 2021-03-22]
  18. Baumgärtner L, Dmitrienko A, Freisleben B, Gruler A, Höchst J, Kühlberg J, et al. Mind the GAP: Security & privacy risks of contact tracing apps. In: 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). 2020 Presented at: 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom); December 29, 2020, to January 1, 2021; Guangzhou, China.
  19. von Wyl V, Bonhoeffer S, Bugnion E, Puhan MA, Salathé M, Stadler T, et al. A research agenda for digital proximity tracing apps. Swiss Med Wkly 2020 Jul 13;150:w20324 [FREE Full text] [CrossRef] [Medline]
  20. Li J, Guo X. Global deployment mappings and challenges of contact-tracing apps for COVID-19. SSRN. Preprint posted online on May 24, 2020 [FREE Full text] [CrossRef]
  21. Ahmed N, Michelin RA, Xue W, Ruj S, Malaney R, Kanhere SS, et al. A Survey of COVID-19 Contact Tracing Apps. IEEE Access 2020 Jul 20;8:134577-134601 [FREE Full text] [CrossRef]
  22. BlueTrace Protocol. BlueTrace.   URL: [accessed 2020-11-09]
  23. OpenTrace. GitHub.   URL: [accessed 2020-11-05]
  24. TousAntiCovid sources. GitLab.   URL: [accessed 2020-11-05]
  25. Next phase of NHS coronavirus (COVID-19) app announced 2020. Department of Health and Social Care. 2020 Jun 18.   URL: [accessed 2020-08-18]
  26. NHS COVID-19 app launches across England and Wales: NHS COVID-19 app launches to help control coronavirus (COVID-19) transmission. Department of Health and Social Care. 2020 Sep 24.   URL: [accessed 2020-10-20]
  27. Veröffentlichung der Corona-Warn-App 2020. Die Bundesregierung. 2020 Jun 16.   URL: https:/​/www.​​breg-de/​themen/​coronavirus/​veroeffentlichung-der-corona-warn-app-1760892 [accessed 2020-08-17]
  28. (Important) Installation of COVID-19 Self-diagnosis Mobile App when you visit Korea 2020. Embassy of the Republic of Korea in the Republic of Kenya. 2020 Mar 18.   URL: [accessed 2020-08-17]
  29. Corona-Warn-Apps sollen grenzüberschreitend funktionieren. Europäische Kommission. 2020 Jun 16.   URL: [accessed 2020-08-18]
  30. Mobile contact tracing apps in EU Member States. European Commission.   URL: https:/​/ec.​​info/​live-work-travel-eu/​coronavirus-response/​travel-during-coronavirus-pandemic/​mobile-contact-tracing-apps-eu-member-states_en [accessed 2021-01-11]
  31. Coronavirus: EU interoperability gateway goes live, first contact tracing and warning apps linked to the system. European Commission. 2020 Oct 19.   URL: [accessed 2020-10-21]
  32. Koronavilkku. Finnish Institute for Health and Welfare.   URL: [accessed 2021-02-11]
  33. Koronavilkku has been downloaded a million times already!. Finnish Institute for Health and Welfare. 2020 Sep 01.   URL: [accessed 2021-02-11]
  34. TraceTogether. Government of Singapore.   URL: [accessed 2020-11-05]
  35. SafeEntry - National digital check-in system. Government of Singapore.   URL: [accessed 2020-11-05]
  36. Radar covid19. Government of Spain.   URL: [accessed 2020-11-05]
  37. Immuni - Sito Ufficiale. Presidenza del Consiglio dei Ministri.   URL: [accessed 2020-11-05]
  38. Immuni - Commissario straordinario per l'emergenza Covid-19. GitHub.   URL: [accessed 2020-11-05]
  39. Infektionsketten digital unterbrechen mit der Corona-Warn-App: Die Corona-Warn-App ist ein wichtiger Baustein der Pandemiebekämpfung. Robert Koch Institut.   URL: [accessed 2020-08-18]
  40. Übersicht zu aktuellen und früherer Zahlen und Fakten zur Corona-Warn-App. Robert Koch Institute.   URL: [accessed 2021-02-11]
  41. Corona-Warn-App. GitHub.   URL: [accessed 2020-11-05]
  42. Corona-Warn-App Open Source Project. The authors of the Corona-Warn-App open-source project.   URL: [accessed 2020-11-05]
  43. Application TousAntiCovid millions de téléchargements. République Française. 2020 Dec 08.   URL: [accessed 2021-02-24]
  44. Immuni - I numeri di Immuni. Presidenza del Consiglio dei Ministri.   URL: [accessed 2021-02-24]
  45. Descargas Radar COVID. Gobierno de España.   URL: [accessed 2021-02-24]
  46. Finnish IFH. Koronavilkku has been downloaded more than 2.5 million times – widespread use increases the app's effectiveness. Finnish Institute for Health and Welfare. 2020 Nov 05.   URL: https:/​/thl.​fi/​en/​web/​thlfi-en/​-/​koronavilkku-has-been-downloaded-more-than-2.​5-million-times-widespread-use-increases-the-app-s-effectiveness [accessed 2021-02-24]
  47. What is the latest data about NHS COVID-19 app downloads? National Health Service England.   URL: [accessed 2021-02-24]
  48. Penido A. Aplicativo Coronavírus-SUS vai alertar contatos próximos de pacientes com COVID-19. Fiocruz. 2020 Aug 11.   URL: https:/​/www.​​index.php/​br/​noticias/​1923-aplicativo-coronavirus-sus-vai-alertar-contatos-proximos-de-pacientes-com-covid-19 [accessed 2020-08-18]
  49. DP^3T: Decentralized Privacy-Preserving Proximity Tracing. GitHub.   URL: [accessed 2020-11-09]
  50. Diagnosis key analysis (dka). GitHub.   URL: [accessed 2021-01-08]
  51. ROBERT -- ROBust and privacy-presERving proximity Tracing protocol. GitHub.   URL: [accessed 2020-11-09]
  52. TCN Coalition and LFPH have merged. Linux Foundation Public Health.   URL: [accessed 2020-11-09]
  53. Cho H, Ippolito D, Yu YW. Contact tracing mobile apps for COVID-19: Privacy considerations and related trade-offs. ArXiv. Preprint posted online on March 30, 2020 [FREE Full text]
  54. Cellan-Jones R, Kleinman Z. NHS Covid-19 app: 12m downloads - and lots of questions. BBC News Service. 2020 Sep 28.   URL: [accessed 2020-10-20]
  55. Davidson H. China's coronavirus health code apps raise concerns over privacy. The Guardian. 2020 Apr 01.   URL: https:/​/www.​​world/​2020/​apr/​01/​chinas-coronavirus-health-code-apps-raise-concerns-over-privacy [accessed 2020-08-17]
  56. Dillet R. French contact-tracing app StopCovid has been activated 1.8 million times but only sent 14 notifications. TechCrunch. 2020 Jun 23.   URL: [accessed 2020-08-18]
  57. Kim M. South Korea is watching quarantined citizens with a smartphone app. MIT Technology Review. 2020 Mar 06.   URL: https:/​/www.​​2020/​03/​06/​905459/​coronavirus-south-korea-smartphone-app-quarantine/​ [accessed 2020-08-17]
  58. Sato M. Why some countries suspended, replaced, or relaunched their covid apps. MIT Technology Review. 2020 Dec 23.   URL: https:/​/www.​​2020/​12/​23/​1015557/​covid-apps-contact-tracing-suspended-replaced-or-relaunched/​ [accessed 2021-01-08]
  59. Welche dieser Probleme sind Ihnen bereits bei der Nutzung der Corona-Warn-App begegnet? Wirtschaftswoche.   URL: https:/​/civey.​com/​umfragen/​11258/​welche-dieser-probleme-sind-ihnen-bereits-bei-der-nutzung-der-corona-warn-app-begegnet [accessed 2021-01-08]
  60. Xinhua NA. I've seen Hangzhou at 4 a.m., just for the early "no code" - "health code" behind the return to work. Xinhua News Agency. 2021 Feb 20.   URL: [accessed 2020-08-17]
  61. Quarantine-Report • Self-Check. Google Play.   URL: [accessed 2021-02-24]
  62. 자가격리자 안전보호. Google Play.   URL: [accessed 2021-02-24]
  63. Butcher I. App Coronavírus-SUS supera 10 milhões de downloads e 1,8 mil registros de exames positivos. 2020 Dec 08.   URL: https:/​/www.​​noticias/​08/​12/​2020/​coronavirus-sus-app-tem-10-milhoes-de-downloads-e-18-mil-registros-de-exames-positivos-para-covid-19/​ [accessed 2021-02-24]
  64. Krippendorff K. Content Analysis: An Introduction to Its Methodology Fourth Edition. Los Angeles, California: SAGE Publishing; 2018.
  65. Radar COVID. GitHub.   URL: [accessed 2020-11-05]
  66. Finnish Institute for Health and Welfare (THL). GitHub.   URL: [accessed 2021-02-11]
  67. NHSX. GitHub.   URL: [accessed 2020-11-05]
  68. TousAntiCovid. Ministère de l'Économie des Finances et de la Relance. 2020 Oct 22.   URL: [accessed 2020-11-05]
  69. The NHS COVID-19 app support website. National Health Service England.   URL: [accessed 2020-11-05]
  70. China launches coronavirus 'close contact detector' app. BBC News Service. 2020 Feb 11.   URL: [accessed 2020-08-17]
  71. Mehta I. China’s coronavirus detection app is reportedly sharing citizen data with police. The Next Web. 2020.   URL: [accessed 2020-08-17]
  72. Self-isolated safety protection app wonder tapa Q&A 7. Ministry of Public Administration and Security. 2020 Mar 05.   URL: [accessed 2020-08-18]
  73. Mozur P, Zhong R, Krolik A. In coronavirus fight, China gives citizens a color code, with red flags. The New York Times. 2020 Mar 01.   URL: [accessed 2020-08-18]
  74. Immuni contact-tracing App: available on all devices. Ministry of Health.   URL: [accessed 2020-08-17]
  75. StopCovid: the project. Ministry of the Economy and Finance. 2020.   URL: [accessed 2020-08-17]
  76. Pan-European Privacy-Preserving Proximity Tracing. PEPP-PT Initiative.   URL: [accessed 2020-08-04]
  77. Bay J. TraceTogether - One month on. Medium. 2020 Apr 20.   URL: [accessed 2020-08-17]
  78. Info Coronavirus COVID-19 - Application Tousanticovid.   URL: [accessed 2021-02-24]
  79. Mitchell D. Radar COVID contact tracing app is now active throughout Madrid. EuroWeekly. 2020 Oct 08.   URL: https:/​/www.​​2020/​10/​08/​radar-covid-contact-tracing-app-is-now-active-throughout-madrid/​ [accessed 2020-10-20]
  80. Du L, Raposo VL, Wang M. COVID-19 contact tracing apps: A technologic Tower of Babel and the gap for international pandemic control. JMIR Mhealth Uhealth 2020 Nov 27;8(11):e23194 [FREE Full text] [CrossRef] [Medline]

DP-3T: Decentralized Privacy-Preserving Proximity Tracing
GAEN: Google/Apple Exposure Notification System
NHS: National Health Service
OSINT: open-source intelligence approach
PEPP-PT: Pan-European Privacy-Preserving Proximity Tracing
QR: quick response
SAP SE: Systeme, Andwendungen, Produkte in der Datenverarbeitung, Societas Europaea
SUS: Sistema Único de Saúde
TCN: Temporary Contact Numbers

Edited by L Buis; submitted 18.01.21; peer-reviewed by T Winkler, M Pobiruchin, T Heponiemi; comments to author 10.02.21; revised version received 03.03.21; accepted 12.03.21; published 26.03.21


©Jan-Patrick Weiß, Moritz Esdar, Ursula Hübner. Originally published in JMIR mHealth and uHealth (, 26.03.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.