After the initial screening of the sources, several clusters of information needs emerged. These were refined and summarized into five categories (Textbox 2). These categories were then used to derive the final item set.

In the following sections, the information we collected for each category is presented. These data are based on our review of the following apps: Corona Warn App (Germany), TousAntiCovid (France), Immuni (Italy), Radar COVID (Spain), Koronavilkku (Finland), NHS (National Health Service) COVID-19 (England and Wales), TraceTogether (Singapore), Self-Diagnosis app (South Korea), Self-Quarantine app (South Korea), and Coronavirus-SUS (Sistema Único de Saúde; Brazil). Of note, China has no single app and instead uses various other apps (eg, WeChat) that have integrated tracing solutions. The order of this list mirrors the availability of information (most to least available) in the 10 selected countries.

After investigating the first category (ie, background information), we found that the provision of primary background information by the issuers of the apps (Table 1) was essential for assessing the issuers’ intentions for the app and the apps’ implications for data privacy. The apps of Germany [41], France [24], Italy [38], Spain [65], Finland [66], England and Wales [67], and Singapore [23] have open-source repositories of their code bases and websites [32,34-37,42,68,69], which act as dedicated information hubs for citizens and tourists. Except for Singapore’s and South Korea’s apps, all countries’ apps were commissioned by their respective health and digital ministries. However, the apps were mainly developed, deployed, and maintained by the private industry sector. Singapore and South Korea have dedicated ministries that are capable of developing contact tracing apps with minor support from the private industry sector. Only the German development team from SAP SE (Systeme, Andwendungen, Produkte in der Datenverarbeitung, Societas Europaea) provided some form of insight into their app development process via a wide range of documents, including the scoping document, technical documentation, and an issue tracker that can be found in their publicly accessible GitHub repository [41]. There was no documentation found for the apps of Brazil, China, and South Korea (eg, documents with information about open-source code or dedicated websites). As such, various publicly available sources, like statements from press conferences, government-issued guidelines, coverage by traditional media, and social media, had to be searched to obtain information about these countries’ apps [28,48,55,57,60,70-73].

Our analysis of the second category (ie, the purpose of the app) showed that all of the apps that were included in this study were developed to help public health officials with tracing and containing the spread of SARS-CoV-2, as stated in the materials that were publicly accessible at the time of our search [26-28,31,32,34-37,42,48,57,60,65,68,72,74,75]. Except for the apps of China and South Korea, all apps notify their users about whether they came into contact with another user who tested positive for COVID-19. China’s app informs its users via a 3-color code system (green, yellow, or red) and uses a quick response (QR) code that can be scanned by authorities and businesses. Green indicates a person who has unrestricted movement, yellow indicates a person who must undergo a 7-day home quarantine, and red indicates a patient with confirmed COVID-19 who needs to be quarantined for 14 days. In this system, it is not clear as to how these codes are assigned and removed [55,60,71]. South Korea uses a slightly different approach compared to those of the other countries. Most apps automatically match possible COVID-19 contact events. However, South Korea allows its users to self-report their health status with the Self-Diagnosis app. If the reported symptoms are indicative of a SARS-CoV-2 infection, health authorities can intervene. If a person tests positive for COVID-19, a case officer from the local government is assigned, and the person who tested positive must report their symptoms via another app called the “Self-Quarantine app.” If people with COVID-19 leave their designated quarantine areas, they and the case officers receive an alert [57].

With regard to the third category (ie, technical information), the Decentralized Privacy-Preserving Proximity Tracing (DP-3T) [49], Temporary Contact Numbers (TCN) [52], BlueTrace [22] and Pan-European Privacy-Preserving Proximity Tracing (PEPP-PT) [76] protocols were found to be used as reference implementations for the various apps (Table 2). DP-3T is used by almost all European countries except for France, which uses PEPP-PT with their own reference implementation for the Robust and Privacy-Preserving Proximity Tracing (ROBERT) protocol [51]. The protocols used for the apps of China and South Korea were not disclosed. The apps of Brazil, Finland, and England and Wales use the Google/Apple Exposure Notification System (GAEN) but do not provide any specific information about which reference protocol they used for app development. The DP-3T, TCN, BlueTrace, and ROBERT protocols all use Bluetooth-based proximity detection approaches. These Bluetooth-based proximity detection approaches use different mathematical models to calculate the likelihood of a potential exposure. The key parameters for these models are the time of being near other devices, the distance to other devices, and the results of an assessment for the contagiousness of a person with COVID-19. The sensitivity of these parameters can be adjusted by the issuer of the app, depending on the number of infections. South Korea relies on users’ self-reports, and when users are quarantined, the country relies on their GPS to alert health authorities about users leaving their quarantine zones [72]. China uses QR codes (to be able to access certain locations, these QR codes have to be scanned upon entry) and other undisclosed sensor data and metadata from smartphones [60,71,73]. Germany, Italy, Brazil, Spain, and England and Wales use the GAEN. The European Union has provided interoperability specifications for cross-border transmission chains between the approved apps that European Union states are planning to implement [29]. In October 19, 2020, the apps of Germany, Ireland, and Italy became interoperable with each other. This was made possible by the European interoperability gateway service, which will include other countries (eg, Finland) in the future [30,31]. Singapore and South Korea have dedicated ministries that are capable of developing contact tracing apps with only minor support from the private industry sector.

The different protocols have various implications with regard to the privacy of users (the fourth category). China and South Korea use fully centralized protocols and have no methods in place for anonymizing collected data (Table 3). The other apps from Germany, France, Italy, Spain, England and Wales, Singapore, and Brazil use different protocols but function similarly to each other. All countries’ apps use Bluetooth to perform tracing; they do not use any other methods of geolocation (eg, using GPS data). As such, the apps only record how long two devices (smartphones) come into contact but not where contacts occur. Singapore’s app requires mobile phone numbers to be registered. However, aside from this feature, none of the apps collect any user-identifying data and only exchange alternating IDs via Bluetooth. If the GAEN is used, the tracing and identification of contact events with people who test positive for COVID-19 are locally conducted on users’ smartphones in a decentralized manner. The apps of Singapore and France also trace contacts locally on smartphones via user logs. However, user logs must be uploaded to the centralized servers of health authorities for report processing. These servers identify contact events with people who test positive for COVID-19 and send warning messages to users. Additionally, Singapore has combined the function of its contact tracing app, TraceTogether, with the function of the SafeEntry check-in system. In this system, a QR code that contains users’ contact information must be scanned before entering a location. This information is then uploaded to a government server. England and Wales have also implemented a similar venue check-in function directly into their app.

In the decentralized approach of Germany, Italy, Spain, Finland, England and Wales, and Brazil, user logs never leave the smartphone. However, if app users from these countries test positive for COVID-19, they can enable their apps to upload a key to the central server of health authorities. This key is then sent to all devices, and health authorities can subsequently derive device IDs and check if they match one of the encounters in the user log. The use of contact tracing apps in Germany, France, Italy, Spain, Finland, England and Wales, Brazil, and Singapore is completely voluntary. Furthermore, the apps, along with their data, can be removed and deleted at any time.

With regard to the fifth category (ie, availability and use), the use of contact tracing apps first occurred in Asian countries, starting with China [70] (integrated tracing functions in existing apps like WeChat or Alipay) and South Korea [28] (Self-Diagnosis app and Self-Quarantine app) in February 2020 and Singapore [77] (TraceTogether) in March 2020. Italy [74] (Immuni), France [75] (their app originally launched as StopCovid and relaunched as TousAntiCovid in October 2020 [78]), and Germany [27] (Corona Warn App) all released their contact tracing apps in June 2020. Brazil expanded the function of its app [48] (Coronavirus-SUS) so that citizens could seek information about COVID-19 (eg, noticeable fake news and recent outbreak locations) and report possible COVID-19 symptoms to check for a potential SARS-CoV-2 infection; a tracing function was added in August 2020. Most of the autonomous communities in Spain adopted the Radar COVID app after COVID-19 testing ended in other regions at the end of August 2020 [79]. Finland also released its app at the end of August 2020 [33]. After a failed attempt at developing a contact tracing app (a result of privacy issues), England and Wales abandoned its first app development approach and released a new app in September 2020. This app also used the GAEN [25,26,54].

Although most countries in our sample regularly reported download numbers, Brazil did not provide regular reports on recent download numbers (Table 4). There are also no comparable download numbers available for China and South Korea, as China integrated their tracing functions into popular existing apps and South Korea used a different tracing system in which not all citizens continuously use one app. For South Korea however, download numbers are available from the Google Play Store; Apple does not report on the number of downloads.

Conducting research on the collected materials and the five categories allowed us to create more specific subcategories and derive a potential item set that was suitable for describing and monitoring nationally coordinated tracing apps (Table 5). As more information was found through the OSINT-based search, the items were phrased in a more detailed manner. This proposed item set for a monitoring framework was applied to the apps of the 10 countries that were analyzed in this study (Multimedia Appendix 1). In total, 19 items were applied to the 11 apps of the 10 countries. This yielded 209 cells of information in the matrix. However, data for 22 cells were not publicly available. These 22 cells were mainly related to information from China, but they were also somewhat related to information from South Korea, Brazil, Spain, and England and Wales. The items that were missing information were primarily related to the protocols that countries used and data privacy issues, particularly those concerning conformance with data protection regulations and the secondary use of data (other purposes).

COVID-19 tracing apps have become instruments in the arsenal of measures for fighting the spread of the disease. Many countries have adopted these instruments in their national pandemic management plans. This study therefore aimed to explore and identify the core features of COVID-19 tracing apps in order to provide a generic framework for conducting cross-country comparisons and monitoring apps over time. We found and screened publicly available information via an OSINT approach for analyzing selected subject areas, which served as initial entry points for collecting data. We then defined five categories to structure the evaluation framework. These categories were based on recurring information, and information for the framework was selected based on obtaining enough publicly available data. The categories were as follows: background information, purpose and workflow, technical information, privacy protection, and availability and use. Based on these categories, we constructed a set of specific items that could be used to evaluate the core features of COVID-19 tracing apps. In order to showcase the item set’s usefulness, it was applied to 10 countries that nationally commissioned COVID-19 tracing apps (ie, Brazil, China, France, Germany, Italy, Spain, Finland, Singapore, South Korea, and the United Kingdom [England and Wales]). Our comparison revealed differences among each countries’ apps, especially with regard to the centralization of the entity of trust and the overall transparency of the apps’ technical makeup. The proposed item set will help researchers evaluate the spread and use of contact tracing apps within and across countries in the future.

The application of OSINT in the emerging field of COVID-19 tracing apps heavily depends on the public availability of essential and comprehensive background information. Ideally, such information can be retrieved from an official website that informs the public and provides technical background information. Information on app development is even more useful, as open-source projects provide detailed insights and involve unrestricted code audits. In open-source projects, the source code and the entire project description are often transparently accessible. Detailed insights on app protocols and their implementation are interesting from both a technical and economic point of view (eg, increasing public acceptance); transparency likely facilitates trust, which is associated with increased public acceptance [6,7]. Our analysis revealed varying degrees of transparency across countries, as some countries disclosed most of the abovementioned information (Germany, France, Italy, Spain, the United Kingdom, and Singapore), while others provided noticeably less information (Brazil, China and South Korea).

As the motivation to download and use an app is the ultimate predictor of an app’s success, information about download numbers, acceptance, and use are the most crucial for differentiating between successful and unsuccessful app designs. According to our initial findings, relative app download numbers were highest in Singapore, Finland, the United Kingdom, and Germany. Although download numbers are indicators of motivation, they do not necessarily reveal information about an app’s actual use [17]. App use can only be measured centrally by certain methods. For instance, Germany’s COVID-19 tracing app analyzes the total number of shared negative and positive test results [50]. However, the characteristics described in our framework could be used as possible predictors of app adoption rates in cross-country adoption research.

An app’s technical design is essential not only for assessing its usefulness, but also for preserving privacy. In particular, the decision to use Bluetooth, GPS, or other means (eg, QR code scanning with corresponding protocols) determines whether the app serves the purpose of tracing or location tracking. The latter could potentially result in greater privacy concerns. Thus, tracking is unlikely to be accepted in many countries. On the other hand, several COVID-19 tracing app users have expressed their interest in obtaining more detailed information about close encounters with people with COVID-19 after receiving rather superficial information from their Bluetooth-based tracing apps [59]. This shows that COVID-19 tracing app developers face the challenge of striking the right balance between maximizing privacy preservation and maximizing usefulness (ie, the range of functionalities). It has yet to be determined which composition of app traits works best in which cultures. The use of Bluetooth is also closely related to the use of the GAEN. The GEAN system seems to be indispensable, as tracing apps that do not use it cannot run in the background of Apple/Android phones. Furthermore, many countries have shifted toward using the GAEN [58]. However, in the future, it might be desirable to find technical solutions for overcoming public health agencies’ and governments’ dependency on Google and Apple.

Another important technical feature in our item set is interoperability, which refers to an app’s ability to operate in a synchronized manner with the COVID-19 tracing apps of other countries. This has become increasingly important, given the likely resurgence of international travel when the pandemic starts to recede. Du and colleagues [80] have stated that the risk of creating a useless Tower of Babel of contact tracing apps is very real, as contact tracing apps’ inability to work across different countries renders them ineffective. Researchers should therefore identify which apps allow for cross-country use. This is particularly interesting to people who live in regions that neighbor other countries, which is quite common in Europe.

The acceptance of COVID-19 tracing apps hinges on both its perceived benefits and perceived barriers (eg, privacy concerns) [8]. The basic and technical information that are outlined in this study lay the foundation for privacy preservation. However, such information must be supplemented with data on a range of additional factors that relate to the entity of trust and anonymity. With regard to the entity of trust, we included indicators that can be used to determine whether a central or decentral data processing method is used. We found that this distinction could and should be determined for contact tracing approaches and report processing approaches. Centrality is a particularly critical factor, as the providers of COVID-19 tracing apps are often governmental agencies, and many users might be reluctant to entrust their government with the exclusive handling of health-related data. In our sample, South Korea and China stood out because they opted for approaches with higher degrees of centrality. Furthermore, the aspects of centrality and a range of other well-known criteria for privacy preservation need to be considered. Therefore, we included automatic data deletion after a fixed period of time, the possibility to opt out, and the need to refrain from storing additional personal data (eg, mobile phone numbers) into our item set.

Several limitations apply to our study. First and foremost, the OSINT approach depends on the public availability of reliable and trustworthy data and information. Second, the great majority of items could be answered for most, but not all, countries. This could have been due to missing information or language issues. Incorporating the insights of experts from the studied countries could have provided us with additional information and a more nuanced picture of app use/development. However, the approach we chose is more feasible, and its pragmatic nature allows for the flexible incorporation of various information sources in a quickly evolving field. Third, we developed the item set by analyzing a limited number of countries, and we cannot exclude the possibility that additional criteria would have emerged if we included additional countries in the analysis.

Ours is one of the first studies to provide a set of criteria for evaluating nationally commissioned COVID-19 tracing apps and to apply such criteria to 10 industrialized countries. Although cross-country comparisons have been previously conducted [6,9], our study provides a more comprehensive yet relatively easy-to-apply evaluation framework that uses various technical factors from publicly available sources as potential determinants of app adoption.

The evaluation of COVID-19 tracing apps is crucial for the assessment of factors that facilitate these apps’ widespread acceptance and usefulness within and across countries. The more people who accept and use the app, the better the virus can be contained. This has been demonstrated by simulation studies [4]. Given that transparency and privacy protection are crucial for building people’s trust in apps, the technical features that are proposed in our framework might play an important role in promoting widespread app adoption. The initial application of our framework to the 10 countries in this study revealed differences in countries’ app adoption rates. Based on the app download numbers, Finland, Germany, and the United Kingdom were more successful in deploying their apps because they chose transparent and decentralized contact tracing and report processing approaches. However, Singapore was the only city-state in our sample, and it had the highest app adoption rates.

The associations between technical features and app success must be addressed more thoroughly in future research. This can be done by analyzing a greater number of countries, and our framework provides the groundwork to do so. We made the item set very concise so that it can be easily shared with scientists in other countries via a collaborative approach or used to survey people in other countries via a crowdsourcing platform that is similar to that of Trang et al [5]. At the end of the pandemic, it will be interesting to see the role that apps actually played in the fight against the pandemic across different countries (ie, outside of those included in this study). Our framework will help studies that aim to evaluate apps’ contributions to the overall management of pandemics, by providing core descriptors of COVID-19 warning apps that can be used as predictors.

Insights on the success factors of tracing apps might prove useful for designing general national health apps. Over time, tracing apps might be combined with health record apps, as there have already been discussions and instances of developers expanding their apps’ features to include contact and symptom diaries, vaccination certificates, and educational resources for public health messaging.