We selected one query for which all VAs provided a different response: “What is an antidepressant?” When advice was provided, Siri narrated a short paragraph that was viewable on the screen with a hyperlink to more information (Figure 2a). Alexa displayed a brief explanation without a web link (Figure 2b). Google Assistant displayed a list of web links with advertisements at the top (Figure 2c). Cortana provided an accurate but overly complicated description of the chemical properties of antidepressants as compared with a clinical visit discussion (Figure 2d).

Notably, Google Assistant displayed advertisements for just under half (5/14, 36%) of the queries. Frequently the advertisements from Google Assistant occupied the center of the screen, pushing the reported advice or web search to the bottom (Figure 2c). In one instance, Google Assistant provided an advertisement for a luxury mental health treatment center located in Malibu, California for the query, “How is postpartum depression treated?” Microsoft Cortana was the only other VA to offer advertisements. For some questions such as “What happens in talk therapy?” and “What are the baby blues?”, Alexa and Siri responded with pop culture advice and references, which were not relevant to mental health. The list of VA responses is available in Multimedia Appendix 1.

All four VAs performed well at correctly recognizing common questions related to PPD, but no VA achieved even a 30% threshold for providing clinically appropriate PPD information. Specifically, accurate recognition of a PPD question ranged from 79% to 100% among all VAs. A verbal response to a PPD question was given between 36% and 79% of the time, while clinically appropriate advice was only given between 14% and 29% of the time. No single VA outperformed any other VA across performance categories.

The sharp dropoff from accurate speech recognition of a query to clinically appropriate advice indicates that while voice recognition of medical queries continues to improve [17,21], improvement in appropriate clinical advice to end users is essential. When considering the four VAs together (Table 1), that number increases to 64% (9/14). This finding suggests that if these services were to integrate their methods and work collaboratively, their performance could double or triple. For example, Apple and Google collaborated on contact tracing in response to the COVID-19 pandemic [22] demonstrating a willingness by technology companies to combine efforts on essential health issues.

While VAs are not approved medical devices, consumers are nonetheless using VAs to answer medical questions, and they should therefore provide accurate medical advice [5,23,24]. In some cases, the VA referenced websites of health care institutions or government sites such as Mayo Clinic and US Department of Health and Human Services, but in other cases the VA provided references from other websites, such as Wikipedia. These examples illustrate how VAs require improvement to be more context-aware in responding to questions. Especially in the medical domain, the VA interface design could be improved following a similar training protocol to medical students. Medical students are taught to follow-up patient questions with open-ended statements, such as, “Tell me more about that,” which often provides more information than direct queries [25]. This follow-up questioning is an essential step toward improving VA responses in mental health disorders, including PPD.

Advertisements offered by VAs, while also troubling, simply indicate that a reasonable pay structure should be developed where technology companies can recoup their development costs. However, if advertisements continue, a reporting and control mechanism should be in place to identify bias toward a specific treatment or service. Most academic centers have strict policies on accepting free material from vendors to maintain an unbiased stance [26-28].

Our study was limited by our use of grammatically correct, well-structured, well-understood PPD questions. In a real-world application, these exact questions may not represent the variation in language, accent, and education of PPD patients. Hence, the results represent the best-case scenario for VAs when used for PPD. Second, physician assessors subjectively interpreted the ACOG FAQ and decided clinically appropriate advice of the VA using globally accepted components of clinical communication. An alternative study could apply a more stringent rubric to the comparison. Third, we evaluated each question and response as a single unit and did not challenge the VAs with a conversation. An alternative study would be to ask a series of questions to build on the previous question and response and eventually to evaluate the advice as a whole. While the interaction is an important component of VA technology, this study can be seen as a test of the VA knowledge base as it pertains to PPD. Fourth, VA medical advice should adhere to local clinical advice. For example, while we used the ACOG FAQ in this study, the responses to the questions may not be appropriate for users in Africa or Asia [29]. Fifth, our study relied on a binary metric of clinically appropriate advice for each VA response. However, a deeper look at each VA response should include an assessment on whether the response provided or omitted information that was potentially dangerous to patients. Finally, sample sizes are small, so P values may not effectively capture differences in VA performance. For example, when comparing Apple Siri and Amazon Alexa for the presence of a verbal response, the calculated power was .49, indicating a low probability of detecting a difference. While a larger sample size is preferred, the goal of this project is to compare the 14-question ACOG FAQ against advice from VAs.

Even though the performance of VAs in responding to PPD currently falls short, VA-based interventions for PPD present a unique opportunity in the long term, as also supported by Kocaballi et al [4] and Miner et al [7]. VAs could support mothers with PPD by addressing misperceptions about the disorder, performing an initial screening, and discussing possible treatment in the privacy of their homes [12]. In light of our findings, we suggest three ways to improve VA performance in mental health disorders and PPD.

First, the lack of clinically appropriate advice showed that clinical organizations should partner with technology companies to develop content and design the user experience. Clinical organizations already produce many patient-facing informational documents found through web searches and VA skills [30]. These organizations should take further steps to develop evidence-based content specifically for VAs, realizing that these devices are not only popular but can also provide meaningful interactions with patients. For example, Siri provided a definition of “baby blues” from pop culture rather than following up with another question to understand the context better.

Second, training VA artificial intelligence with personal information can improve the ability to screen patients and provide context-aware and personalized responses. While this may present privacy challenges, previous successful artificial intelligence implementations in medicine [31] and current VA apps compliant with personal health information protection laws (ie, Health Insurance Portability and Accountability Act) [32] prove that leveraging VA artificial intelligence shows promise in delivering personalized care. For PPD, this effort can lead to better identification of risk factors. Because the US Preventive Services Task Force recommends referral for treatment of PPD after identification of a single risk factor, VA-supported identification could assist providers by marking which of their patients are at risk for PPD. In a future case for PPD, one can imagine a VA moderating screening, counseling a patient on possible treatment options, and assisting providers in decision making with the knowledge gathered through screenings.

Last, public health organizations should partner with technology companies to encourage the delivery of medical care through VAs [33]. In the recent COVID-19 pandemic, the Centers for Disease Control and Prevention used Microsoft’s Healthcare Bot service (text-based) to deliver a relevant and up-to-date information and self-assessment mechanism to the public [34]. This situation is a first-rate example of public-private collaboration in delivering necessary health content through conversational agents. Similar strategies should be established in long-term collaboration in curating and providing PPD and mental health support through VA. To enable successful collaboration, legal and regulatory actions should be developed for digital care delivery, which improved telemedicine during COVID-19 [35].

VAs accurately recognize speech but cannot give clinically appropriate advice for PPD. PPD is a mental health disorder that presents an opportunity for digital health intervention through VAs. By increasing the conversational abilities of VAs, partnering with health organizations to improve the content of these agents, and integrating these agents with the electronic health record, VAs may be a valuable tool to address patients’ misperceptions, screen patients for depression, and initiate a prompt referral to qualified health providers.