Use of the Smartphone App WhatsApp as an E-Learning Method for Medical Residents: Multicenter Controlled Randomized Trial

Background The WhatsApp smartphone app is the most widely used instant messaging app in the world. Recent studies reported the use of WhatsApp for educational purposes, but there is no prospective study comparing WhatsApp’s pedagogical effectiveness to that of any other teaching modality. Objective The main objective of this study was to measure the impact of a learning program via WhatsApp on clinical reasoning in medical residents. Methods This prospective, randomized, multicenter study was conducted among first- and second-year anesthesiology residents (offline recruitment) from four university hospitals in France. Residents were randomized in two groups of online teaching (WhatsApp and control). The WhatsApp group benefited from daily delivery of teaching documents on the WhatsApp app and a weekly clinical case supervised by a senior physician. In the control group, residents had access to the same documents via a traditional computer electronic learning (e-learning) platform. Medical reasoning was self-assessed online by a script concordance test (SCT; primary parameter), and medical knowledge was assessed using multiple-choice questions (MCQs). The residents also completed an online satisfaction questionnaire. Results In this study, 62 residents were randomized (32 to the WhatsApp group and 30 to the control group) and 22 residents in each group answered the online final evaluation. We found a difference between the WhatsApp and control groups for SCTs (60% [SD 9%] vs 68% [SD 11%]; P=.006) but no difference for MCQs (18/30 [SD 4] vs 16/30 [SD 4]; P=.22). Concerning satisfaction, there was a better global satisfaction rate in the WhatsApp group than in the control group (8/10 [interquartile range 8-9] vs 8/10 [interquartile range 8-8]; P=.049). Conclusions Compared to traditional e-learning, the use of WhatsApp for teaching residents was associated with worse clinical reasoning despite better global appreciation. The use of WhatsApp probably contributes to the dispersion of attention linked to the use of the smartphone. The impact of smartphones on clinical reasoning should be studied further.

yes: all primary outcomes were signi cantly better in intervention group vs control partly: SOME primary outcomes were signi cantly better in intervention group vs control no statistically signi cant difference between control and intervention potentially harmful: control was signi cantly better than intervention in one or more outcomes inconclusive: more research is needed Autre : Approx. Percentage of Users (starters) still using the app as recommended after 3 months * Manuscript tracking number * If this is a JMIR submission, please provide the manuscript tracking number under "other" (The ms tracking number can be found in the submission acknowledgement email, or when you login as author in JMIR. If the paper is already published in JMIR, then the ms tracking number is the fourdigit number at the end of the DOI, to be found at the bottom of each published article in JMIR) 1a) Does your paper address CONSORT item 1a? * I.e does the title contain the phrase "Randomized Controlled Trial"? (if not, explain the reason under "other") 1a-i) Identify the mode of delivery in the title Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if o ine products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms.
Does your paper address subitem 1a-i? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "a multicenter, controlled, randomized trial" 1a-ii) Non-web-based components or important co-interventions in title Mention non-web-based components or important co-interventions in title, if any (e.g., "with telephone support").
Does your paper address subitem 1a-ii?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study " smartphone application WhatsApp®" 1a-iii) Primary condition or target group in the title Mention primary condition or target group in the title, if any (e.g., "for children with Type I Diabetes") Example: A Web-based and Mobile Intervention with Telephone Support for Children with Type I Diabetes: Randomized Controlled Trial Does your paper address subitem 1a-iii? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study " an e-learning method for residents teaching" subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential 1b-i) Key features/functionalities/components of the intervention and comparator in the METHODS section of the ABSTRACT Mention key features/functionalities/components of the intervention and comparator in the abstract. If possible, also mention theories and principles used for designing the site. Keep in mind the needs of systematic reviewers and indexers by including important synonyms. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) Does your paper address subitem 1b-i? * Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "multicenter study conducted among rst and second year anesthesiology residents (o ine recruitment). Residents were randomized in two groups (WA and control). The WA group bene ted from daily delivery on the WA application of teaching documents and a weekly clinical case supervised by a senior anesthesiologist. In the control group, residents had access to the same documents via a traditional computer e-learning platform. " 1b-ii) Level of human involvement in the METHODS section of the ABSTRACT Clarify the level of human involvement in the abstract, e.g., use phrases like "fully automated" vs. "therapist/nurse/care provider/physician-assisted" (mention number and expertise of providers involved, if any). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 1b-ii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The WA group bene ted from daily delivery on the WA application of teaching documents and a weekly clinical case supervised by a senior anesthesiologist. In the control group, residents had access to the same documents via a traditional computer e-learning platform." 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-toface assessments in the METHODS section of the ABSTRACT Mention how participants were recruited (online vs. o ine), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in web-based trials Does your paper address subitem 1b-iii? Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "multicenter study conducted among rst and second year anesthesiology residents (o ine recruitment)." "Residents were randomized in two groups of online teaching " Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The emergence of social networks facilitates personal and professional communication and exchange [4]. In addition, the time spent on mobile phone screens per day ("screen time") has increased exponentially since the introduction of the latest generation phones known as smartphones, in particular in young people, leading to a growing interest among teachers for "mobile learning" (m-learning) [1,[5][6][7]. The WhatsApp® (WA) smartphone application is the most widely used instant messaging application in the world, with more than one billion active users per month and more than 40 billion WA messages exchanged each day in 2016. [8]" " In view of its popularity with medical students, it seems interesting to envisage a new use of this application, by orienting it towards an educational objective (with the opportunity to recover screen time from students).
[9] The rst reports of the use of this application for educational purposes date from early 2017 for the teaching of medical students or the training of pathology residents. [10,11] Both of these observational studies showed satisfaction among WA participants and highlighted the ease of use and the quick access to lessons through the application. However, there is no prospective study comparing the effectiveness of WA to any other teaching modality." Does your paper address CONSORT subitem 2b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "We hypothesized that WA as an educational tool could improve the clinical reasoning of anesthesiology residents as observed with face-to-face PEG. The main objective of this study was to evaluate the impact of WA on clinical reasoning applied to emergency situations that were assessed by using the SCT." 3a) Description of trial design (such as parallel, factorial) including allocation ratio 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 3a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Prospective, randomized, multicenter study conducted among rst and second year anesthesiology residents from four French university hospitals with trauma center (Amiens, Caen, Lille and Rouen) comparing WA to a control group" "After inclusion, residents were randomized in two groups (WA and control) with strati cation according to the student's hospital and year of residency" Does your paper address CONSORT subitem 3b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable 3b-i) Bug xes, Downtimes, Content Changes Bug xes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug xes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have in uenced study design such as staff Does your paper address subitem 3b-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable because there was no changes to methods after trial commencement Does your paper address CONSORT subitem 4a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The inclusion criteria were: residents who had a mobile phone that could download the WA application and who had attested their agreement to use it for this study. Non-inclusion criteria were: refusal to participate, non-compatibility of a mobile phone to download WA and failure to download WA. " 4a-i) Computer / Internet literacy Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clari ed.
Does your paper address subitem 4a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The inclusion criteria were: residents who had a mobile phone that could download the WA application and who had attested their agreement to use it for this study. Non-inclusion criteria were: refusal to participate, non-compatibility of a mobile phone to download WA and failure to download WA. " subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential 4b) Settings and locations where the data were collected 4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. o ine), e.g., from an open access website or from a clinic, and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email con rmation, phone calls) were used to detect/prevent these.
Does your paper address subitem 4a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "After o ine inclusion, residents were randomized in two groups (WA and control) with strati cation according to the student's hospital and year of residency" "Concerning intervention or evaluation, this was a purely app-or web-based trial without face-to-face components between resident and teachers. " 4a-iii) Information giving during recruitment Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results.
Does your paper address subitem 4a-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "All information about the trial was sent by e-mail. The participation agreement was collected by e-mail or telephone from each resident " subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 4b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Concerning intervention or evaluation, this was a purely app-or web-based trial without face-to-face components between resident and teachers. " 4b-i) Report if outcomes were (self-)assessed through online questionnaires Clearly report if outcomes were (self-)assessed through online questionnaires (as common in webbased trials) or otherwise.
Does your paper address subitem 4b-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Concerning intervention or evaluation, this was a purely app-or web-based trial without face-to-face components between resident and teachers. " "At the end of the teaching period, the two groups had the same formative evaluation by 27 SCT and 30 MCQ sent by e-mail and completed during the month following the end of the teaching period. Students returned SCT and MCQ by e-mail after completing them. The residents of the two groups who responded to the nal evaluation completed an online satisfaction questionnaire " 4b-ii) Report how institutional a liations are displayed Report how institutional a liations are displayed to potential participants [on ehealth media], as a liations with prestigious hospitals or universities may affect volunteer rates, use, and reactions with regards to an intervention.(Not a required item -describe only if this may bias results) 5) The interventions for each group with su cient details to allow replication, including how and when they were actually administered subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 4b-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Votre réponse 5-i) Mention names, credential, a liations of the developers, sponsors, and owners Mention names, credential, a liations of the developers, sponsors, and owners [6] (if authors/evaluators are owners or developer of the software, this needs to be declared in a "Con ict of interest" section or mentioned elsewhere in the manuscript).
Does your paper address subitem 5-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study This part is developed in the introduction: "The WhatsApp® (WA) smartphone application, developed by WhatsApp Inc.

5-ii) Describe the history/development process
Describe the history/development process of the application and previous formative evaluations (e.g., focus groups, usability testing), as these will have an impact on adoption/use rates and help with interpreting results.
subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 5-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable

5-iii) Revisions and updating
Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was "frozen" during the trial. Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b).
Does your paper address subitem 5-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable

5-iv) Quality assurance methods
Provide information on quality assurance methods to ensure accuracy and quality of information provided [1], if applicable.
Does your paper address subitem 5-iv?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The WA group bene ted from daily delivery on the WA application of teaching documents specially prepared to be easily readable on a smartphone (from Monday to Thursday, morning and afternoon; 2 to 4 documents/day). These documents were inspired by the most recent guidelines concerning the management of traumatic hemorrhagic shock and were validated by anesthesiology teachers (VC, BV and BD -v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing owcharts of the algorithms used Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing owcharts of the algorithms used. Replicability (i.e., other researchers should in principle be able to replicate the study) is a hallmark of scienti c reporting.
Does your paper address subitem 5-v?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Yes, screenshot available in appendix

5-vi) Digital preservation
Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login.
Does your paper address subitem 5-vi?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study

5-vii) Access
Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of speci c group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi).
Does your paper address subitem 5-vii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "a mobile phone that could download the WA application " subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 5-viii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The WA group bene ted from daily delivery on the WA application of teaching documents specially prepared to be easily readable on a smartphone (from Monday to Thursday, morning and afternoon; 2 to 4 documents/day). These documents were inspired by the most recent guidelines concerning the management of traumatic hemorrhagic shock and were validated by anesthesiology teachers (VC, BV and BD) [17,18]. Every Friday, residents were given a "step-by-step" clinical case on WA during 3-4 hours, supervised by a senior anesthesiologist (TC) who asked questions to residents (to create an interest in the clinical cases) and answered to the residents with feedback and validation (or correction if necessary) as described in PEG. [12] Several examples of the use of WA for learning purposes during the protocol (in French, with a transcript in English) are presented in the Multimedia Appendix 1. The total duration of teaching was three weeks. In the control group, residents had access to the same documents via a computer e-learning platform, and the senior anesthesiologist was available by e-mail. They had access to the three clinical cases with their answers but had no live interaction with a teacher. The two groups had the same program and the same learning objectives. "

5-ix) Describe use parameters
Describe use parameters (e.g., intended "doses" and optimal timing for use). Clarify what instructions or recommendations were given to the user, e.g., regarding timing, frequency, heaviness of use, if any, or was the intervention used ad libitum.
subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 5-ix?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The WA group bene ted from daily delivery on the WA application of teaching documents specially prepared to be easily readable on a smartphone (from Monday to Thursday, morning and afternoon; 2 to 4 documents/day). " "Every Friday, residents were given a "step-by-step" clinical case on WA during 3-4 hours, supervised by a senior anesthesiologist (TC) who asked questions to residents (to create an interest in the clinical cases) and answered to the residents with feedback and validation (or correction if necessary) as described in PEG"

5-x) Clarify the level of human involvement
Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21generalizability).
Does your paper address subitem 5-x?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "These documents were inspired by the most recent guidelines concerning the management of traumatic hemorrhagic shock and were validated by anesthesiology teachers (VC, BV and BD) [17,18]. Every Friday, residents were given a "step-by-step" clinical case on WA during 3-4 hours, supervised by a senior anesthesiologist (TC) who asked questions to residents (to create an interest in the clinical cases) and answered to the residents with feedback and validation (or correction if necessary) as described in PEG " subitem not at all important 1 2 3 4 5 essential subitem not at all important 1 2 3 4 5 essential 6a) Completely de ned pre-speci ed primary and secondary outcome measures, including how and when they were assessed 5-xi) Report any prompts/reminders used Report any prompts/reminders used: Clarify if there were prompts (letters, emails, phone calls, SMS) to use the application, what triggered them, frequency etc. It may be necessary to distinguish between the level of prompts/reminders required for the trial, and the level of prompts/reminders for a routine application outside of a RCT setting (discuss under item 21 -generalizability).
Does your paper address subitem 5-xi? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "In case of non-response, residents had two reminders by e-mail before being considered lost to follow-up. "

5-xii) Describe any co-interventions (incl. training/support)
Describe any co-interventions (incl. training/support): Clearly state any interventions that are provided in addition to the targeted eHealth intervention, as ehealth intervention may not be designed as stand-alone intervention. This includes training sessions and support [1]. It may be necessary to distinguish between the level of training required for the trial, and the level of training for a routine application outside of a RCT setting (discuss under item 21 -generalizability. Does your paper address subitem 5-xii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable in our work (no co-interventions) subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 6a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The primary objective was medical reasoning evaluated by SCT.  7b) When applicable, explanation of any interim analyses and stopping guidelines Does your paper address CONSORT subitem 6b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study not applicable (no changes to trial outcomes) 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size Describe whether and how expected attrition was taken into account when calculating the sample size.
Does your paper address subitem 7a-i?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "In regard to our previous publication involving SCT in anesthesiology residents, we assumed that a difference of 6% (with a standard deviation (SD) of 6%) between the two groups would be clinically signi cant. [12,19] Based on these ndings, assuming that the SD was the same between the populations and using a power of 0.90 with a level of statistical signi cance at 0.05, it was estimated that 22 students should be analyzed in each group. It has been shown in a randomized study on e-learning that about a quarter of students included do not participate or are lost to follow-up.
[20] Based on these ndings it was estimated that a minimum of 28 students should be included in each group to be able to analyze 22 students." 8a) Method used to generate the random allocation sequence NPT: When applicable, how care providers were allocated to each trial group 8b) Type of randomisation; details of any restriction (such as blocking and block size) 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned Does your paper address CONSORT subitem 7b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable to our protocol Does your paper address CONSORT subitem 8a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "After o ine inclusion, residents were randomized in two groups (WA and control) using an online open-access application with strati cation according to the student's hospital and year of residency [17]. " Does your paper address CONSORT subitem 8b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "with strati cation according to the student's hospital and year of residency " 10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 9? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "After o ine inclusion, residents were randomized with their last names in two groups " Does your paper address CONSORT subitem 10? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "residents were randomized with their last names in two groups (WA and control) by TC " 11a-i) Specify who was blinded, and who wasn't Specify who was blinded, and who wasn't. Usually, in web-based trials it is not possible to blind the participants [1, 3] (this should be clearly acknowledged), but it may be possible to blind outcome assessors, those doing data analysis or those administering co-interventions (if any).
Does your paper address subitem 11a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Prospective, randomized, unblinded, multicenter study " subitem not at all important 1 2 3 4 5 essential 11b) If relevant, description of the similarity of interventions (this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) 11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Informed consent procedures (4a-ii) can create biases and certain expectations -discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator".
Does your paper address subitem 11a-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "All information about and during the trial was sent by e-mail, participants knew that the WA group was the "intervention" group" Does your paper address CONSORT subitem 11b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study 12a-i) Imputation techniques to deal with attrition / missing values Imputation techniques to deal with attrition / missing values: Not all participants will use the intervention/comparator as intended and attrition is typically high in ehealth trials. Specify how participants who did not use the application or dropped out from the trial were treated in the statistical analysis (a complete case analysis is strongly discouraged, and simple imputation techniques such as LOCF may also be problematic [4]).
Does your paper address subitem 12a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Residents who did not responded to the nal evaluation were excluded of the Does your paper address CONSORT subitem 12b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study not applicable (no subgroup analysis)

X26-i) Comment on ethics committee approval
Does your paper address subitem X26-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The Ethics and Evaluation Committee for Non-Interventional Research of Rouen University Hospital approved the study (n° E2017-37). "

x26-ii) Outline informed consent procedures
Outline informed consent procedures e.g., if consent was obtained o ine or online (how? Checkbox, etc.?), and what information was provided (see 4a-ii). See [6] for some items to be included in informed consent documents.
Does your paper address subitem X26-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The participation agreement was collected online by e-mail or telephone from each resident." subitem not at all important 1 2 3 4 5 essential RESULTS 13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center 13b) For each group, losses and exclusions after randomisation, together with reasons X26-iii) Safety and security procedures Safety and security procedures, incl. privacy considerations, and any steps taken to reduce the likelihood or detection of harm (e.g., education and training, availability of a hotline) Does your paper address subitem X26-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Participants did not receive any documents during the weekends and were free to stop the courses at any time." Does your paper address CONSORT subitem 13a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Among 142 eligible anesthesiology residents, 62 (44%) accepted to participate and were randomized: 32 in the WA group and 30 in the control group. " "Concerning the preliminary short evaluation conducted after randomization but before teaching, 19 residents responded in the WA group vs. 22 in the control group. For the nal evaluation, 22 residents responded in each group." subitem not at all important 1 2 3 4 5 essential 14a) Dates de ning the periods of recruitment and follow-up Does your paper address CONSORT subitem 13b? (NOTE: Preferably, this is shown in a CONSORT ow diagram) * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Concerning the preliminary short evaluation conducted after randomization but before teaching, 19 residents responded in the WA group vs. 22 in the control group. For the nal evaluation, 22 residents responded in each group. The ow chart of the study is presented in gure 1."

13b-i) Attrition diagram
Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other gures or tables demonstrating usage/dose/engagement.

Does your paper address subitem 13b-i?
Copy and paste relevant sections from the manuscript or cite the gure number if applicable (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The ow chart of the study is presented in gure 1." Does your paper address CONSORT subitem 14a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study 14a-i) Indicate if critical "secular events" fell into the study period Indicate if critical "secular events" fell into the study period, e.g., signi cant changes in Internet resources available or "changes in computer hardware or Internet delivery resources" Does your paper address subitem 14a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable (no secular events) Does your paper address CONSORT subitem 14b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable (no early stop) Does your paper address CONSORT subitem 15? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study 16-i) Report multiple "denominators" and provide de nitions Report multiple "denominators" and provide de nitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at speci c prede ned time points of interest (in absolute and relative numbers per group). Always clearly de ne "use" of the intervention.
subitem not at all important 1 2 3 4 5 essential 17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% con dence interval) Does your paper address subitem 16-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Among 142 eligible anesthesiology residents, 62 (44%) accepted to participate and were randomized: 32 in the WA group and 30 in the control group. " "Concerning the preliminary short evaluation conducted after randomization but before teaching, 19 residents responded in the WA group vs. 22 in the control group. For the nal evaluation, 22 residents responded in each group. The ow chart of the study is presented in gure 1."

16-ii) Primary analysis should be intent-to-treat
Primary analysis should be intent-to-treat, secondary analyses could include comparing only "users", with the appropriate caveats that this is no longer a randomized sample (see 18-i).
Does your paper address subitem 16-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Votre réponse subitem not at all important 1 2 3 4 5 essential 17b) For binary outcomes, presentation of both absolute and relative effect sizes is recommended Does your paper address CONSORT subitem 17a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "On the preliminary evaluation (before teaching), there was no signi cant difference between WA and control groups for SCT (64 ± 7 % vs. 62 ± 6 %, respectively; p=0.41) or MCQ (8 ± 1 /10 vs. 7 ± 2 /10, respectively; p=0. 33) showing no difference in clinical reasoning or medical knowledge. For the nal evaluation (after teaching), we found a signi cant difference between WA and control groups for SCT (60 ± 9 % vs. 68 ± 11 %, respectively; p<0.01) but no difference for MCQ (18 ± 4 /30 vs. 16 ± 4 /30, respectively; p=0.22). " 17a-i) Presentation of process outcomes such as metrics of use and intensity of use In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational de nitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is de ned (e.g., timeout after idle time) [1] (report under item 6a).
Does your paper address subitem 17a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable in our work 18) Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing prespeci ed from exploratory subitem not at all important 1 2 3 4 5 essential Does your paper address CONSORT subitem 17b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study " the preliminary evaluation (before teaching), there was no signi cant difference between WA and control groups for SCT (64 ± 7 % vs. 62 ± 6 %, respectively; p=0.41) or MCQ (8 ± 1 /10 vs. 7 ± 2 /10, respectively; p=0.33) showing no difference in clinical reasoning or medical knowledge. For the nal evaluation (after teaching), we found a signi cant difference between WA and control groups for SCT (60 ± 9 % vs. 68 ± 11 %, respectively; p<0.01) but no difference for MCQ (18 ± 4 /30 vs. 16 ± 4 /30,respectively;p=0.22). " Does your paper address CONSORT subitem 18? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable in our work

18-i) Subgroup analysis of comparing only users
A subgroup analysis of comparing only users is not uncommon in ehealth trials, but if done, it must be stressed that this is a self-selected sample and no longer an unbiased sample from a randomized trial (see 16-iii).
Does your paper address subitem 18-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "The residents of the two groups who lled the nal evaluation were asked to ll a satisfaction questionnaire: 20 (67%) in the WA group and 13 (43%) in the control group. There was a difference between WA and control groups with a better global satisfaction rate in the WA group (9 ± 1 /10 vs. 8 ± 2 /10, respectively; p<0.05), a better felling that the lessons met the learning objectives in the WA group (9 ± 1 /10 vs. 8 ± 2 /10, respectively; p<0.05) and a feeling that the teachers were more available in the WA group (10 ± 1 /10 vs. 9 ± 2 /10, respectively; p<0.01). " 22-i) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use).
Does your paper address subitem 22-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "This randomized, multicenter study is the rst to focus on the impact of WA on clinical reasoning in medical students. We found that the use of WA instead of a traditional e-learning platform to teaching a speci c topic is associated with Does your paper address subitem 22-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Given the increasing use of smartphones by health workers (including in operating room or in intensive care unit), it also seems appropriate to consider a future work to assess the quality of clinical reasoning between two populations of physicians with or without usual smartphone use in hospital."

20-i) Typical limitations in ehealth trials
Typical limitations in ehealth trials: Participants in ehealth trials are rarely blinded. Ehealth trials often look at a multiplicity of outcomes, increasing risk for a Type I error. Discuss biases due to nonuse of the intervention/usability issues, biases through informed consent procedures, unexpected events.

21) Generalisability (external validity, applicability) of the trial ndings
NPT: External validity of the trial ndings according to the intervention, comparators, patients, and care providers or centers involved in the trial subitem not at all important 1 2 3 4 5 essential Does your paper address subitem 20-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Nevertheless, our study has many limitations. First, the Cronbach coe cient α in our SCT evaluation was low. The minimum coe cient usually retained for normative evaluations is 0.7 but in our work it was only a formative evaluation integrated into teaching. The limited number of SCT probably explains this low coe cient. However, teaching in a really speci c and specialized area made it di cult to nd at least 60 SCT (as is usually recommended) without being redundant [16]. Second, residents' participation in our work was limited: only 62 of 142 residents participated. As previously observed, self-training with e-learning is impacted by a signi cant dropout rate [21]. In our work, only 22 out of 30 residents participated in the nal evaluation. As this teaching was optional, participation in our study represented additional personal work for the residents.
It is therefore possible that the majority of residents were discouraged by this prospect. In addition, residents without smartphones or who did not wish to use WA logically refused to participate."

21-i) Generalizability to other populations
Generalizability to other populations: In particular, discuss generalizability to a general Internet population, outside of a RCT setting, and general patient population, including applicability of the study results for other organizations subitem not at all important 1 2 3 4 5 essential OTHER INFORMATION 23) Registration number and name of trial registry Does your paper address subitem 21-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "We did not nd any improvement in medical knowledge in our work but the smartphone was seen as an alternative to conventional e-learning and not as a complement. It is interesting to note that in the literature, most of the educational bene ts reported with smartphone use come from very "visual" specialties (dermatology, pathology) and that this tool, that allows easy communication of iconography, is probably more relevant in this context than in "less visual" medical specialties.[10,22]" 21-ii) Discuss if there were elements in the RCT that would be different in a routine application setting Discuss if there were elements in the RCT that would be different in a routine application setting (e.g., prompts/reminders, more human involvement, training sessions or other co-interventions) and what impact the omission of these elements could have on use, adoption, or outcomes if the intervention is applied outside of a RCT setting.
Does your paper address subitem 21-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study "Residents point out two limitations: the di culty of participating in Friday clinical cases in parallel with their usual activities and the di culty of consulting documents on small screens on smartphone. Unlike for PEG, there was no time dedicated speci cally to WA clinical case resolution but residents had to respond in addition to their usual activities.
[12]" Does your paper address CONSORT subitem 23? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study Not applicable (no registration number for pedagogical study in France) Does your paper address CONSORT subitem 24? * Cite a Multimedia Appendix, other reference, or copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study All the trial procotol is in the method part Does your paper address CONSORT subitem 25? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brie y explain why the item is not applicable/relevant for your study " Funding support was provided solely from departmental sources." X27-i) State the relation of the study team towards the system being evaluated In addition to the usual declaration of interests ( nancial or otherwise), also state the relation of the study team towards the system being evaluated, i.e., state if the authors/evaluators are distinct from or identical with the developers/sponsors of the intervention. yes no Autre : STOP -Save this form as PDF before you click submit To generate a record that you lled in this form, we recommend to generate a PDF of this page (on a Mac, simply select "print" and then select "print as PDF") before you submit it.
When you submit your (revised) paper to JMIR, please upload the PDF as supplementary le.
Don't worry if some text in the textboxes is cut off, as we still have the complete information in our database. Thank you!