Published on 20.07.16 in Vol 4, No 3 (2016): Jul-Sept
Psychologist in a Pocket: Lexicon Development and Content Validation of a Mobile-Based App for Depression Screening
Background: Language reflects the state of one’s mental health and personal characteristics. It also reveals preoccupations with a particular schema, thus possibly providing insights into psychological conditions. Using text or lexical analysis in exploring depression, negative schemas and self-focusing tendencies may be depicted. As mobile technology has become highly integrated in daily routine, mobile devices have the capacity for ecological momentary assessment (EMA), specifically the experience sampling method (ESM), where behavior is captured in real-time or closer in time to experience in one’s natural environment. Extending mobile technology to psychological health could augment initial clinical assessment, particularly of mood disturbances, such as depression and analyze daily activities, such as language use in communication. Here, we present the process of lexicon generation and development and the initial validation of Psychologist in a Pocket (PiaP), a mobile app designed to screen signs of depression through text analysis.
Objective: The main objectives of the study are (1) to generate and develop a depressive lexicon that can be used for screening text-input in mobile apps to be used in the PiaP; and (2) to conduct content validation as initial validation.
Methods: The first phase of our research focused on lexicon development. Words related to depression and its symptoms based on the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) and in the ICD-10 Classification of Mental and Behavioural Disorders: Clinical Descriptions and Diagnostic Guidelines classification systems were gathered from focus group discussions with Filipino college students, interviews with mental health professionals, and the review of established scales for depression and other related constructs.
Results: The lexicon development phase yielded a database consisting of 13 categories based on the criteria depressive symptoms in the DSM-5 and ICD-10. For the draft of the depression lexicon for PiaP, we were able to gather 1762 main keywords and 9655 derivatives of main keywords. In addition, we compiled 823,869 spelling variations. Keywords included negatively-valenced words like “sad”, “unworthy”, or “tired” which are almost always accompanied by personal pronouns, such as “I”, “I’m” or “my” and in Filipino, “ako” or “ko”. For the content validation, only keywords with CVR equal to or more than 0.75 were included in the depression lexicon test-run version. The mean of all CVRs yielded a high overall CVI of 0.90. A total of 1498 main keywords, 8911 derivatives of main keywords, and 783,140 spelling variations, with a total of 793, 553 keywords now comprise the test-run version.
Conclusions: The generation of the depression lexicon is relatively exhaustive. The breadth of keywords used in text analysis incorporates the characteristic expressions of depression and its related constructs by a particular culture and age group. A content-validated mobile health app, PiaP may help augment a more effective and early detection of depressive symptoms.
JMIR Mhealth Uhealth 2016;4(3):e88
Depression has been identified as the most prevalent clinical disorder and main cause of illness among college students and young adults [, ]. In this age group, depression manifests itself in poor academic performance, problems in communication, strained relations with family and peers, lesser interaction time with friends, and increased frequency in drinking and smoking behaviors [ ]. It may also be signaled by loss of weight or increase in appetite and poor sleeping patterns [ ]. In worse cases, severe depression is manifested in suicidal thoughts, plans and attempts, and self-harming behaviors [ ].
Timely recognition of depressive symptoms is not without challenges. First, mood is transient, changing, and must be captured at the time of experience . Second, primary strategies or traditional techniques used in assessment, such as face-to-face interviews and questionnaires, may overlook the presence of depression or produce negative recall bias [ ]. Third, cultures influence the understanding and acceptance of mental illness. In many Asian countries, stigma, fears of misconception, and “losing one’s face” [ - ] hinder help-seeking behaviors. In the Philippines, despite having the highest depression incidence (over 4.5 million) in Southeast Asia, only few will seek help [ ], while a significant number underreport self-harm and suicide cases due to religious reasons [ ]. Others remain ill-informed, avoid acknowledging their condition, or treat depression as normal sadness. Filipino college students who experience depression tend to suffer in silence, although symptoms are highly observed [ ].
Language, and the words that one uses, is a reflection of an individual’s mental state, exposing how one organizes and interprets his/her world and revealing thoughts and feelings. The study of language and emotions, specifically lexical or text analysis, rests on the assumption that language features can be markers of mental health and that mental states and personal characteristics are reflected in the words people use in natural language . The choice of spoken or written vocabulary conveys an individual’s thoughts irrespective of the intentions during communication [ ]. Moreover, mental processes that may be out of awareness influence how an individual selects the words to be used by revealing itself through lexical leakages. Regardless of the contextual meaning of words, the selection, per se, is determined by a mental preoccupation with a particular theme. In other words, an individual, possessing a certain thought pattern (eg, I am worthless), may reveal these thoughts through spoken and written language.
In studying depression and its symptoms, everyday language reveals the cognitive mechanisms activated . Depression involves negative schemas that guide attention towards and enhance recall of negative experiences [ ]. Continually being reinforced, active negative schemas progress and develop into negative views of the self, the world, and the future. However, at times, latent or dormant negative schemas expose themselves only upon the experience of a stressor or cognitive load/task that interferes with a person’s attempts to reduce unwanted negative thinking [ ]. Either way, schemas control one’s beliefs and automatic thoughts which are accessible and are manifested as words indicating negative emotions and negative thinking [ ] or sometimes suicide and death [ , ]. According to the Self-Focus Model of Depression, depressed or depression-prone individuals tend to ruminate and interpret events in terms of themselves [ ]. Depressed individuals generally engage in higher levels of self-focus resulting in a higher activation of negative self-schema. The depressive self-focusing style then maintains and exacerbates the disorder or heightens vulnerability. Modalities, such as increased social media activity, raised negative affect, high self-attentional focus, heightened relational and medicinal concerns, and greater expression of religious involvement, may characterize and signal depression onset [ ].
Worldwide, mobile phone ownership has reached more than 6 billion . Daily life events and digital behaviors, such as Internet usage, SMS text messaging (short message service, SMS), online chatting, blogging and social media access, are converging around mobile devices [ , ], making the technology a highly integral part of day-to-day activities, specifically in the Asia Pacific. In the Philippines, mobile phones with app capabilities are increasingly utilized for communication and entertainment [ ], especially in the 18 to 24 age bracket [ ].
The popularity of mobile technology is harnessed in the delivery of health care and information via mobile health (mHealth) . Mobile technology’s adoption in mental health care prevents stigmatization since it attracts no undue attention to its user [ , ]. Furthermore, as it is highly integrated in daily routine, mobile devices have the capacity for Ecological Momentary Assessment (EMA), where behavior is captured in real-time or closer in time to experience, in the natural environment, and with a multitude of measurements over time [ , ]. It could also augment clinical assessment, particularly mood disturbances such as depression, and make it possible to study cognition and other processes through the analysis of daily activities, such as language use [ ].
According to Rude and colleagues, language is a medium in detecting depression and depression proneness in individuals . In studies involving novel avenues of communication, activities in social networking sites (SNS) could signal or reveal symptoms of depression or expose at-risk and fatal suicidal behaviors: disclosing feelings of depression through status updates on Facebook [ , ], decreasing levels of social activity accompanied by increasing levels of negative emotions, and interest in medication [ , ], reading more tips and facts about depression [ ], hinting at signs of hopelessness and meaninglessness, even somatic pains through Facebook [ ], posting ruminating behaviors on Twitter [ ], and expressing suicidal feelings or communicating suicide-related behavior on social media [ ].
Tapping into the pervasive presence of mobile technology in daily functioning and its utilization in mental health care, this paper presents the development and validation of a depression lexicon for the mobile mental health app Psychologist in a Pocket (PiaP), an open source Android-based app designed to detect symptoms of depression using EMA . PiaP is envisioned to be a novel screening approach for depression symptoms to augment traditional clinical assessment ( ). It is not intended to diagnose or to replace mental health professionals. Specifically, the objectives of the study are (1) to develop a depression lexicon using both bottom-up and top-down approaches for automated text analysis functions; and (2) to validate this lexicon in terms of its content or adequacy in representing its domain.
The PiaP app is designed to be a measurement tool for depressive symptoms employing a novel method by performing text analysis directly on a mobile device. One of the first stages in developing a test or a measurement tool is to create items to represent the domain. In this case, the initial step in PiaP’s development is to generate a lexicon of keywords to represent depressive symptoms. While the development of the lexicon is purported for the text analysis function of PiaP, the depression lexicon or database of depressive words and phrases may be used in any type of text analysis or sentiment analysis that could be computer or mobile-based.
In building a lexicon of a particular subject or domain such as depressive symptoms, the top-down and bottom-up approaches may be utilized in order to represent an entire domain . The top-down approach, also called the gold-standard approach, starts with a domain and from there, fills in details that would be descriptive of that domain using expert knowledge. It establishes the principles with which the domain may be communicated, and for each principle, determines on the basis of competence or observational data, what words are employed to communicate the same or a particular meaning. The bottom-up approach starts with individuals and their linguistic elements and then records and analyzes each word while testing if the same words or expressions always have the same meaning.
In building the depression lexicon, both the gold-standard descriptions (ie, DSM and ICD classification systems) and the accounts of the individuals themselves who experience the symptoms were utilized to ensure that the lexicon encompasses most of the domain. Furthermore, the language culture of the target population must be considered to make certain that the lexicon reflects both the traditional and slang ways of symptom expression.
Building of Categories
The depressive symptoms listed and described under Depressive Episode in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5)  and in the ICD-10 Classification of Mental and Behavioural Disorders: Clinical Descriptions and Diagnostic Guidelines [ ] were the basis for the categories of the depression lexicon ( ). We created keywords under each category to represent their domain or construct. In other words, DSM-5 depressed mood/irritable mood symptom and ICD-10 depressed mood symptom correspond to the PiaP category termed Mood which has keywords encompassing and representing symptoms of depressed mood or irritable mood.
|Mood||Depressed mood||Clearly abnormal depressive mood|
|Interest||Markedly diminished interest or pleasure in all or almost all activities||Marked loss of interest or ability to enjoy activities that were previously pleasurable|
|Appetite and weight||Significant weight loss or gain, or increase or decrease in appetite||Changes of appetite (decrease or increase), with the corresponding weight change|
|Sleep||Sleep problems||Sleep alterations of any kind|
|Psychomotor agitation||Psychomotor agitation or retardation||Changes of psychomotor activity, with agitation or inhibition|
|Psychomotor retardation||Psychomotor agitation or retardation||Changes of psychomotor activity, with agitation or inhibition|
|Fatigue||Fatigue or loss of energy||Lack of vitality or increase of fatigability|
|Guilt and self-esteem||Feelings of guilt, worthlessness, negative self-appraisal||Disproportionate self-reproaches and feelings of excessive guilt or inadequacy; loss of confidence and self-esteem and feelings of inferiority|
|Concentration||Diminished concentration or indecisiveness||Complaints about or decrease of the ability to concentrate and think, accompanied by a lack of decision and vacillation|
|Suicide||Recurrent thoughts of death or suicidal ideation||Recurrent thoughts of death or suicide or any suicidal behavior|
|Alcohol and substance use||Substance abuse||Excessive consumption of alcohol|
|Histrionic behavior||Histrionic behavior||Histrionic behavior|
Symptoms, such as anxiety, substance use, and histrionic behavior, which are associated symptoms or features of depression according to these diagnostic systems, were also included in the lexicon. The keywords in each category were gathered using the approaches of focus group discussions with college students, interviews with mental health professionals, review of depression scales, and spelling variation of keywords generated.
Focus Group Discussions
Focus group discussions were conducted with 76 college students with a mean (SD) age of 17.28 (1.14) and 61% female (46/76) whose depression symptoms based on Beck’s Depression Index II (BDI-II) scores ranged from 14 (mild) to 63 (severe) (). The purpose of the focus group discussions was to gather words, expressions, and symbolic representations of depression. This bottom-up process of lexicon building creates an inclusive representation of the linguistic ways people use to express the experience of depression. Participants were randomly assigned to 1 of the 7 focus group discussion sessions (10 to 11 participants per session), each lasting for 45 to 60 minutes. Before the conduct and recording of each session, 2 of the researchers acting as moderator and note-taker, provided a quick briefing of the study. Informed consent from participants was also obtained prior to the sessions.
Questions centered on how the participants define and describe their depression, what words they typically use to express depression in mobile text inputs and in social media, and how they recognize the signs of depression in another person’s text input or social media activity. At the end of each discussion, participants were presented with a set of emoticons and emojis depicting negative emotions. They were asked to choose the emoticons and emojis they frequently used to describe the emotions and feelings associated with the experience of depressive symptoms.
All responses were transcribed verbatim into Excel and examined by 3 of the researchers for analysis and synthesis of possible themes. Transcripts were coded according to identified patterns and relationships.
Interviews With Mental Health Professionals
As part of the top-down or gold standard method of lexicon building, 5 selected mental health professionals, 2 psychiatrists, 2 clinical psychologists, and 1 counselor facilitator of a support group for depression and substance use, were interviewed. The purpose of the interview was to determine how depressive symptoms among Filipino adolescents are linguistically expressed via mobile technology. Specifically, questions revolved around typical words and phrases of a patient or client who experiences depressive symptoms, ways to identify depressed individuals through novel communication like social media, and feasibility of mobile apps for depression.
The selection criteria of the mental health professionals for interview included (1) more than 10 years experience in their chosen fields (mean 19 years); (2) are officially recognized by the Philippine Regulatory Commission as a licensed psychologist, psychiatrist, or counselor; and (3) have been working with adolescents and young adults suffering from depression. Letters of request for interview were sent to 12 mental health professionals. Of those, 5 agreed to participate in the interview. They were also requested to confirm the findings and the generated keywords from the focus group discussions. Each interview was tape recorded and transcribed verbatim into Excel to infer the main points.
Review of Depression Scales
Another top-down approach in lexicon building was to inspect established and psychometrically sound measurement tools for depression. Words that depict depressive symptoms and variables related to depression, such as negative affect, were extracted from 18 scales, questionnaires, and inventories (). Nouns, verbs, adjectives, and adverbs that relate to depressive symptoms were gathered and frequencies were computed for each keyword found in these sources.
Spelling Variation of Keywords
Young Filipinos typically abbreviate or change the spelling of certain words and mix Filipino/Tagalog and English words whilst communicating. This even coined the terms “textolog” and “tag-lish”. Written language in communication modes such as text messages or SNS has unique characteristics, thus a detection tool must encompass the peculiarities of language expression of different cultures.
To take into consideration the unique culture of Filipinos in inputting text in mobile devices, 500 printed forms containing the list of all main keywords and their derivatives (eg, verb changes) gathered from the previously mentioned methods were distributed randomly to students in 4 universities and colleges (3 in Metro Manila and 1 in Central Luzon). Instructions were to provide at least 3 abbreviations or spelling variations of the keywords. Through convenient selection, data was only considered from 328 students who were able to complete the forms within 2 weeks.
Categorization of Keywords
In the top-down approach, all keywords are generated based on the previously defined categories and are therefore inherently categorized. For the bottom-up approach, all keywords generated by the focus group participants need to be classified into one of the existing categories. This task was initially performed by 2 of the authors (PGFC and RMR). Keywords were distributed and grouped according to the symptom they represent. This keyword categorization was then subjected to content validation by 8 experts.
Content Validation by Experts
According to Kaplan and Saccuzzo , the adequacy of representation of the construct or domain the test or tool is designed to measure is reflected by its content-related evidence for validity. Content validation is one the psychometric procedures that index a test’s validity or its ability to measure what it purports to measure. A typical method of content validation involves multiple judges rating each item in the test in terms of its relevance to the content [ ].
Content validation was conducted to ensure that each keyword belonged to the correct category. Mental health practitioners (N=22) comprised of licensed clinical psychologists, psychiatrists, guidance counselors, and psychology instructors were requested to validate the list. Of those, 36% (8/22) agreed to conduct the content validation. These experts (1) have at least 10 years experience in their chosen fields (mean 17.13 years); (2) are officially recognized by the Philippine Regulatory Commission as licensed psychologists, psychiatrists, or counselors; (3) work with adolescents and young adults with mental health problems; and (4) are currently professionally involved in at least 2 fields such as in the academe and clinical practice.
Using Lawshe’s content validity ratio (CVR) in deriving the content validity index (CVI), the experts rated whether the keywords were “essential”, “useful but not essential”, or “not necessary” to the performance of the construct [, ]. CVR per keyword is depicted by the ratio of the number of experts indicating a keyword as “essential” over the total number of experts. The greater the number of the experts indicating a keyword as “essential”, the greater is the keyword’s content validity. For the 8 validators, each item or keyword must reach a CVR equal to or more than 0.75 to be included in the lexicon [ , ]. The CVI is the mean CVR of all the retained keywords representing the commonality of judgments regarding the validity of the lexicon. The overall content validity is considered high since the value of the CVI approached 1. For the CVRs that did not reach the acceptable CVR level, further selection of keywords to be retained is still possible. For every keyword, 2 judgement points were awarded per acceptance by a validator or when a validator rates the item as “essential”. Afterwards, mean judgement points per keyword were calculated. If a keyword’s CVR is between 0.0 and 0.5 and the mean of judgment points is greater than 1.5, then the keyword is accepted [ ].
Building of Categories
Since the depression lexicon categories were based on the symptoms described in the DSM-5 and ICD-10, the current lexicon is comprised of 13 symptom categories: mood, interest, appetite and weight, sleep, psychomotor agitation, psychomotor retardation, fatigue, guilt and self-esteem, concentration, suicide, anxiety, alcohol and substance use, and histrionic behavior.
Focus Group Discussions
From the focus group discussions, words typically used to express depressive symptoms were gathered. As seen in, 27% (21/78) of the responses indicate being “sad” and “lonely” as major descriptions of people with depression. Specifically, the words “sad”, “unhappy”, and “loneliness” as well as the word “loner” were used most often. In Filipino/Tagalog, some describe the experience of depression as having no focus (“tulala”, “lutang”, “malayo ang iniisip”) and as being disturbed or messed up (“wala sa sarili”).
|Responses||n (%)||Cumulative %|
|Sad; lonely; unhappy||21 (27)||27|
|No focus; disturbed||11 (14)||41|
|Isolation; lack of interest; low interaction||7 (9)||50|
|Sleep problems||7 (9)||59|
|Hopelessness; loss of meaning in life||6 (8)||67|
|Fatigue; stressed||5 (6)||73|
|Uneasiness; instability||4 (5)||83|
|Eating problems||2 (3)||92|
|Low self-esteem||2 (3)||95|
|Have no emotional support||1 (1)||99|
|Pretending to be happy||1 (1)||100|
As seen in, 56% (37/66) of the responses suggest that people who are depressed reveal their depression online. SNS serve as emotional outlets, thus allowing them to verbalize their feelings and release their sadness, frustrations, and problems. This capability to express emotions is somewhat foreshadowing, especially for those about to become clinically depressed. However, 20% (13/66) of the responses indicate that depression may be disguised by expressing opposite emotions. Still, 9% (6/66) of the responses suppose that people with depression use SNS and text messages as a signal and a means to reach out for help.
|Responses||n (%)||Cumulative %|
|Social network sites are emotional outlets||37 (56)||56|
|Hide depression or show the opposite of depression||13 (20)||76|
|Social network sites and text messages signal help||6 (9)||85|
|Sadness and troubles are expressed via social network sites (mildly depressed > severely |
|Seek attention via social network sites or text messages||3 (5)||99|
|Simply follow trends||1 (1)||100|
As seen in, 35% (18/50) of the responses indicate that they are able to recognize people with depression from SNS posts and text messages through the use of sad words. These may include not only personal messages but also quotations or lyrics of sad songs. In addition, 24% (12/50) of the responses claim that they do notice someone experiencing depression, particularly via the changes in behavior and topics of messages as observed in the history of postings and group messages or messages sent to predefined contacts, usually consisting of close friends. However, depressed people, even though they may post texts online, are thought to be difficult to talk with and are hesitant to open up about their depression in face-to-face interactions, as 8/50 (16%) responses point out that people with depression may be very aloof and keep quiet about their condition. They may not answer truthfully when asked directly about their feelings (despite having posted sad texts online). Depressed people have difficulty expressing themselves when confronted but would post messages, seemingly asking people for help or putting forth a message and reaching out for people.
|Responses||n (%)||Cumulative %|
|Presence of “sad” words||18 (36)||36|
|Change in behavior or topics||12 (24)||60|
|Asking for help||8 (16)||76|
|Only recognized if the reader is in a close relationship with the depressed individual||5 (10)||86|
|Postings or text messages as insincere or inconsistent||4 (8)||94|
|Might be misinterpreted||2 (4)||98|
|Only mental health professionals can recognize||1 (2)||100|
Interviews with Mental Health Professionals
From the interviews with 5 mental health professionals, it was confirmed that language in any form, written or verbal, is able to reveal signs of depression. The following points have emerged. First, clients or patients usually tell them when they are depressed. Some of the words, phrases, or sentences typically used by the clinically depressed are “can’t eat”, “can’t sleep”, “just going through the motions”, “no purpose or meaning”, “not interested”, “I’m tired all the time”, “I feel worthless”, “guilty”, “It’s not going to work whatever I do”, Nothing will change“, ”My head/stomach/muscles ache/s“, I’m losing weight”, and “I’ll never get well”.
Second, description of symptoms, such as that of the DSM and depression tests, may differ from the exact expressions of the individual when communicating with others. For instance, suicidal thoughts among the participants may be expressed as “Lord, please take my life”. Another example is that some symptoms described in the diagnostic systems such as psychomotor retardation may be difficult to translate into words but can be represented by specific actions in sentences such as “I find it tiring to dress up” or “I have a hard time making decisions”. This suggests the possibility that the symptoms, as indicated in items of traditional paper-and-pencil tests, may be general descriptions or collective terms for the experiences of the individual. Personal descriptions of depression may differ from the words used in these tests.
Third, current and contemporary language styles play an important role in developing a lexicon. The use of emoticons and emojis in expressing feelings is a trend in SNS and text messaging. Depressed mood, for example, may be expressed using “:-(”, which may mean “sad” or “lonely”.
Finally, it is feasible to express depression through text messages, blogs, or SNS. A clinical psychologist with 13 years of experience handling young adults mentioned that such apps are new technologies adolescents use; however, to detect depression through these apps, developers must consider all the possible words that maybe be used. Due to the integration of mobile devices in daily activities, a mobile app for depression may be useful for individuals who need help.
Review of Depression Scales
Keywords gathered from the review of depression scales were mostly statements in the first-person point of view. The words were grouped according the depressive symptoms, as stated in the DSM-5 and ICD-10 (). Words with the highest frequency convey either guilt or lowered self-esteem. This is followed by depressed mood and decreased interest or pleasure. However, no items related to alcohol consumption or substance use and abuse were found in these scales.
|Depressive symptoms||Frequency (%)||Cumulative, %|
|Feelings of guilt and low self-esteem||148 (33.0%)||32.96|
|Depressed mood||103 (22.9)||55.90|
|Loss of interest or pleasure||40 (8.9)||64.81|
|Psychomotor agitation||27 (6.0)||78.62|
|Increase or decrease in appetite or weight||20 (4.5)||88.20|
|Diminished concentration||17 (3.8)||91.98|
|Sleep problems||15 (3.3)||95.32|
|Suicidal thoughts and behavior||12 (2.7)||98.00|
|Psychomotor retardation||8 (1.8)||99.78|
|Histrionic behavior||1 (0.2)||99.78|
|Consumption of alcohol||0 (0.0%)||100.00|
Spelling Variation of Keywords
A total of 823,869 spelling variations of keywords were generated. The spelling variations were extended by the researchers by determining all possible combinations of individual words within a keyword ().
|Category||Keyword||Variation 1||Variation 2|
|Depressed Mood||Can not stop crying||Cant stp crying||Cnt stop cryin|
|Interest||Want to detach||Wnt 2 detach||Wanna detach|
|Sleep||Trouble falling asleep||Trouble fallng asleep||Trble fallin aslp|
|Suicide||Take my life||Tke my life||Take my lyf|
|Guilt/self-esteem||I am worthless||I’m wrthless||Im worthlss|
Categorization of Keywords
The PiaP lexicon development phase yielded a database consisting of 13 categories patterned after DSM-5 and ICD-10 symptom criteria for depressive episodes. The categorization and frequency of keywords per category is shown in. For the lexicon draft, we were able to gather 1762 main keywords, 9655 derivatives with different tenses or arrangement of words of the main keywords, and 823,869 spelling variations. The keywords included negatively-valenced words like “sad”, “unworthy”, or “tired” which are almost always accompanied by personal pronouns such as “I”, “I’m” or “my” and in Filipino “ako” or “ko”.
|Categories||Main Keywords, n||Derivatives, n||Spelling variations, n||Total, n|
|Appetite and weight||216||1357||19,032||20,605|
|Guilt and self esteem||180||1000||30,067||31,247|
|Alcohol and substance use||63||315||20,875||21,253|
Content Validation by Experts
The CVR per keyword was computed and averaged to the CVR per PiaP category. The categories of appetite and weight (CVR=0.98), suicide (CVR=0.97), and guilt and self-esteem (CVR=0.94) obtained the highest ratios (). On the other hand, the lowest ratios were for the psychomotor retardation (CVR=0.80) and psychomotor agitation (CVR=0.78) categories. The mean of all CVRs yielded an overall CVI of 0.90. For the 8 validators, an acceptable CVR per item or keyword should be greater than or equal to 0.75. The overall CVI was determined to be 0.90.
The content validity testing resulted to 1498 keywords, 8911 derivatives of main keywords, and 783,140 spelling variations, with a total of 793, 553 keywords which comprise the test-run version of the PiaP lexicon ().
|Appetite and weight||0.98|
|Guilt and self-esteem||0.94|
|Alcohol and substance use||0.91|
|Categories||Main keywords, n||Derivatives, n||Spelling variations, n||Total, n|
|Appetite and weight||213||1353||19,018||20,584|
|Guilt and self-esteem||173||984||29,837||30,994|
|Alcohol and substance use||53||315||20,760||21,128|
The current study involves lexicon development and content validation for a lexicon of depressed words to be used in mobile mental health apps, such as the PiaP. In lexicon development, we found that (1) language used in contemporary avenues such as social media and mobile technology serve as channels for expressing depression-associated emotions while subtly asking for help and avoiding stigmatization; (2) depressive language consists of words with negative undertones plus pronouns pertaining to the self; and (3) gold standard depiction of depressive symptoms, such as that of tests and diagnostic systems, may be different from the exact verbal expressions of the individual experiencing it. In addition, the uniqueness of a group or of a specific culture’s language expressions should be taken into account. A measurement tool, in this case a depression lexicon, with a high content validity shows its potential in detecting the domain, which are depressive symptoms.
The focus group discussions with the college students pointed to the idea that with the use of social media incorporated in their mobile phones, it becomes easier to simultaneously express one’s self and to relate and reach out to others. Language use is analyzed in detecting depression and depression proneness as shown in a study using an essay writing activity with students expressing their thoughts and feelings about college . With social media offering a platform where language can be studied, it is even possible to identify people with depression in their free-texts and novel ways of communication [ , ]. College students’ disclosure of feelings or emotions related to depression can likewise be determined through status updates and postings on Facebook [ ]. Students who may be experiencing depression spend more time on Facebook and could reveal more about themselves online rather than interpersonal communication [ ]. Previous studies also support our focus group discussion findings in terms of recognizing depressive symptoms through increasing levels of negative emotions [ ], reading more tips and facts about depression [ ], hinting at signs of hopelessness and meaninglessness and posting ruminating behavior, and expressing suicidal feelings or communicating suicide-related behavior on social media [ - ].
Depressive language consists of words with negative undertones plus pronouns pertaining to the self as evidenced by keywords gathered from focus group discussions, interviews with professionals, and depression tests. Our interviews and discussions with mental health professionals confirmed that depressed clients often have a negative description of themselves, of their situation and environment, and of their lives and futures that may be revealed in social media. These findings are supported by Beck’s cognitive theory , and Pyszczynski and Greenberg’s self-focus model [ ], which describe individuals experiencing depression as using more words with negative connotations and often making self-references when sharing their experiences. Whenever people are posting in social media, sending emails or text and instant messages, reacting to a friend’s status message, or using search browsers, the schema, negative or positive, with which their minds operate can be observed regardless of the purpose or intention of communication through the words they choose to type in their mobile devices.
Gold standard depictions of depressive symptoms, such as that of tests and diagnostic systems, may be different from the exact verbal expressions of the individual experiencing depression. This is supported by a study by Neuman and colleagues when they created the Pedesis system by gathering depressive words as experienced and expressed by individuals who have depression (eg, blogs and social media) . People have vast ways of expressing depression, which may not be encompassed by a list created by experts or by diagnostic systems, such that the classification provided by the DSM or ICD. Even descriptions made by mental health professionals mainly consist of symptoms and signs rather than expressions of depression [ ]. Culture or generational differences makes it even more complicated when trying to make a representation of expressing depression in words. With novel platforms for communication such as mobile technology and social media, it is indeed challenging to encompass the variety of ways people may express themselves [ , ]. A guideline of what to look for is provided by the gold standards but it is in the examination of the lexical expressions of people experiencing depressive symptoms that depression can be adequately characterized [ , , ]. In addition, the current study’s findings show the texting behaviors of Filipinos, particularly among college students, involve abbreviating or changing the spelling of words, which, to an outsider appears almost incomprehensible. The inclusion of abbreviations or spelling variations increased the bulk of keywords, but since text analysis relies on the lexicon for detecting the symptoms, disregarding this cultural and generational peculiarity, would make it impossible to capture relevant depressive language.
The content validation procedure is a measure of validity of the lexicon in detecting words that can be related to depressive symptoms using text inputs. The data suggests that depressive language can express the majority of the symptom categories. This might indicate that symptoms more cognitive in nature are better expressed in verbal behavior. Schemas supporting such symptoms influence automatic thoughts that become observable in language use or choice of words. However, the identification of motor or physiological symptoms, such as psychomotor retardation and agitation, was less obvious. These symptoms are likely to be better observed in bodily movements and gestures and reported by others. Individuals who exhibit these symptoms may not be able to notice them themselves.
With its sound content validity, the depression lexicon adequately represents words that can be related to depressive symptoms and may be able to help in the detection of symptoms using text analysis in the mobile mental health app PiaP. An ongoing study involves a test trial of PiaP with selected college students and will investigate the individual keywords and their ability to discriminate and detect individuals who may have depressive symptoms through the psychometric process of item analysis. In addition, data will be used to calculate the internal consistency of the depression lexicon. Further trials with college students will also determine the normative structure of the population chosen.
The study has significance in the fields of mental health (eg, clinical psychology, psychiatry, counseling) and health informatics by seeking to deliver not just a psychometrically sound instrument but also a highly usable tool for the assessment of depression. Mobile phones with app capabilities have enabled ordinary individuals to do almost anything and everything with ease. The flexibility and proliferation of mobile phone software programs have allowed professionals in many fields in science and medicine to accomplish tasks related to gathering pertinent physiological data from their patients to aid diagnosis and help monitor the progress of physical or pharmacological treatment . Psychologists, psychotherapists, guidance counselors, and other mental health professionals may take advantage of this timely approach to monitor their clients, patients, or students.
The field of mHealth has much potential in developing countries [, ], even more so mobile mental health, which in some places is still in its infancy. The mental health profession has been heavily relying on traditional and conventional assessment tools such as pen-and-paper tests, observations, and interviews to gather data. Mobile mental health solutions such as the PiaP may contribute to the diagnosis process in a proactive way as well as monitor current symptomatology. Together with other technology-based tools that allow EMA, our approach may help provide increased statistical power of datasets gathered due to the amount of observations available. With the introduction of the new technique in screening for depressive symptoms described in this paper, the mental health field may begin exploring other possibilities like providing treatment aids with the use of mHealth technology. This may be accomplished in collaboration with experts and professionals in mobile technology and computer science.
Lastly, this study is intended to support suicide prevention among students by contributing a dependable and suitable tool for guidance counselors and mental health experts in the academe. Although depressive signs and symptoms may or may not lead to suicide, it is advantageous for the early detection of signals (such as non-fatal self-harm behaviors) to facilitate prompt action.
Prior Works in Lexicon Development
The lexicon development used in the PiaP has taken into consideration the processes employed in creating earlier lexica from other researches to be able to provide a comprehensive representation of the depressive language used by college students.
First, the current depression lexicon underwent an exhaustive process using both the top-down and bottom up processes. Pedesis  was built using the bottom-up process only, by harvesting the Web for metaphors in which the target terms (ie, depression) embedded are used by individuals who may or may not be experiencing depressive symptoms. De Choudhury et al [ ] built a lexicon of depression-related terms based on the social media platform Twitter. Emotex [ ] includes unigrams, emoticons, negations, punctuations and hashtags, emotions and moods in social media language, and utilized words from regular individuals to build their lexicon. On the other hand, in the development of the current lexicon used by PiaP, descriptions from mental health professionals and established literature in the field of emotion studies as well as words depicting depression in diagnostic system such as DSM and ICD were also included.
Second, the developed lexicon is culturally sensitive, having included language and jargon used by a specific group of people, particularly college students. In addition, the app also analyzes emoticons in SNS and abbreviations or spelling variations of text inputs which are novel ways of expressing feelings and emotions. De Choudhury et al  utilized a dataset which classified emotions in various technological contexts and used a list of mood words from a blogging website. It also incorporated various lexica for positive and negative emotions and for basic emotions, but novel words used in social media communication such as emoticons, emojis, or abbreviations were not included. In Xue and colleagues’ study [ ], although the tweets they examined were a combination of English and Chinese words, the lexicon they built only included English words pertaining to stress, negative emotion, degree, and negation. However, Chinese tweets needed to be translated to English before submitting to text analyses.
Third, the initial validation of the lexicon offers content-related evidence for its psychometric property of validity. Pedesis  employs inter-rater or inter-judge reliability testing; however, they did not test for domain representativeness of the words in the lexicon itself.
Current and Future Work
Currently, PiaP research in the Philippines is working with college students to establish construct validity against tests and scales for depression and related constructs such as BDI-II, CES-D scale, Satisfaction with Life Scale, and Affect Balance Scale. This approach may show the possible correlations with constructs similar or related to depressive symptoms. Also, PiaP is being administered to selected college students to gather data for item analysis which will examine the usage of words and its ability to discriminate between individuals who are depressed or non-depressed. Furthermore, PiaP data will be used to establish normative structure for college students. In Germany, a third depression lexicon (in German) is being initialized. In addition, work is being performed on the behavioral indicators of depression using sensors and voice markers. Future research will also include the incorporation of physiological test plug-ins such as mobile electroencephalography (mEEG).
By utilizing the ubiquity and functionalities of mobile technology, the immediate detection of symptoms of one of the most prevailing clinical disorders worldwide may be improved using text analysis. The generation of this depression lexicon is exhaustive, utilizing both top-down and bottom-up approaches. The breadth of keywords used in text analysis also incorporates the characteristic expressions of depression and its related constructs by a particular culture and age group. A content-validated mobile health app, such as PiaP, may help augment early detection of depression signs by providing a lexicon, which effectively characterizes words related to depressive symptoms.
We would like to thank, from Germany, Mr Paul Smith and Ms Sarah Winter for developing the PiaP prototype and Mr Eugen Seljutin and Mr Marko Jovanović for the additional technical support, and, from the Philippines, Ms Myra Gurango and Mr Jovan Lorenz Torres for the initial Filipino translations, Mr Angelo Dullas for gathering additional data for the keyword spelling variations, and Dr Francine Rose de Castro-Bofill for co-moderating our focus group discussions.
Conflicts of Interest
Multimedia Appendix 1
Paper Presentation at the 11th Biennial Conference, Asian Association of Social Psychology (AASP) and 52nd Convention, Psychological Association of the Philippines (PAP) 2015, Cebu City, Philippines, 19 -22 August.PDF File (Adobe PDF File), 1MB
Multimedia Appendix 2
Paper presentation at the InPACT International Psychological Applications Conference and Trends 2016, Lisbon, Portugal, 30 April - 02 May.PDF File (Adobe PDF File), 234KB
- Whiteford HA, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, et al. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. Lancet 2013 Nov 9;382(9904):1575-1586. [CrossRef] [Medline]
- World Health Organization. Health for the World's Adolescents: A Second Chance in the Second Decade. 2014. URL: http://apps.who.int/adolescent/second-decade/files/1612_MNCAH_HWA_Executive_Summary.pdf [accessed 2015-09-09] [WebCite Cache]
- Lee RB, Sta MM, Estanislao S, Rodriguez C. Factors associated with depressive symptoms among Filipino university students. PLoS One 2013;8(11):e79825 [FREE Full text] [CrossRef] [Medline]
- Cairns KE, Yap MB, Pilkington PD, Jorm AF. Risk and protective factors for depression that adolescents can modify: a systematic review and meta-analysis of longitudinal studies. J Affect Disord 2014 Dec;169:61-75. [CrossRef] [Medline]
- Tuisku V, Kiviruusu O, Pelkonen M, Karlsson L, Strandholm T, Marttunen M. Depressed adolescents as young adults - predictors of suicide attempt and non-suicidal self-injury during an 8-year follow-up. J Affect Disord 2014 Jan;152-154:313-319. [CrossRef] [Medline]
- Moskowitz DS, Young SN. Ecological momentary assessment: what it is and why it is a method of the future in clinical psychopharmacology. J Psychiatry Neurosci 2006 Jan;31(1):13-20 [FREE Full text] [Medline]
- aan het Rot M, Hogenelst K, Schoevers RA. Mood disorders in everyday life: a systematic review of experience sampling and ecological momentary assessment studies. Clin Psychol Rev 2012 Aug;32(6):510-523. [CrossRef] [Medline]
- Clement S, Schauman O, Graham T, Maggioni F, Evans-Lacko S, Bezborodovs N, et al. What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychol Med 2015 Jan;45(1):11-27. [CrossRef] [Medline]
- Li H, Zhang T, Chi H, Chen Y, Li Y, Wang J. Mobile health in China: current status and future development. Asian J Psychiatr 2014 Aug;10:101-104. [CrossRef] [Medline]
- Gulle RD. Inkcanto: Read this and save a life. 2012. URL: http://www.interaksyon.com/lifestyle/inkcanto-read-this-and-save-a-life [accessed 2015-09-09] [WebCite Cache]
- Miguel-Baquilod M. Global School-Based Student Health Survey (GSHS) in the Philippines 2003-2004. 2004. URL: http://www.who.int/chp/gshs/2004_GSHS_PHL_Final_Report.pdf [accessed 2015-09-09] [WebCite Cache]
- Redaniel MT, Lebanan-Dalida MA, Gunnell D. Suicide in the Philippines: time trend analysis (1974-2005) and literature review. BMC Public Health 2011;11:536 [FREE Full text] [CrossRef] [Medline]
- Minor KS, Bonfils KA, Luther L, Firmin RL, Kukla M, MacLain VR, et al. Lexical analysis in schizophrenia: how emotion and social word use informs our understanding of clinical presentation. J Psychiatr Res 2015 May;64:74-78. [CrossRef] [Medline]
- Bucci W, Maskit B, Hoffman L. Objective measures of subjective experience: the use of therapist notes in process-outcome research. Psychodyn Psychiatry 2012 Jun;40(2):303-340. [CrossRef] [Medline]
- Coppersmith G, Dredze M, Harman C, Hollingshead K. From ADHD to SAD: Analyzing the Language of Mental Health on Twitter Through Self-Reported Diagnoses. 2015 Presented at: 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality; 2015 June 5; Denver, Colorado.
- Beck JS. Cognitive Behavior Therapy, Second Edition: Basics and Beyond. New York, NY: The Guilford Press; 2011.
- Romero N, Sanchez A, Vazquez C. Memory biases in remitted depression: the role of negative cognitions at explicit and automatic processing levels. J Behav Ther Exp Psychiatry 2014 Mar;45(1):128-135. [CrossRef] [Medline]
- Haaga DAF, Solomon A. Production-based assessment in cognitive-behavioral therapy. In: Brown GP, Clark DA, editors. Assessment in Cognitive Therapy. New York, NY: Guilford Press; 2014:50-68.
- Baddeley JL, Daniel GR, Pennebaker JW. How Henry Hellyer's use of language foretold his suicide. Crisis 2011;32(5):288-292. [CrossRef] [Medline]
- Li TM, Chau M, Yip PS, Wong PW. Temporal and computerized psycholinguistic analysis of the blog of a Chinese adolescent suicide. Crisis 2014;35(3):168-175. [CrossRef] [Medline]
- Pyszczynski T, Greenberg J. Self-regulatory perseveration and the depressive self-focusing style: a self-awareness theory of reactive depression. Psychol Bull 1987 Jul;102(1):122-138. [Medline]
- De Choudhury M, Gamon M, Counts S, Horvitz E. Predicting Depression via Social Media. 2013 Presented at: 7th International AAAI Conference on Weblogs and Social Media; 2013 Jul 14-18; Boston, MA URL: https://www.aaai.org/ocs/index.php/ICWSM/ICWSM13/paper/download/6124/6351 [WebCite Cache]
- Gaggioli A, Riva G. From mobile mental health to mobile wellbeing: opportunities and challenges. Stud Health Technol Inform 2013;184:141-147. [Medline]
- Kemp S. Digital, Social & Mobile Worldwide in 2015. 2015. URL: http://wearesocial.net/tag/statistics/ [accessed 2015-09-09] [WebCite Cache]
- Umali E, McCool J, Whittaker R. Possibilities and expectations for mHealth in the Pacific Islands: insights from key informants. JMIR Mhealth Uhealth 2016;4(1):e9 [FREE Full text] [CrossRef] [Medline]
- Nielsen Company Philippines Inc. The Asian Mobile Consumer Decoded. 2014. URL: http://www.nielsen.com/ph/en/insights/news/2014/asian-mobile-consumers.html [accessed 2015-09-14] [WebCite Cache]
- Labrague LJ. Health Science Journal. 2014. Facebook use and adolescents' emotional states of depression, anxiety, and stress URL: http://hypatia.teiath.gr/xmlui/bitstream/handle/11400/1481/817.pdf?sequence=1 [WebCite Cache]
- Varshney U. A model for improving quality of decisions in mobile health. Decision Support Systems 2014 Jun;62:66-77. [CrossRef]
- Brian RM, Ben-Zeev D. Mobile health (mHealth) for mental health in Asia: objectives, strategies, and limitations. Asian J Psychiatr 2014 Aug;10:96-100. [CrossRef] [Medline]
- Epstein J, Bequette A. Smart phone applications in clinical practice. J Ment Health Couns 2013 Oct;35(4):283-295 [FREE Full text] [CrossRef]
- Torous J, Friedman R, Keshavan M. Smartphone ownership and interest in mobile applications to monitor symptoms of mental health conditions. JMIR Mhealth Uhealth 2014;2(1):e2 [FREE Full text] [CrossRef] [Medline]
- Yang C, Linas B, Kirk G, Bollinger R, Chang L, Chander G, et al. Feasibility and acceptability of smartphone-based ecological momentary assessment of alcohol use among African American men who have sex with men in Baltimore. JMIR Mhealth Uhealth 2015;3(2):e67 [FREE Full text] [CrossRef] [Medline]
- Garcia C, Hardeman RR, Kwon G, Lando-King E, Zhang L, Genis T, et al. Teenagers and texting: use of a youth ecological momentary assessment system in trajectory health research with latina adolescents. JMIR Mhealth Uhealth 2014;2(1):e3 [FREE Full text] [CrossRef] [Medline]
- Rude S, Gortner E, Pennebaker J. Language use of depressed and depression-vulnerable college students. Cogn Emot 2004 Dec;18(8):1121-1133. [CrossRef]
- Whitehill JM, Brockman LN, Moreno MA. “Just talk to me”: communicating with college students about depression disclosures on Facebook. J Adolesc Health 2013 Jan;52(1):122-127 [FREE Full text] [CrossRef] [Medline]
- Wright KB, Rosenberg J, Egbert N, Ploeger NA, Bernard DR, King S. Communication competence, social support, and depression among college students: a model of facebook and face-to-face support network influence. J Health Commun 2013;18(1):41-57. [CrossRef] [Medline]
- De Choudhury M, Counts S, Gamon M. Not All Moods are Created Equal! Exploring Human Emotional States in Social Media. : AAAI Press; 2012 Presented at: Sixth International AAAI Conference on Weblogs and Social Media; 2012 Jun 4-7; Dublin, Ireland.
- Park S, Lee SW, Kwak J, Cha M, Jeong B. Activities on Facebook reveal the depressive state of users. J Med Internet Res 2013;15(10):e217 [FREE Full text] [CrossRef] [Medline]
- Schwartz HA, Eichstaedt J, Kern M, Park G, Sap M, Stillwell D, et al. Towards Assessing Changes in Degree of Depression Through Facebook. : From Linguistic Signal to Clinical Reality; 2014 Presented at: Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality; 2014 Jun 27; Baltimore, MD p. 118-125.
- Nambisan P, Zhihui L, Kapoor A, Patrick T, Cisler R. Social Media, Big Data, and Public Health Informatics: Ruminating Behavior of Depression Revealed Through Twitter. : IEEE; 2015 Presented at: 48th Hawaii International Conference on System Sciences Conference (HICSS); 2015 Jan 5-8; Kauai, HI p. 2906-2913.
- Robinson J, Rodrigues M, Fisher S, Herrman H. Suicides and Social Media: Findings From the Literature Review. 2014. URL: http://www.youngandwellcrc.org.au/wp-content/uploads/2014/07/Robinson_2014_Suicide-and-Social-Media_Findings-from-the-Literature-Review.pdf [accessed 2015-09-10] [WebCite Cache]
- Bitsch J, Ramos R, Ix T, Ferrer-Cheng PG, Wehrle K. Psychologist in a Pocket: towards depression screening on mobile phones. Stud Health Technol Inform 2015;211:153-159. [Medline]
- Neuman Y, Cohen Y, Assaf D, Kedma G. Proactive screening for depression through metaphorical and automatic text analysis. Artif Intell Med 2012 Sep;56(1):19-25. [CrossRef] [Medline]
- American Psychiatric. Diagnostic and Statistical Manual of Mental Disorders, (DSM-5). Arlington, VA: Amer Psychiatric Pub; 2013.
- World Health Organization. The ICD-10 Classification of Mental and Behavioural Disorders: Clinical Descriptions and Diagnostic Guidelines. Geneva: World Health Organization; 1992.
- Kaplan RM, Saccuzzo DP. Psychological Testing: Principles, Applications, and Issues. Belmont, CA: Wadsworth Publishing; 2012.
- Effendi M, Matore EM, Khairani AZ. Assessing Content Validity of IKBAR Among Field Experts in Polytechnics. Aust J Basic App Sci. 2015. (7) p. 255-257 URL: http://ajbasweb.com/old/ajbas/2015/April/255-257.pdf [WebCite Cache]
- Wilson FR, Pan W, Schumsky DA. Recalculation of the Critical Values for Lawshe's Content Validity Ratio. Meas Eval Couns Dev 2012 Mar 14;45(3):197-210. [CrossRef]
- Ayre C, Scally AJ. Critical Values for Lawshe's Content Validity Ratio: Revisiting the Original Methods of Calculation. Meas Eval Couns Dev 2013 Dec 13;47(1):79-86. [CrossRef]
- Miller LA, Lovler RL, McIntire SA. Foundations of Psychological Testing: A Practical Approach. Thousand Oaks, CA: SAGE Publications; 2012.
- LAWSHE CH. A Quantitative Approach to Content Validity. Pers Psychol 1975 Dec;28(4):563-575. [CrossRef]
- Hasan M, Rundensteiner E, Agu E. EMOTEX: Detecting Emotions in Twitter Messages. EMOTEX: ASE; 2014 Presented at: 2nd ASE International Conference on Big Data Science and Computing/ 6th ASE International Conference on Social Computing/ 3rd ASE International Conference on Cyber Security; 2014 May 27-31; Standford, CA p. 27-31.
- Xue Y, Li Q, Li J, Feng L, Clifton DA, Clifford GD. Detecting adolescent psychological pressures from micro-blog. In: Health Information Science. New York, NY: Springer International Publishing; 2014:83-94.
- Huguet A, McGrath PJ, Wheaton M, Mackinnon SP, Rozario S, Tougas ME, et al. Testing the feasibility and psychometric properties of a mobile diary (myWHI) in adolescents and young adults with headaches. JMIR Mhealth Uhealth 2015;3(2):e39 [FREE Full text] [CrossRef] [Medline]
|BDI-II: Beck’s Depression Index II|
|CES-D: Center for Epidemiologic Studies-Depression|
|CVI: content validity index|
|CVR: content validity ratio|
|DSM-5: Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition|
|EMA: ecological momentary assessment|
|ESM: experience sampling method|
|ICD-10: The ICD-10 Classification of Mental and Behavioural Disorders: Clinical Descriptions and Diagnostic Guidelines|
|mHealth: mobile health|
|PiaP: Psychologist in a Pocket|
|SNS: social networking sites|
Edited by G Eysenbach; submitted 27.10.15; peer-reviewed by C Yang, B Jeong; comments to author 03.04.16; revised version received 07.05.16; accepted 24.05.16; published 20.07.16
©Paula Glenda Ferrer Cheng, Roann Munoz Ramos, Jo Ágila Bitsch, Stephan Michael Jonas, Tim Ix, Portia Lynn Quetulio See, Klaus Wehrle. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 20.07.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.