This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Globally, the uptake of health screening is suboptimal, especially in men and those of younger age. In view of the increasing internet access and mobile phone ownership, ScreenMen, a mobile Web app, was developed to improve health screening uptake in men.
This study aimed to evaluate the utility and usability of ScreenMen.
This study used both qualitative and quantitative methods. Healthy men working in a banking institution were recruited to participate in this study. They were purposively sampled according to job position, age, education level, and screening status. Men were asked to use ScreenMen independently while the screen activities were being recorded. Once completed, retrospective think aloud with playback was conducted with men to obtain their feedback. They were asked to answer the System Usability Scale (SUS). Intention to undergo screening pre- and postintervention was also measured. Qualitative data were analyzed using a framework approach followed by thematic analysis. For quantitative data, the mean SUS score was calculated and change in intention to screening was analyzed using McNemar test.
In total, 24 men participated in this study. On the basis of the qualitative data, men found ScreenMen useful as they could learn more about their health risks and screening. They found ScreenMen convenient to use, which might trigger men to undergo screening. In terms of usability, men thought that ScreenMen was user-friendly and easy to understand. The key revision done on utility was the addition of a reminder function, whereas for usability, the revisions done were in terms of attracting and gaining users’ trust, improving learnability, and making ScreenMen usable to all types of users. To attract men to use it, ScreenMen was introduced to users in terms of
This study showed that ScreenMen was acceptable to men in terms of its utility and usability. The preliminary data suggested that ScreenMen might increase men’s intention to undergo screening. This paper also presented key lessons learned from the beta testing, which is useful for public health experts and researchers when developing a user-centered mobile Web app.
In the past decade, many Web-based interventions have been developed to improve health outcomes of the public. Web-based interventions not only have a wider reach but also are less labor intensive and less resource exhaustive as compared with conventional health interventions; in addition, they can be interactive, personalized, and fun, which makes learning more effective and ultimately leads to improved users’ health behavior. In addition, the impact of Web-based health interventions is further amplified with the flux of mobile technology into health care. The mobile phone, which is a good platform to deliver health care to people anytime and anywhere, is widely available and affordable these days, including in developing countries [
Web-based interventions could be deployed to improve health screening uptake as well. Health screening plays an important role to detect and treat diseases at an early stage, which leads to reduced mortality rate [
Many types of interventions to improve the uptake of health screening in men, including those using partners’ involvement, educational workshops, reminder phone calls, and letters have been evaluated [
In view of the increasing internet access and mobile phone ownership in Malaysia, as well as in the world [
Before ScreenMen was finalized, a beta testing was conducted. Beta testing aims to test a software with end users in a real-world setting to identify and rectify any potential issue before being released. This is particularly important for a mobile Web app as Web-based technologies are growing and changing rapidly [
Thus, this study aimed to evaluate ScreenMen with men from the community in terms of its utility and usability, as well as to present the key revisions made to improve the utility and usability in ScreenMen, on the basis of the feedback obtained.
This study used a mixed-methods design to evaluate the utility and usability of ScreenMen with end users. The mixed-methods approach helps to triangulate the findings using qualitative and quantitative methods. The qualitative assessment was done using the retrospective think-aloud method with the aid of a topic guide, which contained questions on utility and usability [
This study was conducted with healthy men from a banking institution in Kuala Lumpur, the capital of Malaysia. Unlike alpha testing with experts, which was done at the developer’s site, beta testing is conducted at the users’ settings, which was men’s working place in this study. Men from a banking institution were chosen as they were reported to have increased work-related stress, which affected their psychological as well as physical health, including smoking and alcohol overuse, depression, body posture issue, and visual problem [
The same banking institution where the needs assessment was conducted in the earlier phase was selected as the recruitment site for this beta testing. This study was approved by the banking institution. Men who have a mobile phone and were from the banking institution were recruited to participate in the beta testing. They were purposively sampled according to their job position, age, education level, and screening status to achieve maximal variation of the feedback on ScreenMen. A Microsoft Excel spreadsheet that contained the participants’ demography was used to plan the recruitment to ensure equal representation from each sampling criterion. Men who participated in the needs assessment phase were first contacted and arranged for interviews. Then, the snowballing method was used to recruit new participants, where the recruited participants were asked to recommend their colleagues to participate in the study. New participants were also included in addition to those who had participated in the needs assessment phase to gather more feedback on ScreenMen. All participants were reimbursed with MYR 50 (approximately US $ 12) for their time participating in this study, which took about an hour.
The sample size of a usability study is often small. Studies have shown that the optimum sample size to detect sufficient usability problem is 10 users [
ScreenMen is a mobile-responsive Web app aimed to be disseminated via smartphone. It aims to educate men, empower men, and improve men’s behavior on health screening. ScreenMen was developed to contain male-sensitive attributes (such as using car maintenance analogy), as well as evidence-based recommendation for health screening gathered from the USPSTF [
Learn: This section contains a short educational video to demystify the misconceptions on health screening, which were identified in the needs assessment.
Assess: This is an interactive section where users can interact with ScreenMen to assess their health risks and obtain personalized health advice, as well as the evidence-based health screening they need, on the basis of their health profile. There were 15 health conditions that were being assessed in ScreenMen, and they include obesity, unhealthy diet, physical activity, tobacco use, alcohol misuse, high blood pressure, diabetes, dyslipidemia, colorectal cancer, lung cancer, HIV, syphilis, hepatitis B, hepatitis C, and depression. This section is algorithm-driven and attempts to mimic a real-life clinical consultation with a doctor. A health report can be generated at the end of this section.
Ask: In this section, there is a list of frequently asked questions about screening, which men can read if they would like to have further clarifications about screening. The questions were developed on the basis of the comprehensive framework on barriers and facilitators to health screening in men [
Prepare: This section aims to prepare the users to undergo health screening by providing basic logistic information such as where to screen, when to screen, and cost of screening.
In-depth interviews (IDIs) and focus group discussions (FGDs) were conducted for data collection. At the time of the appointment, the researchers first briefed the participants about the study using a participant information sheet. The participants were encouraged to ask questions and were informed that they could stop the study at any time. Once agreed to participate, the participants were asked to sign a consent form and fill up the demography form, including intention to undergo screening. Then, the participants were given a smartphone with ScreenMen activated on the screen. They were asked to use it themselves and notify the researchers once they had finished using it. All on-screen activities were being recorded using a free screen recording app (AZ Screen Recorder by Hecorat). The researchers were present in the same room to observe the participants’ behavior when using ScreenMen and take field notes, as well as assist the users, only when necessary.
Postintervention questionnaire.
No. | Item | Strongly agree | Agree | Neither agree nor disagree | Disagree | Strongly disagree | |
1 | I think that I would like to use this website frequently | —a | — | — | — | — | |
2 | I found the website unnecessarily complex | — | — | — | — | — | |
3 | I thought the website was easy to use | — | — | — | — | — | |
4 | I think that I would need the support of a technical person to be able to use this website | — | — | — | — | — | |
5 | I found the various functions in this website well integrated | — | — | — | — | — | |
6 | I thought there was too much inconsistency in this website | — | — | — | — | — | |
7 | I would imagine that most people would learn to use this website very quickly | — | — | — | — | — | |
8 | I found the website very cumbersome to use | — | — | — | — | — | |
9 | I felt very confident using the website | — | — | — | — | — | |
10 | I needed to learn a lot of things before I could get going with this website | — | — | — | — | — | |
11 | Does the website help you to understand more about your health risks? | — | Yes | ||||
— | No | ||||||
12 | Does the website help you to understand more about health screening? | — | Yes | ||||
— | No | ||||||
13 | Do you intend to go for health screening in the future? | — | Yes, in the next 1 month | ||||
— | Yes, in the next 6 months | ||||||
— | Yes, in the next 1 year | ||||||
— | Yes, in the next 2 years | ||||||
— | Yes, in the next 5 years | ||||||
— | No, I do not intend to go for health screening | ||||||
14 | Would you recommend this website to your family or friends? | — | Yes | ||||
— | No |
aIndicate boxes for participant’s response.
Once completed, the participants were asked to answer the postintervention questionnaire, which contains the validated 10-question System Usability Scale (SUS) [
Then, retrospective think aloud with playback was conducted. Using a topic guide, the researchers started the interview by asking the participants to provide their overall opinion on the Web app; to comment on its contents and layout (usability); to explain how, if the Web app helped them to understand more about health and screening; and to suggest any other part of the Web app that can be improved. The on-screen recording was played to assist the participant in the retrospective think-aloud process. They were probed to comment on the content and layout when going through each section of ScreenMen. All conversations during the retrospective think aloud were audio-recorded. For FGDs, all procedures were similar to those of IDIs, except that only 1 participant was given the project’s mobile phone, whereas others used their own mobile phones and during the feedback session, and the ScreenMen Web app was projected on the screen and navigated by the researcher, page by page, to assist the retrospective think aloud, instead of playing the on-screen recording of each participant.
The analysis was performed after completing all data collection. Subsequently, the researchers met to discuss the issues and proposed the revisions to be done on ScreenMen. The qualitative data obtained were analyzed using a framework approach to systematically guide the revision of the Web app. After each interview or discussion session, the researchers discussed and compiled a list of comments and issues on ScreenMen, with the aid of the field notes taken. The researchers then listened to the audio recording to triangulate and check for additional comments. Unlike the usual approach in a qualitative study, the audio-recordings were not transcribed verbatim in this study as the purpose of this beta testing was to capture the users’ feedback on the Web app rather than to provide an in-depth understanding of the users’ experience. The list of comments compiled was then coded under utility or usability and by section (
For the quantitative data, all data were managed and analyzed using the IBM SPSS Statistics version 21. First, the SUS score from each participant and a mean SUS score for all participants were calculated. The SUS score was interpreted using the adjective rating scale developed by Bangor [
In total, 24 men participated in the beta testing with 14 IDIs and 2 FGDs (5 participants in each FGD). They were conducted from February to March 2017. The details of the participants are shown in
The participants found the ScreenMen useful as they could learn more about their health risks, what screening to go for, and what they could do to improve their health. The same was found for older men who had undergone screening, as ScreenMen contains information they never knew before, such as colorectal cancer and the unnecessary screening tests:
I like all of it, it tells you your health, everything about where you are (in terms of health), and what you should do to improve it.
Using this web, people know what diseases they should check
Now I understand about the importance of health screening. We don’t know that our lifestyle could actually affect our health. Using this website, you know what to be improved upon.
Some mentioned that they were glad to learn about the unnecessary screening tests. One participant suggested to highlight this more to ensure all users get it:
My key take home message from this website is some of the tests are unnecessary, for example, the liver or kidney test, as it may over or under detect the disease. Nowadays, there are a lot of external blood test centers, they normally package ECG, heart stress test, and everything together and sell you thousands of Ringgit (Malaysian currency). I didn’t know that those are actually unnecessary. So, this is something I got to know now. It’s good to know that those are actually not useful for screening. This information is a little secluded and need to be highlighted better.
A participant was glad as:
...it contains localized contents for us (Malaysian), unlike the UK or US websites.
One participant mentioned that using ScreenMen may trigger men to take care of their health:
ScreenMen is easy to use, can add more knowledge and act as a trigger to take care of health when going through the website, unlike those who do not receive anything and do not do anything about health.
Men also felt that ScreenMen was convenient to use.
We have limited time for screening. With this, we can check at anywhere, we can have the information and what can we do (to improve health). It is just like talking to a doctor or consultant.
It is good for people. People are always with handphone. With application like this, one doesn’t need to go anywhere, at home also can do, at office also can do.
will share this website with friends via Facebook Group.
On the other hand, 1 participant raised the issue that users may not use ScreenMen again after using it once:
It’s good. Will I use the website? Yes. But subsequently will I continue to use it repeatedly again, it remains a question mark.
The participants also suggested that a reminder function may be useful as users may not act instantly after using ScreenMen. Thus, the research team added a function where users can input their email, and an event entitled
Some participants suggested that it would be good to have a list of screening centers with phone numbers to make appointments on the website, as that may facilitate users to take action to screen. However, the research teams decided not to include the list to avoid being perceived as using it for commercial reason.
Characteristics of participants in the beta testing (N=24).
Characteristics | Statistics | |
20-29 | 7 (29) | |
30-39 | 10 (42) | |
40-59 | 7 (29) | |
Age (years), mean (range) | 37 (23-56) | |
Malay | 10 (42) | |
Chinese | 10 (42) | |
Indian | 3 (13) | |
Others | 1 (4) | |
Senior manager | 7 (29) | |
Officer | 5 (21) | |
Sales advisor | 5 (21) | |
Clerk | 7 (29) | |
Secondary school | 5 (21) | |
Certificate/diploma | 4 (17) | |
Degree | 13 (54) | |
Postgraduate | 2 (8) | |
Unmarried | 10 (42) | |
Married | 14 (58) | |
Yes | 9 (38) | |
iOS | 11 (46) | |
Android | 12 (50) | |
Windows | 1 (4) | |
Yes | 13 (54) |
Overall, participants mentioned:
[ScreenMen] is quite user-friendly, comfortable to look at, and not too cluttered. The interface is easy to understand, not too complicated.
The participants also felt assured to use the Web app as it was stated upfront that the Web app does not capture any identifiable information from them.
There were several key issues with revisions to improve ScreenMen, and they were grouped under 3 themes: (1) attracting and engaging users, (2) ensuring effective learning, and (3) catering for the widest range of users’ characteristics.
The home page of ScreenMen outlined the 4 key sections of the Web app. Some participants found that the home page contained too much information to read and felt that it may put off users. Thus, the home page was simplified to include only the main objective of what users may gain from this Web app (
Men were less interested in screening as they did not understand about screening and its importance. Describing the Web app as a platform to learn about screening did not interest the users. However, men wanted to be healthy, and they more readily received information that can keep them healthy. Thus, the objective on the home page was framed in terms of learning about users’ health risks and ways to stay healthy, instead of learning about health screening. Additional health information, such as erectile dysfunction and urinary symptoms, were also added as requested by men, to provide more information than just screening.
The participants felt that there was a lack of credibility on the home page. They mentioned that the credibility of Web app is crucial to gain users’ trust so that they continued to use the Web app. To address this issue, they suggested to enlarge the university’s logo on the home page.
The ScreenMen Web app attempted to attract men using Dr ScreenMen, a Superman-resembling doctor avatar at the home page. Although some liked Dr ScreenMen figure as it encourages them to be strong, especially those from lower educational level, others had no comment on the Dr ScreenMen figure. One participant suggested to make Dr ScreenMen provide various types of reaction, but this was not done because of technical complexity and the potential impact on Web loading time.
The home page of ScreenMen before and after revision.
The menus of ScreenMen before and after revision.
From the researchers’ observation, the Learn, Assess, Ask, and Prepare menus were unclear, and participants were confused about these concepts. For example, some thought that they could ask questions to a doctor at the
Users are allowed to navigate freely to any section of ScreenMen by using the icons on the home page. This was done to cater to users who had already understood the basics of health screening and the repeat users. However, some users were confused as they went to the third section directly from the home page and did not go through the first and second section. Thus, the navigation links on the home page were removed. Users who liked to skip any section could use the
Most participants agreed that the car maintenance analogy was very useful in helping them learn about health screening. The only comment on this was to use the word
At the time of the usability testing, it was found that some users lost their attention at the third section (Ask). The
ScreenMen was developed on the basis of the USPSTF guidelines. It advocates evidence-based screening and encourages users to avoid unnecessary screening. To fulfill the
User’s list of screening recommendation without and with emphasis of not recommended screening.
The newly added section to encourage users to avoid unnecessary screening.
Some participants commented that there was too much information to be read in ScreenMen, especially for people with a lower literacy level. The information in ScreenMen was thus simplified to present only relevant and brief information. Links for additional information were incorporated throughout the Web app for users who may want more information about certain topics.
Developed from the medical perspective, users had problems answering some of the questions in ScreenMen. For example, some participants had difficulties answering
Another example was the blood pressure reading. Most users could not remember their blood pressure reading but remembered that their blood pressure was normal from the previous health screening. To provide a more accurate advice, an option of
The health assessment section was developed to mimic a real-life clinical consultation with a doctor. This section starts with Dr ScreenMen greeting the users, obtaining users’ age, and followed by health assessment, topic by topic. Dr ScreenMen asks question and provides advice on each health condition on the basis of users’ answers. At the end of this section, users can view the summary of their health status with screening recommendation.
However, some participants commented that men who are busy may not like to go through this process and would prefer a shorter mode. Though the consultation mode is more ideal for learning as it breaks the session into chunks, a
Though ScreenMen was developed for men, some participants suggested that it could also be used by women as a woman might be the person taking care of her husband or father in a family. Some of the sentences were thus rephrased to accommodate female users; for example,
Malaysia consists of 3 main ethnic groups: Malay, Chinese, and Indian. There were only 2 languages available in ScreenMen for beta testing (English and Malay). Some Chinese participants mentioned that their parents may need the Mandarin version as they were not literate in English and Malay languages. The Mandarin language was thus added to ScreenMen. No issue was raised regarding having Tamil language on ScreenMen as Indians are usually literate in English and Malay.
Apart from language, some sections of ScreenMen might be sensitive to certain ethnic groups. For example, all users were assessed in terms of alcohol intake, which may not be relevant to Muslim users as alcohol intake is prohibited in the religion. However, the Muslim participants reassured the research team that it was not an issue as the option
Another concern was the sexually transmitted disease assessment. Personal information such as having multiple sexual partners, having sex with men, and injecting drugs was being asked of the users. However, the participants mentioned that they had no hindrance in answering these as no identifiable information was being recorded, and these were important for them to know.
Only 23 participants answered the postintervention questionnaire as 1 participant was called for work urgently. The details of the postintervention quantitative evaluation are shown in
For intention to undergo screening, 8 out of 23 men (35%) planned to attend screening earlier than intended after using the ScreenMen (no intention to 2 years, n=1; 5 years to 1 year, n=1; 2 years to 1 month, n=1; 1 year to 6 months, n=3; and 6 months to 1 month, n=2); 14 out of 23 men (61%) did not change; whereas 1 out of 23 men (4%) planned to screen later (1 month to 6 months). In terms of stage of behavior change, 4 out of 12 (33%) men, who were in precontemplation stage, changed to either contemplation or preparation stage after using ScreenMen (
Options of consultation or quick mode for health assessment.
Quantitative evaluation after using the ScreenMen (N=23).
Postintervention evaluation | Statistics | |
System Usability Scale score, mean (SD) | 76.4 (7.72) | |
Understand more about their health risks, n (%) | 23 (100) | |
Understand more about health screening, n (%) | 23 (100) | |
Will recommend ScreenMen to others, n (%) | 23 (100) | |
Earlier | 8 (35) | |
No change | 14 (61) | |
Later | 1 (4) |
Intention to screen by stage of behavior change before and after using ScreenMen (N=23).
Stage of behavior change | Preintervention, n (%) | Postintervention, n (%) |
Precontemplation (>6 months) | 12 (52) | 8 (35) |
Preparation/contemplation (≤6 months) | 11 (48) | 15 (65) |
Total | 23 (100) | 23 (100) |
This study found that ScreenMen is acceptable to men in terms of its utility and usability. The participants found ScreenMen useful as they could learn more about their health risks and the evidence-based health screening they should go for. They also felt that ScreenMen was convenient to be used and may trigger men to undergo screening. The quantitative data showed that many men planned to undergo screening earlier than initially intended after using ScreenMen, although there was no significant difference when analyzed on the basis of the stage of behavior change using McNemar test. The participants also felt that ScreenMen was user-friendly and comfortable to look at. The SUS score also indicated that ScreenMen had good usability.
Tackling the issue of health screening is not an easy endeavor, and ScreenMen had to be further improved in terms of utility and usability. The key improvement in terms of utility was the addition of a reminder function. Past studies have shown that reminder interventions including those using a letter, email, and short message service were effective in increasing screening uptake [
This study has taken a male-sensitive approach to improve men’s behavior. ScreenMen was developed specifically for men and incorporated with male-familiar contents such as the car maintenance concept and the
The outcomes of this utility and usability testing may appear differently if ScreenMen was developed as a mobile app instead of a mobile Web app. The reminder function would be easily built, and more interesting functions such as alert, monitoring function, daily health messages, and integration with social media can be included. However, the research team decided to develop ScreenMen in the form of mobile Web app for the ease of dissemination. Though this hindered having more useful functions in ScreenMen, reaching out to men is seen as a more important factor, as a health screening mobile Web app or mobile app is not something being sought after by men as they do not see the importance of health screening, unlike for exercise or diet apps. A Web app has a broader dissemination than an app as it can be accessed instantly without needing to download and install, can be shared quickly among friends, and can be viewed on a computer as well [
The findings from this beta testing reinforced the importance of conducting testing with end users. Though many iterations of testing were done with experts during alpha testing, some of the issues were not captured. For example, the fruit intake per day question was not seen as a problem to experts but posed difficulties for users to answer. Other than that, the experts felt that the amount of text was just right; however, it was still too much for some participants. Apart from testing with experts, the development team has also considered many usability guidelines such as the International Organization for Standardization and the International Electrotechnical Commission (ISO/IEC) 25010 software product quality model and Nielsen usability diagram [
There are several strengths and limitations in this study. The strength of this study was that we managed to sample men from a wide demographic range, which gave rise to maximal variations of the qualitative findings. The multifaceted approach used (quantitative and qualitative, observation and retrospective think aloud with playback, IDI and FGD) allowed the study to gather a rich amount of data and enabled data triangulation. With regard to limitations, the design of the utility section in the questionnaire limited the data analysis. The questions on
This study showed that ScreenMen is acceptable to men in terms of its utility and usability. Men are able to learn more about their health risks and screening via ScreenMen. The preliminary data suggested that ScreenMen might increase men’s intention to undergo screening and may potentially improve the actual uptake of health screening as well. Further evaluation in the form of randomized controlled trial should be conducted to determine the effectiveness of ScreenMen in improving the uptake of evidence-based health screening. Apart from that, this study also allowed further refinement of ScreenMen to improve its utility and usability. We have shared the key lessons learned from this beta testing, which might be useful for public health experts and researchers to develop user-centered mobile Web apps in the future.
The framework used for data analysis.
embodied conversational agent
focus group discussion
in-depth interview
Malaysian Ringgit
National Health and Morbidity Survey
System Usability Scale
United States Preventive Services Task Force
The authors would like to thank all participants and the banking institution for approving this study to be conducted with the staff. This study was funded by the University of Malaya Research Programme (RP041A-15HTM) and partially funded by the Postgraduate Research Grant (PG265-2016A). Publication of this paper was cosponsored by the
None declared.