This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.
Mobile electronic platforms provide exciting possibilities for health behavior promotion. For instance, they can promote smoking cessation, moderate alcohol consumption, healthy eating, and physical activity. Young adults in Sweden are proficient in the use of technology, having been exposed to computers, smartphones, and the internet from an early age. However, with the high availability of mobile health (mHealth) interventions of varying quality, it is critical to optimize the usability of mHealth interventions to ensure long-term use of these health promotion interventions.
This study aims to investigate the usability of an mHealth intervention (LIFE4YOUth) targeting health behaviors among high school students through heuristic evaluation and usability testing.
A preliminary version of the LIFE4YOUth mHealth intervention, which was aimed at promoting healthy eating, physical activity, smoking cessation, and nonrisky drinking among high school students, was developed in early 2019. We completed a total of 15 heuristic evaluations and 5 usability tests to evaluate the usability of the mHealth intervention prototype to improve its functioning, content, and design.
Heuristic evaluation from a total of 15 experts (10 employees and 5 university students, both women and men, aged 18-25 years) revealed that the major usability problems and the worst ratings, a total of 17 problems termed
High school students have high expectations of digital products. If an mHealth intervention does not offer optimal functions, they may cease to use it. Optimizing the usability of mHealth interventions is a critical step in the development process. Heuristic evaluation and usability testing in this study provided valuable knowledge about the prototype from a user’s perspective. The findings may lead to the development of similar interventions targeting the high school population.
Chronic diseases are the leading cause of death and disability worldwide. Globally, up to an estimated 80% of cases of heart disease, stroke, and type 2 diabetes and more than 30% of cancers can be prevented by reducing smoking, harmful alcohol use, improving diet, and engaging in regular physical activity [
Mobile platforms provide exciting possibilities for the promotion of health behaviors through mobile health (mHealth) interventions. Previous research has shown that interventions targeting multiple health behaviors at the same time might be effective in improving the general lifestyle among adults [
Young adults are referred to as digital natives; many are proficient in the use of technology, having been exposed to computers, smartphones, and the internet from an early age. Indeed, young adults have the highest level of smartphone ownership among all age groups [
In a participatory design approach, research is undertaken with, rather than on, people, allowing researchers to gain an understanding of context-specific requirements and challenges [
This study aims to investigate the usability of an mHealth intervention (LIFE4YOUth) targeting health risk behaviors among high school students through heuristic evaluation and usability testing.
A preliminary version of the LIFE4YOUth mHealth intervention targeting health risk behaviors among high school students was developed in early 2019. LIFE4YOUth is one of 7 multiple mHealth interventions in the MoBILE (mHealth–Multiple Lifestyle Behaviors) research program (funded by Forte 2018-01410, principal investigator: ML) aimed at promoting healthy eating, physical activity, smoking cessation, and nonrisky drinking in 7 different populations in the health care system. The intervention includes information about health behaviors, tips on behavior change strategies, and activities. The formative research process of developing a novel multiple mHealth intervention is described in detail in a study protocol elsewhere [
Recruitment of participants for the heuristic evaluation was undertaken by members of the research team through paper advertisements (posters) in public areas at Linköping University, Sweden. The inclusion criteria for the heuristic evaluation were university students and employees, both women and men, aged 18 to 25 years, at the Faculty of Medicine and Health Sciences at Linköping University who were willing to participate and owned a mobile phone. Participants showed their interest by contacting the research leader via email or telephone.
School staff at 5 high schools selected for convenience in Östergötland, Sweden, were contacted via email and informed about the research project. Students from all schools were invited to participate in the usability testing. The recruitment of participants was performed by school staff through paper advertisements (posters and leaflets), digital advertisements (student email and school website), and information disseminated in the classrooms. The inclusion criteria for the usability tests were high school students, both female and male, aged 15 to 18 years, willing to participate and owning a mobile phone. High school students showed their interest by contacting the research leader via email or telephone.
All participants provided written informed consent before participation in all study procedures (heuristic evaluation and usability tests).
A total of 15 experts (10 employees and 5 university students) were recruited. For the heuristic evaluation, a set of 10 heuristics published by Nielsen [
We selected Nielsen’s 10 heuristics because they have been thoroughly tested and are quick and easy to apply. When applying heuristic evaluation, participants evaluate an app to find usability problems, assign them to a specific category of heuristics, and assign a severity rating [
The procedure itself was a two-part process: the participants first familiarized themselves with the system and its usage with reference to the materials and training provided by the assistant. Then, they independently applied the 10 heuristics, as given in
The introduction took place in the beginning of May 2019 in a conference room at Linköping University, Sweden. The evaluations were performed wherever the participants preferred and were sent to the research assistant in a prepaid envelope. After 10 days, a reminder was sent by a text message. Heuristic evaluations from all participants (n=15) were gathered during the last week of May 2019.
Visibility of system status: The system should always keep users informed about what is going on through appropriate feedback within reasonable time
Match between system and the real world: The system should speak the users’ language, with words, phrases, and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order
User control and freedom: Users often choose system functions by mistake and will need a clearly marked
Consistency and standards: Users should not have to wonder whether different words, situations, or actions mean the same thing
Error prevention: Even better than good error messages is a careful design, which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action
Recognition rather than recall: Minimize users’ memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialog to another. Instructions for the use of the system should be visible or easily retrievable whenever appropriate
Flexibility and efficiency of use: Accelerators—unseen by the novice user—may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions
Aesthetic and minimalist design: Dialogs should not contain information that is irrelevant or rarely needed. Every extra unit of information in a dialog competes with the relevant units of information and diminishes their relative visibility
Help users recognize, diagnose, and recover from errors: Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution
Help and documentation: Although it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large
0: Not a usability problem
1: Cosmetic problem only—need not to be fixed unless extra time is available
2: Minor usability problem—fixing this should be given low priority
3: Major usability problem—important to fix, should be given high priority
4: Usability catastrophe—imperative to fix this before product can be released
The usability test consisted of a human-computer interaction evaluation, which focused on the perceptions and performance of users in a laboratory setting. It entailed the completion of 30 goal-oriented tasks by targeted end users [
The participants were asked to complete the 30 tasks, and the test manager asked the participants to explain their actions as they performed them using a think-aloud method [
Go to the start page and explain what you think of the layout
Where can you find support to drink less?
Log what you drink. Explain how you interpret the graphs for the alcohol consumption you logged
You know that people feel healthier after exercising and want to find out more about getting started. How would you look for this information?
If you want to plan a physical activity, how do you find that activity, and specify your level of involvement?
You can set personal goals for your eating habits under the “Timeline” tab. Explain how to set your goal and what the graph shows you
What information is there under the “Risks” tab in the Diet module? Does it give you a good overview?
Go to the Smoking module. What do you think about the scope of the information provided on the first page?
If you want to know more about the benefits of quitting smoking, how would you go about this?
The 10 heuristics were pooled, and the identified problems were categorized as major issues [
The usability tests were recorded and transcribed verbatim by a professional transcription company. All transcripts were checked and validated by the first author (UM). Analysis of the video recordings was inspired by program theory development using an inductive approach, taking both verbal and visible conduct into consideration [
The analysis of the SUS score was conducted according to the scoring strategy of Brooke [
The heuristic evaluation resulted in a total of 121 usability problems and 131 heuristic violations reported by 15 participants. The usability problems identified through heuristic evaluation are summarized and presented by place of occurrence (eg, alcohol, diet, physical activity, and smoking modules as well as the start page), number of heuristic violations, and mean severity ratings in
The line in
Of the 10 types of heuristic violations depicted in
Number of usability problems identified presented by place of occurrence, counts of heuristic violations identified by the 15 participants.
Presentation of the frequencies of 10 heuristic violations sorted by Nielsen’s heuristics reported by the 15 participants.
Violations | Number of participants |
Visibility | 4 |
Match | 33 |
User control | 6 |
Consistency | 32 |
Error | 8 |
Recognition | 8 |
Flexibility | 21 |
Aesthetics | 10 |
Recover | 5 |
Help | 3 |
Most severity ratings were in the major severity category, indicating that fixing the problem should be given high priority (severity rating 3). The problems categorized with the highest severity rating were in the alcohol and diet modules. Most of the minor usability problems, that is, fixing those problems should be given low priority (severity rating 2), were also in the alcohol and diet modules. There were also a total of 17 problems reported as usability catastrophes across all modules, indicating the imperative to fix the problem before the product can be released (severity rating 4).
When analyzing the nature of usability problems, the heuristic evaluation revealed that major usability problems and catastrophic ratings concerned shortcomings in displaying easy-to-understand information to the users or technical errors. Examples of these types of usability problems provided by participants are as follows:
Navigation unclear. There’s a constant mixture of headings [in the various modules], unclear headlining, different selectable functions, sometimes with hidden text. You don’t know where you are
If the diagram’s x- and y-axes don’t have any labels, you don’t understand because the table headings are unclear and don’t stand alone.
In the figures it’s unclear which direction the scales go in, you don’t understand the colors, it’s confusing.
Several choices of wording, and the concepts are difficult and complicated. Think about having simple, consistent wording and simpler language.
It’s not possible to save all the choices you’ve entered, you can’t change your choices. The planning disappears when you browse through the tabs.
You can’t go back easily. In other words, it’s not possible to find your way “home” easily, there’s no home button.
Number of severity ratings (minor, major, and catastrophic) for each module as reported by the 15 participants.
A total of 5 high school students, 3 adolescent girls and 2 adolescent boys, participated in the usability testing. The analysis of the data from the transcripts revealed 3 main categories of barriers that limited usability: (1) design, (2) content, (3) and functionality.
And then I wonder if you could make it a bit more fun or something /.../ to have some background color other than white.
Perhaps it would be good to have images? I don’t think it captures the user’s interest much [without images].
Regarding the font size, participants preferred a larger font size and suggested clearly defined headlines:
/.../ an intervention that looks a bit half-hearted and where things aren’t, like, centered or whatever. Perhaps it looks less professional than it actually is, and so you don’t trust it as much /.../ because you’ve put so much effort into it being scientifically correct, so you could put a bit of effort into it looking good or, like, fixing it.
/.../ and bigger, clearer headings. I think bullet points are good, easy to read /.../ I don’t think you get particularly excited using it.
The second category refers to both the quality and quantity of intervention
/.../ there’s a lot of text. It gets a bit, like, too much text to manage to read through everything. /.../ you don’t want there to be too much text. It should be, like, quite quick and easy.
You can quite easily get tired with a lot of text.
Sometimes less is more /.../ if it’s simpler, it’s easier.
The quality of the content referred to the terminology whereby language was perceived as too heavy, too complicated, and difficult to comprehend. Indeed, words, phrases, and concepts were perceived as unfamiliar and not tailored to the target age group. Thus, engaging with the content was too taxing and limited the usability:
What on earth is “moderate level” and what on earth is “strenuous level”?
“Dietary index”? What does “dietary index” mean?
Very strange words. People don’t use them at all!
The third category
I think fewer stages would be good.
Yes, because otherwise perhaps it’s a bit so-so, that you go in and first you can click there, and then click there /.../ in other words, it gets too much.
Participants stated that time was precious and they did not want to spend time navigating unnecessarily, such as entering what they had been eating or how many activity minutes they had participated in. Thus, engagement with the app must be effective and targeted to facilitate usability:
You should be able to get an overview extremely quickly.
Like, how many times I ate fruit or berries last week? Yes, I might have eaten fruit once. I might have eaten it ten times, I might not have eaten it at all. I don’t know. It was a bit difficult.
Participants also described the importance of a logical flow between different components of the app. In addition, the features that guided or prompted navigation of the intervention could facilitate usability:
You need a bit more help orientating yourself, where you are. Or some kind of main menu that comes up, and then you can tap on alcohol and after you’ve tapped on that subheadings appear. Then you can choose between them.
Maybe [it would help if] everything is in categories instead, and you tap, and maybe then it appears. Not showing everything there from the start ... instead, you can go into what you’re interested in.
Furthermore, relating to the logical flow within the intervention, there was a desire for consistency between the different modules to improve the usability of the intervention:
Because maybe you can’t have the same subheadings for everything, but that they’re still very consistent, that there are reminders and text messages, then they should be in all [modules] so you can get familiar with it and find things.
The results of the SUS scores are given in
Result of the System Usability Scale.
Questions | P1a | P2 | P3 | P4 | P5 | Average |
1. I think that I would like to use this app | 4 | 4 | 3 | 2 | 3 | 3.2 |
2. I found the app unnecessarily complex | 1 | 0 | 2 | 1 | 1 | 1 |
3. I found the app easy to use | 1 | 3 | 3 | 2 | 2 | 2.2 |
4. I think I would need support from a technical person to use this app | 4 | 4 | 3 | 3 | 4 | 3.6 |
5. I found the various functions in this app were well integrated | 2 | 3 | 4 | 2 | 2 | 2.6 |
6. I thought there was too much inconsistency in this app | 1 | 2 | 3 | 2 | 2 | 2 |
7. I would imagine that most people would learn to use this app very quickly | 0 | 3 | 4 | 3 | 4 | 2.8 |
8. I found the app very cumbersome to use | 1 | 3 | 3 | 2 | 4 | 2.6 |
9. I felt very confident using the app | 1 | 2 | 3 | 1 | 4 | 2.2 |
10. I needed to learn a lot of things before I could start using this app | 4 | 4 | 3 | 4 | 4 | 3.8 |
SUSb score (sum×2.5) maximum 100 | 47.5 | 70 | 77.5 | 55 | 75 | 66.6 |
aP1-5: person 1-5.
bSUS: System Usability Scale.
As described in our previous protocol study for a participatory design [
The heuristic evaluation revealed that the major usability problems and the catastrophic ratings concerned shortcomings such as information display and comprehensive information, meaning that the intervention needed to speak the users’ language with consistent information that appears in a natural and logical order for the users. The results from the usability testing showed that design (aesthetics and clarity), content (quality and quantity), and functionality (effortless and logic flow) enabled usability. The findings of this study are consistent with those of previous research, which found that participants wanted features that reduced the amount of time and effort required from them [
Engagement was an issue closely related to usability. For instance, participants explained that when the intervention did not appear to have a logical flow, they would quickly cease to use it. Previous research has stressed that engagement refers both to how a user interacts with technology and their emotional response to it [
According to previous research, the most important factors during the design process are flexibility and responsiveness to the input and feedback of the target audience [
A limitation of the study is the small and partly nonrepresentative sample that highlights the need for caution when interpreting the results. The results cannot be used for far-reaching conclusions.
Combining heuristic evaluation and usability testing is a strength of this study. Heuristic evaluation provided insights to developers about potential usability problems, particularly in terms of identifying problems with user interface usability. The results from the heuristic evaluation were also used as inspiration to create tasks applied in the usability tests. Usability tests provided knowledge regarding whether specific tasks could be performed in the sequences of actions they were designed for to give direct input into how real users used the system.
The heuristic evaluations were performed wherever the participants preferred, for example, in the participant’s home. This was done to facilitate participation and to optimize that the participants felt no time pressure. Hence, the validity of the data could not be controlled for. This study was not conducted to identify every usability problem with the mHealth intervention but instead to show how heuristic evaluation and usability testing with a small number of users could identify a large proportion of usability problems and assist in making significant improvements to an mHealth intervention targeting multiple health behaviors. The methods used were valuable in identifying not only major areas and themes that needed modification but also smaller, easily fixed problems that users encountered.
Through participatory design using heuristic evaluation and usability testing, this study resulted in in-depth knowledge regarding the aspects of intervention content and structure that end users (eg, high school students) considered important. This knowledge can be used when designing an mHealth intervention targeting multiple health behaviors. In summary, heuristic evaluation showed that the major usability problems and the catastrophic ratings concerned information display and language use. Usability testing showed that design (aesthetics and clarity), content (quality and quantity), and functionality (effortless and logic flow) enabled usability. This knowledge is valuable in guiding further development of a final version of the novel multiple mHealth intervention program LIFE4YOUth.
Protocol used for participants to report usability problems tied to the place of occurrence, usability problem, problem description, heuristics violated, and severity rating scoring.
mobile health
randomized controlled trial
System Usability Scale
This study was funded by a grant from the Swedish Research Council for Health, Working Life and Welfare (Forte 2018-01410, principal investigator: ML).
MB owns a private company that develops and distributes lifestyle interventions for use in health care settings.