To begin to understand how design affects usability and effectiveness of glanceable displays, we conducted an online simulation-based experiment that manipulated 3 design dimensions of glanceable feedback displays: stylization, salience, and granularity. In the experiment, stylization and salience were crossed in a 2×2 factorial structure, while granularity was varied across multiple levels, ranging from showing progress in 1% to 20% increments. A total of 202 Amazon Mechanical Turk (MTurk) participants were randomly assigned to 1 of 16 interface conditions. In each condition, participants were presented with a smartwatch display showing feedback on progress toward a daily step goal and were asked to estimate the level of progress that was shown.

The primary aim of the study was to investigate how the design dimensions influenced users’ accuracy in assessing goal progress displayed on a smartwatch at a glance. The primary outcome was the error in participants’ estimates of the displayed progress. A secondary aim was to examine how each design dimension affected perceived usability and acceptability.

Given our goal of systematically evaluating the impact of design variations on users’ ability to rapidly comprehend step goal progress at a glance, we opted to conduct an online simulation-based experiment. Like any research methodology, this approach presents a set of trade-offs. Its most obvious limitation is the lack of ecological validity—the fact that participants did not interact with a real smartwatch in their natural environments. However, given our research question, this limitation is less critical than it might first appear, and it is made up for by the method’s advantages, including feasibility and scalability. Our rationale for using an online experiment was threefold.

First, the primary question addressed by this study is how variations in design influence participants’ ability to accurately perceive the level of progress toward a goal. This question concerns visual perception (ie, the ability to perceive small differences between representations) and interpretability of different types of visualization (ie, the ability to tell what the feedback display is trying to show). Such questions have routinely been studied using surveys and online experiments, including in the context of graphical representations for communicating health risks and results from medical tests [16-21]. Importantly, for our research question, an online experiment provides a conservative measure of the potential impact of different feedback designs, since participants are able to focus their attention on the experimental task without the distraction of other tasks that may interfere with how accurately they perceive feedback in the midst of everyday life.

Second, an online experiment enabled us to feasibly study the impacts of design with a high level of statistical rigor. Because we were concerned that small details of each design could impact performance in unforeseen ways, we created a total of 16 different interfaces. Since we expected the differences in objective performance across interfaces to be fairly small, we determined that over 200 participants would be needed to estimate these effects (power calculations are presented below), and that such a number could only be reasonably obtained by using an online platform such as Amazon MTurk. Conducting an equivalent study in the field would be prohibitively expensive, with the time and resource costs far outstripping the importance of the research question the study was trying to answer.

Third, using a simulation greatly facilitated our measurement approach. To measure user performance, we needed to compare participants’ estimates of their step goal progress to a “true” value; in addition, we needed to be able to ensure that participants were making a reasonable effort to adhere to the study procedures. Given this need for the objective “ground truth” for comparison and to know the precise state of the watch display when the participant was engaging with the experimental task, we determined that a simulation was necessary. Controlling the “glancing” behavior in the wild would be impossible, and while moderated sessions in a user testing laboratory could have been used in place of a simulation, they would not allow us to scale effectively without huge expenses that would not be merited by the expected significance of the results.

To frame the overall experiment, participants were told that they had recently bought a smartwatch, and they were using it to track their steps as they tried to increase their activity and walk 10,000 steps per day. Each experimental task was further set up with a brief scenario: participants were told that they had an important video call in a few minutes and that while they were waiting for the meeting to start, they decided to play an online game of Memory. To make sure they were not late for the meeting, they were asked to tap the spacebar a few minutes after they started playing the game to check the time, as they were told that the simulation time did not flow at the same rate as real-world time.

Tapping the spacebar brought up an image of a smartwatch that displayed time and step-goal progress using the design for the condition to which the participant was randomized. The image of the display was sized to approximate the size of a smartwatch viewed from a foot and a half (50 cm) away, the approximate distance of the wrist from the eyes when a person looks at a watch. The smartwatch image remained on the screen for 5 seconds, after which it was hidden to mimic the act of putting the arm down (Figure 1). As soon as they saw the time and activity progress, participants could resume playing the game (ie, the game remained active even while the smartwatch display was showing) in order to mimic the return to the primary task after glancing at a watch. Overall, 15 seconds after the smartwatch display was hidden, the game was interrupted, and participants were shown a 2-question questionnaire asking them to report (1) the time they saw on the display and (2) the percentage of completion of the daily step goal shown on the watch.

All study procedures were hosted on a study website, which was linked from the task we created on the Amazon MTurk platform. When they entered the study website, participants were presented with consent information that described the study procedures, after which they were presented with a brief training that explained the display design for their assigned condition. After reading the training material, participants were given a training task where they were asked to estimate the progress on the daily step goal shown on the display that used their assigned design. After they entered their estimate, the system provided the correct answer to further assist with learning how to interpret the display accurately.

Following the training, participants were presented with 3 experimental tasks. Each experimental task followed the structure outlined in the Experimental Task section. However, each scenario was contextualized to represent a different time of day—morning, midday, or afternoon—and featured a distinct level of step-goal completion when the smartwatch was displayed (Figure 2). After they completed all 3 experimental tasks, participants were asked to complete a questionnaire that assessed their perceptions of the usability and interpretability of the smartwatch display they used in the experiment.

After they submitted the questionnaire, participants were issued a code that they could enter in the MTurk task form to request payment for completing the task. Participants were compensated US $2.50 for completing the experiment. The compensation was chosen to match a US $15/hour wage, adjusted for the duration of the task (~10 minutes). The full flow of the experiment is shown in Figure 3.

When they entered the experiment, each participant was automatically randomized, with the probability of 1 out of 16 (6.25%), to 1 of the 16 experimental designs of progress feedback. All instructions for interpreting the display, the test session at the start of the experiment, as well as each of the 3 experimental tasks used the feedback design to which the participant was randomized.

Each participant completed 3 experimental tasks that used different levels of step-goal progress: 36%, 72%, and 116%. The numbers were chosen to represent low, moderate, and goal-met-and-exceeded progress on the daily step goal, and to not be round, easily guessable numbers. To control for order effects, the order of the 3 tasks was randomized for each participant. This meant that, for instance, a participant may see the interface that represents 116% goal attainment first, followed by interfaces that show 36% and 72% goal attainment. Each level of progress was shown to each participant only once.

The objective of our study was to find out how the tested design dimensions affect the ability of participants to accurately assess the smartwatch progress feedback display at a glance. Consequently, our primary outcome was the absolute error of the participants’ progress estimates, calculated as the absolute difference between the progress estimate and the true progress value, which allowed us to assess the overall accuracy of participants’ ratings without taking into consideration whether their ratings were above or below the true value of the goal progress.

Our secondary outcomes included the self-reported ratings of the displays’ usability and usefulness. These were assessed using a questionnaire that participants completed after they completed all 3 experimental tasks. The questionnaire was based on the System Usability Scale [33] and a questionnaire used by Shami et al [34] to assess the perceived strengths and weaknesses of glanceable displays. The constructs assessed and the questions used are shown in Textbox 1. Questions Q6-Q7 and Q12-Q14 were adapted from the System Usability Scale; questions Q1-Q5 and Q8-Q11 (except Q10) were adapted from Shami et al [34]. All questions were assessed on a 5-point Likert scale that ranged from strongly disagree to strongly agree. Q10 was an attention check question that we used to filter out Amazon MTurk workers who were answering questions without paying even minimal attention.

Recruitment was done through the Amazon MTurk platform. We created an MTurk task that linked to the study website that hosted the experiment. Eligible participants were required to be 18 years of age or older and could take part in the experiment only once. Attempts at participating more than once were recognized based on the MTurk worker ID and were filtered out prior to the data analysis.

To ensure high data quality—MTurk workers sometimes click through a task to get paid without actually engaging with the task [36,37]—prior to conducting the experiment, we developed 2 attention checks that we used as criteria for a priori exclusion of participants from the dataset. The first check used a metric called string edit distance [38], which measures the similarity of 2 strings in terms of the number of operations (insertions, deletions, etc) that are required to transform one string into the other. In our pilot work, we found that the edit distance of 2 effectively identified responses to the goal-progress estimation that were clearly fallacious (eg, the same number was provided as an answer to all 3 experimental tasks). We, therefore, removed from the analytical data set all participants for whom the edit distance between their estimates of goal progress and the correct answer was greater than or equal to 2. The second exclusion criterion used the attention check question in the poststudy questionnaire (Q10 in Textbox 1). The question asked participants to skip the question and leave it unanswered. Participants who answered the question (ie, who clicked on any radio button on the Likert scale for that question) were excluded from the dataset.

The study protocol was approved by our university’s Behavioral and Social Sciences Institutional Review Board.

We recruited 400 participants on Amazon MTurk, targeting adults who were interested in smartwatches and physical activity. Participants were recruited in 4 waves of 100 participants each. Each wave of recruitment was followed by the exclusion of participants who violated one of the criteria for data integrity validation. We would then start the next wave if the required number of participants had not been reached.

After applying data integrity criteria, 202 participants remained in the final dataset. Of these, 35.64% (n=72) identified as female. In terms of smartwatch usage, 29.2% (n=59) were current users, 38.61% (n=78) were former users, and 32.18% (n=65) had never used a smartwatch. The age distribution was skewed toward younger and middle-aged adults. The largest age group was 25-34 years (n=87, 43.07%), followed by 35-44 years (n=57, 28.21%) and 45-54 years (n=33, 16.34%). Participants aged 18-24 years accounted for 6.93% (n=14) of the sample, while those aged 55-64 years and older comprised 3.47% (n=7) and 1.98% (n=4), respectively.

Table 1 presents the results of the GEE analysis examining the effects of salience and stylization on the accuracy of participants’ goal-progress estimates. Since the outcome variable was square-root transformed, the reported β coefficients should be squared to interpret effects on the original scale. To make the results interpretable, we note that the median absolute error across all conditions was 14 (IQR 4-47) points. Our findings indicate that stylization significantly reduces the accuracy of progress assessments. Compared to low-stylized representations, high-stylized ones led to an increase in error by 4.52 points, rising from 8.98 for low-stylized conditions to 13.50 points for high-stylized conditions. We found no significant impact for salience, however.

Table 2 presents the results of a GEE analysis examining the effects of salience, stylization, and granularity on the accuracy of participants’ goal-progress estimates. By including all 3 factors in a single model, we isolated the marginal impact of each variable. In this combined linear model, granularity demonstrated a marginally significant independent effect on estimation error (P=.08). However, because granularity was unevenly distributed across feedback designs—with half of the panels using the finest granularity level (1%)—and preliminary diagnostics suggested a nonlinear relationship, we conducted a follow-up analysis. As reported in Table 3, we used a model specifically parameterized with a quadratic term to account for this complexity. These results revealed a U-shaped relationship: error decreased as granularity decreased, up to a point, after which further decreases in granularity led to higher error.

We examined 5 levels of feedback display granularity: 1%, 4%, 5%, 10%, and 20%. Given that the outcome variable was square-root transformed, predicted values were squared to interpret error on the original scale. Decreasing granularity from 1% to 4% reduced the predicted error from 20.19 to 18.24, a decrease of 1.95 points. A further decrease in granularity from 4% to 5% resulted in a smaller reduction of 0.36 points, lowering the predicted error to 17.88 at the 5% granularity level. Transitioning from 5% to 10%, the error began to increase slightly, rising by 0.26 points to 18.14 at 10% granularity. When granularity decreased from 10% to 20%, the predicted error sharply increased by 12.32 points, reaching 30.46. These findings reveal a U-shaped relationship between granularity and error, suggesting that when feedback is presented on smartwatch-sized screens, moderate levels of granularity enhance accuracy, whereas high granularity and overly coarse representations hinder users’ ability to estimate goal progress correctly.

According to the model, the optimal granularity level is around 7.1%. Since this value was not directly tested in our feedback designs, it suggests that the lowest predicted error would occur at a granularity level of 5% or 10%.

As shown in Textbox 1, we grouped the subjective perception items from the postquestionnaire into 7 categories: noticeability, comprehension, relevance, complexity, division of attention, attractiveness, and usability. To examine the impact of stylization and salience on participants’ subjective perceptions of the feedback displays, we conducted linear regression analyses.

Tables 4-9 present the results of the linear regression analyses examining the effects of stylization and salience on participants’ perceptions of the interfaces, including comprehension, relevance, complexity, division of attention, attractiveness, and usability (see Table S1 in Multimedia Appendix 1 for the impact of stylization on noticeability). Perceptions were measured using a 5-point Likert scale across 7 categories. Stylization was found to negatively influence 6 of the 7 dimensions of user perceptions: comprehension, relevance, complexity, attention, attractiveness, and usability. When using high-stylized feedback designs, participants reported greater difficulty comprehending the information (–0.486 points; P=.003), perceived the interface as more complex (–0.690 points; P<.001), and felt that less relevant information was provided (–0.476 points; P=.002). Additionally, they found it harder to maintain focus or shift attention between the interface and the game (–0.378 points; P=.022), and rated the interfaces as less attractive (–0.338 points; P=.047) and less usable (–0.532 points; P=.001).

Compared to modeling granularity as a continuous predictor, treating it as a categorical variable provided a clearer understanding of how different granularity levels impacted subjective responses. Tables 10-12 present the results of linear regression analyses examining the effects of granularity on participants’ perceptions of the interfaces, including comprehension, complexity, and division of attention (see Tables S2-S5 in Multimedia Appendix 1 for the impact of granularity on noticeability, relevance, attractiveness, and usability).

Among the 5 granularity levels (1%, 4%, 5%, 10%, and 20%), the 10% granularity level yielded the most favorable perceptions. Relative to the 1% granularity group, participants in the 10% group reported the interface as easier to comprehend (+0.656 points; P=.003), less complex (+0.766 points; P=.005), and easier to maintain focus on (+0.523 points; P=.03).

To further evaluate the performance of the 10% group, we reran the analysis using the 10% group as the reference category (Tables S6-S12 in Multimedia Appendix 1). Consistent with the previous comparison, we found that participants in all other granularity groups rated the feedback designs as harder to comprehend (1%: –0.656; P=.003; 4%: –0.591; P=.04; 5%: –1.091; P<.001; 20%: –0.702; P=.03). Participants in the 1% (–0.766 points; P=.005) and 5% (–1.195 points; P=.002) groups also perceived the feedback designs as more complex compared to the 10% group. Similarly, participants in the 1% (–0.523 points; P=.03) and 5% (–0.913 points; P=.006) groups found it more difficult to maintain focus, compared to the 10% group. Additionally, compared to the 10% group, participants in the 5% group perceived the feedback design as harder to notice changes in (–0.634 points; P=.024), less informative (–0.779 points; P=.009), and less useful overall (–0.848 points; P=.006).

Prior smartwatch experience was associated with decreased accuracy in progress assessment. As shown in Table 1, compared to people who have never used a smartwatch before, being a previous smartwatch user—but not a current user—increased the error by approximately 7.46 points, with the error rising from 11.13 to 18.59 points (median absolute error 14, IQR 4-47 points).

Despite exhibiting lower accuracy in progress estimation compared to participants who had never used a smartwatch, previous smartwatch users reported more favorable subjective perceptions across several measures. As seen in Tables 6, 8, and 9, compared to participants who had never used a smartwatch, previous smartwatch users found the interface easier to maintain focus on (+0.874 points, P<.001) and more useful (+0.566 points, P=.002). However, they also perceived the feedback displays as more complex, rating them 0.754 points lower in simplicity P=.001). In Table 13, compared to current smartwatch users, previous smartwatch users also perceived the feedback design as harder to notice changes in (–0.773 points, P=.001).

Current smartwatch users did not show significant differences in accuracy compared to participants who had never used a smartwatch, but reported more positive subjective perceptions. Specifically, as seen in Table S1 in Multimedia Appendix 1, Tables 7 and 9, they rated the feedback designs as easier to notice changes in (+0.354 points; P=.053), easier to maintain or shift attention (+0.559 points; P=.007), and more useful (+0.512 points; P=.007), all on a 5-point scale.

Our study examined how variations across 3 design dimensions—stylization, salience, and granularity—influenced participants’ ability to interpret glanceable smartwatch feedback displays.

Results showed that high stylization negatively affected the accuracy of progress assessments. Regarding granularity, levels of 5% and 10% could be seen as a sweet spot, resulting in lower interpretation errors. Participants in the 10% granularity group also reported more favorable subjective perceptions compared to those in other granularity conditions. Salience did not influence either the accuracy of interpreting progress or participants’ subjective perceptions.

Interestingly, participants who had previously used a smartwatch but were no longer actively using one performed worse in assessing progress compared to those who had never used a smartwatch. However, both previous and current smartwatch users reported more positive subjective evaluations of feedback displays than participants who had never used a smartwatch.

Our results indicate that high stylization negatively affected interpretation accuracy, increasing progress errors by 4.52 points. This finding should be considered alongside several factors that may influence the real-world viability of stylized displays. First, the observed accuracy gap might narrow with increased familiarity; while our study used a brief practice period, long-term use of a specific stylized interface could result in users learning to better interpret its visual patterns, potentially improving performance over time. Second, stylization may serve as a privacy-preserving layer, offering users a degree of discretion by preventing onlookers from easily discerning personal data. Third, the aesthetic qualities of such interfaces might boost user engagement and check frequency, which could enhance overall effectiveness [22,39,40].

In contrast, low-stylized displays are preferred when feedback precision is critical for decision-making. This is particularly relevant in safety-critical contexts or data-driven health management, such as monitoring blood glucose levels [41] or tracking caloric balance. In these scenarios, the requirement for high levels of precision makes stylized feedback a suboptimal choice. Ultimately, the use of stylization involves a trade-off between the necessity for interpretative accuracy and the user’s potential preference for discretion and aesthetic engagement.

We found that the relationship between granularity and accuracy in progress assessment is nonlinear. Accuracy initially increases as granularity decreases (eg, from 1% to 5%), but the rate of increase slows down between 5% and 10% and then reverses direction. This pattern indicates the presence of an inflection point—an optimal granularity level, estimated at 5%-10% in our data—at which accuracy peaks. Beyond this point, accuracy declines as granularity increases further.

These results are not completely unexpected, as there are situations where high levels of feedback granularity may not be helpful. For instance, in very small feedback displays, such as smartwatch complications, feedback granularity can quickly exceed users’ ability to visually discriminate between states of the display. In such situations, a lower-granularity display may more effectively convey changes in the state through more easily perceived jumps than a high-granularity display. Similarly, in many contexts, there are limits to the level of granularity that is needed for decisions. For instance, Hawley et al [16] and Zikmund-Fisher et al [20] found that variations in the granularity of the graphs about the risk of taking different medications did not have a significant impact on people’s choices of which medicine to take. The researchers used n-in-100 (ie, among 100 people, the number of people who have side effects) as low granularity and n-in-1000 as high granularity. Their results indicated that when the level of granularity met the upper limit of the demand, participants’ assessments became unaffected by further increases in granularity.

Our result shows that the sweet spot for goal-progress feedback displays may be at 5% or 10% granularity, which aligns with previous research. For instance, Kong et al [42] observed that participants estimating the proportion between 2 rectangles (displayed in a constrained space) tended to round their answers to multiples of 5. Our results showed the same tendency—participants frequently reported progress estimates in multiples of 5. This suggests that interfaces designed with 5% or 10% granularity may support faster and more accurate assessments, as users can more easily map perceived progress to familiar reference points.

Taken together, these results suggest that pursuing very high granularity does not necessarily improve users’ ability to notice changes or make informed decisions. Conversely, very low granularity may increase errors in progress assessment and fall short in contexts where greater precision is required [41]. A granularity level of 5% or 10% appears to strike an effective balance—supporting precision, enabling users to notice changes, and reducing estimation errors.

Lastly, our results indicate that previous smartwatch users made more errors in perceiving progress compared to those who had never used a smartwatch. Specifically, their error was 7.46 points higher—representing the largest effect on interpretation accuracy observed in our study. While we are not sure why this discrepancy exists, one intriguing possibility is that people who are less effective at interpreting information displayed on smartwatches are more likely to abandon them. This possibility might be reflected in participants’ subjective perceptions. Compared to individuals who have never used a smartwatch, previous smartwatch users rated the feedback display as more complex (–0.754 points; P=.001; Table 6). Even when compared to current smartwatch users, previous smartwatch users still perceived the feedback display as more complex (–0.773 points; P=.001; Table 13). However, as shown in Table 7 and Table 9, previous smartwatch users also found the interface more useful (+0.566 points; P=.002; Table 9) and reported it was easier to maintain focus while using it (+0.874 points; P<.001; Table 7). Current smartwatch users similarly rated the feedback display as more useful (+0.512 points; P=.007; Table 9) and easier to focus on (+0.559 points; P=.007; Table 7). Notably, current users did not report any statistically significant negative perceptions of the feedback display interface.

If this is the case, researchers who develop smartwatch-based interventions may want to use prior smartwatch use as a tailoring variable for providing additional training or using alternative forms of intervention feedback. Investigating how predictive this variable is for other intervention effects could be an interesting avenue for future research.

Our experience in constructing these interfaces suggests a broader challenge regarding how to cleanly operationalize design dimensions when transitioning from theories to practical applications. While design dimensions such as granularity, stylization, and salience are conceptually distinct, their implementation in a real-world context like smartwatch displays may be inherently constrained by practical requirements.

For instance, the tension between exact estimations and low-granularity designs reflects the trade-off between accuracy and interpretability. For glanceable displays, while providing accurate data is important, designs need to remain interpretable to be efficiently accessed under cognitive load. While asking participants to estimate the range of progress might have aligned more naturally with low-granularity interfaces, using an exact estimation task across all conditions allowed us to observe how different designs influence this cognitive balance. This indicates that practical constraints can make it difficult to cleanly separate a single design dimension from its functional impact on performance. Another operationalization challenge for stylization and granularity is further discussed in the Limitations section.

These observations suggest that design dimensions that are theoretically distinct are often not orthogonal in practice. Operationalizing a single dimension in isolation, without accounting for these functional interdependencies, risks producing interfaces that are either visually incoherent or practically irrelevant.

In this paper, we presented a study that tested 16 glanceable feedback displays to examine the effects of 3 design dimensions—stylization, salience, and granularity—on the accuracy of progress assessment. Our findings show that stylization negatively affected both the accuracy of assessing goal progress and perceptions of comprehension, relevance, complexity, attention, attractiveness, and usability. In contrast, granularity levels of 5% and 10% may be optimal for displaying smartwatch feedback. We found no significant differences between high-salience and low-salience interfaces in terms of interpretive accuracy and subjective perceptions. Interestingly, participants who had previously used a smartwatch but were no longer using one exhibited larger errors in interpreting progress feedback. However, both previous and current smartwatch users reported higher subjective perceptions compared to those who had never used a smartwatch. Our results extend the research on glanceable feedback and can help researchers design effective glanceable smartwatch feedback displays.