Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, December 24 through Wednesday, December 26 inclusive. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 21.12.18 in Vol 6, No 12 (2018): December

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/12023, first published Aug 23, 2018.

This paper is in the following e-collection/theme issue:

    Original Paper

    Usability of a Personal Air Pollution Monitor: Design-Feedback Iterative Cycle Study

    1Washington State University, Spokane, WA, United States

    2University of Washington, Seattle, WA, United States

    3Washington State University, Everett, WA, United States

    4Washington State University, Seattle, WA, United States

    Corresponding Author:

    Glen E Duncan, PhD

    Washington State University

    Box 1495

    Spokane, WA, 99210-1495

    United States

    Phone: 1 5093587875

    Email:


    ABSTRACT

    Background: There is considerable evidence that exposure to fine particulate matter (PM2.5) air pollution is associated with a variety of adverse health outcomes. However, true exposure-outcome associations are hampered by measurement issues, including compliance and exposure misclassification.

    Objective: This paper describes the use of the design-feedback iterative cycle to improve the design and usability of a new portable PM2.5 monitor for use in an epidemiologic study of personal air pollution measures.

    Methods: In total, 10 adults carried on their person a prefabricated PM2.5 monitor for 1 week over 3 waves of the iterative cycle. At the end of each wave, they participated in a 30-minute moderated focus group and completed 2 validated questionnaires on usability and views on research. The topics addressed included positives and negatives of the monitor, charging and battery life, desired features, and changes to the monitor from each previous wave. They also completed a log to record device wear time each day. The log also provided space to record any issues that may have arisen with the device or for general comments during the week of collection.

    Results: The major focus group topics included device size, noise, battery and charge time, and method for carrying the device. These topics formed the basis of iterative design changes; by the final cycle, the device was reasonably smaller, quieter, held a longer charge, and was more convenient to carry. System usability scores improved systematically across each wave (median scores of 50-66 on a 100-point scale), as did median daily wear time (approximately 749-789 minutes).

    Conclusions: Both qualitative and quantitative measures showed an improvement in device usability over the 3 waves. This study demonstrates how the design-feedback iterative cycle can be used to improve the usability of devices manufactured for use in large epidemiologic studies on personal air pollution exposures.

    JMIR Mhealth Uhealth 2018;6(12):e12023

    doi:10.2196/12023

    KEYWORDS



    Introduction

    Considerable evidence proves that exposure to fine particulate matter (PM2.5) air pollution is related to various adverse health outcomes [1-9]. Despite regulation, PM2.5 remains a serious problem in the United States, where currently 20 counties affecting over 23 million people do not meet the federal PM2.5 standards [10], and even short-term increases in the PM2.5 levels may cause tens of thousands of excess deaths per year [9,11,12].

    Although progress has been made in elucidating the biological mechanisms for PM2.5-related health effects, the cost, complexity, and burden on study subjects have made it difficult to conduct personal exposure assessments to better understand effects of acute and chronic exposures, where and when exposures occur, and how individual lifestyle factors affect exposures in humans under free-living conditions. Indeed, most epidemiologic studies have not measured true personal exposures. Instead, they have relied upon measurements made at central monitoring sites as exposure “surrogates,” resulting in considerable concern within the exposure assessment community as to the impact of such exposure error on disease estimates [13]. More sophisticated geospatial models have been used to try to capture spatial variations within urban areas [14] but often, the assumption is that the modeled ambient concentration at a subject’s residential address is a reasonable estimate of personal exposure, which is also incorrect because individuals are exposed to multiple locations in the course of daily living; for example, in the few studies that have used personal exposure monitoring instruments, substantial variations were found among individuals living within the same urban area and even within the same neighborhood [13,15,16]. Moreover, individuals tend to spend close to 90% of their time in indoor environments [17], and this is often not considered in air pollution epidemiologic studies. Recent meta-analyses of the issue have concluded that characteristics of the participants and their microenvironments can greatly affect the representativeness of such proxies and that greater attention is needed to evaluate the effects of measurement error [18,19].

    The purpose of this study was to use the design-feedback iterative cycle to improve the usability of a portable PM2.5 monitor. This methodological paper describes the testing and refinement of the device for use in an epidemiologic study of personal air exposure measurements and clinical and biological outcomes in a large sample of twins recruited from a community-based registry. Although this study only reports on the usability aspects of the personal air pollution monitor, members of our team have published on the performance attributes of the sensor components for measuring PM2.5 and other endotoxins [20-25].


    Methods

    Recruitment

    Participants for this study were drawn from the community- based Washington State Twin Registry. We chose to use twin pairs for the design-feedback iterative cycle portion of the study because twin pairs would ultimately be recruited for the epidemiologic cohort portion of the study. Only twin pairs who did not reside at the same address were eligible for the study; this ensured differential environmental exposures. For the present iterative cycle portion of the study, only pairs within the Puget Sound (King, Snohomish, Pierce, and Kitsap counties) were recruited because in-person assessments were required. The larger epidemiologic cohort portion of the study will enroll twin pairs from a wider geographic extent.

    Three different versions of the personal air pollution monitor were given to participants across 3 cycles from February 2016 to August 2017. The same subjects took part in each of the 3 cycles; this ensured systematic feedback on the changes in the device over time. After each cycle, the research team used the feedback from the focus groups to modify the design of the monitor for use in the subsequent cycle.

    Device Description

    Briefly, all versions of the air pollution monitor were designed by researchers at the University of Washington for personal monitoring of PM2.5 exposures. The design requirements were to create a relatively small battery-operated wearable monitor that could provide continuous timestamped and geocoded data on PM2.5 exposures during the day (ie, at least 12 hours). Although the form factor and electronic design evolved with each version of the monitors tested in the study, generally, all monitors included an onboard real-time optical particle sensor, Global Positioning System (GPS), real-time clock, data logging to a memory card, 3-axis accelerometer for physical activity tracking, and sound sensor for noise exposure monitoring. Monitors also include a microblower and plastic cartridge assembly used to collect time-integrated sample particles throughout the period that the monitor is powered on.

    Measures and Procedures

    For the first cycle of the study, the twins came in for a study visit during which they received oral and written instructions on device use and for completing a daily wear log. The wear log was used to record wear time (start and stop times) for each day of the weeklong collection period. The log also provided space to record any issues that may have arisen with the device or any open-ended comments during the week of collection. They were also provided 2 validated questionnaires to assess device usability (the System Usability Scale, SUS [26]) and general views on research (Research Attitudes Questionnaire, RAQ [27,28]). The SUS contains 10 questions addressing the ease of use of the device. RAQ is a 7-question survey that measures the participants’ attitude toward medical research. Both SUS and RAQ use a 5-point Likert scale for each question with 1 being “strongly disagree” and 5 being “strongly agree.” For scoring SUS, 1 is subtracted from each odd item response and 5 from each even-numbered response. The converted responses are added up and multiplied by 2.5, providing a range of possible values from 0 (most negative experience) to 100 (most positive experience). The RAQ scale is created by summing the 7 questions to get a score that ranges from 7 to 28.

    Each twin carried on their person the air pollution monitor, an ActiGraph accelerometer (ActiGraph WGT3X-BT, Pensacola, FL) for objective measurement of physical activity, and a GPS monitor (Qstarz BT-Q100XT, Taipei, Taiwan) to place exposures within a space and time framework [29]. The physical activity and GPS data are not reported in this paper because the main purpose of wearing those devices in the iterative cycle portion of the study was to ensure that it was feasible to wear all 3 devices in the epidemiologic cohort portion of the study. Participants collected data for 1 week, at which time participants returned the devices and participated in an in-person focus group. The second and third waves proceeded in the same way; however, the study materials were sent to participants via FedEx, and the focus groups were conducted via teleconference. The focus groups lasted approximately 30 minutes and were moderated by 2 trained research staff. Topics addressed included positives and negatives of the air pollution device, device charging and battery life, desired features for subsequent versions, and pros and cons of changes to the device that occurred since the previous wave.


    Results

    The first focus group was completed by 5 twin pairs, the second by 4, and the final by 3 with the same participants carried over through each group. One pair was no longer interested in participating after the first wave of collection owing to increased responsibilities at work, and 1 pair was unavailable to participate in the final wave of collection because they were preparing for a multi-week vacation. The initial group of 10 consisted of 7 women and 3 men, the second group of 7 women and 1 man, and the final group of 5 women and 1 man. In total, 80% (8/10) participants were white, 80% (8/10) were aged >50 years, 80% (8/10) had completed some college, and 60% (6/10) were from households with incomes over the Washington state median based on Census data. Four of the twin pairs were monozygotic. A script for conducting the focus group was used to facilitate discussion. Notes were taken by a member of the research team, and the focus groups were recorded and transcribed by the research coordinator.

    Text from both the focus group transcripts and the open-ended comments in the data collection log was analyzed to identify the main topics of importance. We created a corpus of both positive and negative open-ended comments and then converted the corpus to a document term matrix to determine which words were used most frequently. As shown in Table 1, the topics that emerged from the analysis across the 3 cycles of collection were device size, noise level, battery and charge time, and the method for carrying the device (belt clip, lanyard, etc). Focus group comments showed relatively slight improvements between cycle 1 and 2 with larger improvements from cycles 2 to 3; for example, the research team made the device substantially smaller from cycles 1 to 3 (initial size 5.21×0.91×2.76 inches, final size 4.25×0.88×2.56 inches; Figure 1). A muffler was added after cycle 2 to reduce the noise level of the device (from the microblower used to collect particles). In addition to the muffler, the cycle 3 device was programmed for intermittent use of the microblower for particle sampling instead of constant sampling, which also reduced the overall noise level.

    The battery and charging protocol improved over the 3 cycles as well. After cycle 1, lights were added to the device to make it easier to determine when it was on and charging. The twins remarked on the helpfulness of the lights. After cycle 2, the battery was changed to extend its life and simplify the charging process. The new battery was expected to last for 16 hours, meaning the twins were able to run the device all day and then charge it overnight.

    A final major point of discussion was how the device was carried. In cycles 1 and 2, participants were provided with different types of belt clips. All participants had issues with the belt clips for both cycles; several participants dropped the device owing to the clips or because they could not figure out how to best attach the clips to the device. For cycle 3, the research team moved away from the belt clips and provided a lanyard for the device (Figure 2). The lanyard could be worn around the neck or attached to a bag. Reactions to the lanyard were positive, though 1 device dropped after falling out of the lanyard. The research team has already addressed this issue by securing the devices within the lanyard with a cable tie or attaching the lanyard clip through a corner strap hole on the monitor’s enclosure.

    Figure 3 illustrates a box plot of the median wear time over the 3 cycles, showing an increase from 749 minutes per day (range 122-931 minutes) in cycle 1 to 789 minutes per day (range 594-847 minutes) in cycle 3. Figure 4 illustrates a box plot of median SUS scores over each cycle with usability scores improving from 50 in cycle 1 to 66.2 in cycle 3. In both figures, variability is shown as 1.5 times the inter-quartile range (1.5×interquartile range).

    The RAQ scores demonstrated a favorable view of research in general; the average score for all participants was 24.3 (range 19-28). Moreover, focus group feedback demonstrated an interest in learning more about the purpose of the device and seeing the data being collected. Most of the participants did not feel they would want to own a device like this; however, they did bring up a few specific scenarios when the device would be helpful, such as if one had a respiratory issue or if one lived in an area with high levels of air pollution.

    Table 1. Major thematic topics and comments from focus groups over 3 consecutive design-feedback iterative cycles for testing a personal air pollution monitor.
    View this table
    Figure 1. Size comparison of a personal air pollution monitor in cycle 2 (left) to cycle 3 (right) of the design feedback iterative cycle.
    View this figure
    Figure 2. Lanyards were used in cycle 3 to allow participants to carry the air pollution monitor around the neck or attached to a bag.
    View this figure
    Figure 3. Median System Usability Scale scores (100 scale) over 3 consecutive design-feedback cycles. SUS: System Usability Scale.
    View this figure
    Figure 4. Median wear time (minutes per day) over 3 consecutive design-feedback cycles.
    View this figure

    Discussion

    Principal Findings

    This methodological paper described how the design-feedback iterative cycle was used to improve the usability of a personal air pollution monitor. In general, data gathered from the focus groups and questionnaires showed an increase in satisfaction and usability with the device over each successive iterative cycle. The device will subsequently be deployed in a large epidemiologic study enrolling 150 twin pairs to examine associations between air pollution exposures measured in time and space on markers of inflammation and cardiometabolic risk. Because twin pairs will be used in the larger epidemiologic study, we chose to use twin pairs for device testing and feedback in this study. Although it is possible that the twins may have discussed data collection with each other and thus potentially influenced compliance and survey and focus group responses, we do not believe this was a major concern. First, it is important to note that the twins did not live together, a condition of enrollment in the epidemiologic study because we are specifically interested in how “place” influences air pollution exposure. Thus, this level of intimate personal contact would not have occurred. Second, there was within-pair variability in wear time and in the comments provided on the questionnaires and during the focus groups, demonstrating that each twin member had his or her own unique experience using the device. Although twins generally agreed with one another on issues such as the size of the device and how much noise it made, these comments were made by others in the group as well. The focus group feedback and questionnaire data suggest that the device could still undergo an additional 1 or 2 iterative cycles to further improve usability, but any further changes could potentially result in a loss of functionality of the device; for example, the questionnaire data showed an increase in system usability scores over the 3 cycles. The SUS scores indicate that the user experience by the end of the design-feedback cycle had improved. Although this demonstrates a large relative improvement in usability, the final score still suggests further room for improvement. However, further changes to reduce size and noise level (still among the chief complaints) would likely have a negative effect on our ability to obtain valid and reliable data on PM2.5 air pollution. It is also important to note that participants were probably using their mobile phone or similar device as a size comparison, instead of similarly sized and commercially available personal air exposure monitors such as the MicroPEM (1.5×1.75×5 inches) or SidePak AM520 (5.1×3.7×3.1 inches).

    Prior use and experience with a system or a tool can impact SUS scores on subsequent follow-up testing [30] and thus, it could be argued that participants provided better scores over time because of greater knowledge about the monitor (ie, the learning process). However, it is important to note that none of the participants had any experience with the device at baseline, so any learning that would have occurred would have been consistent across all participants. Introducing the device to novel participants at each wave would have introduced a number of important differences in subject characteristics (eg, age, sex, race or ethnicity, education level, and income level) that would have likely confounded the SUS results; therefore, we chose to keep the subjects consistent across waves. Finally, substantive changes were made to the device after each wave, including the addition of battery lights and a muffler, improved battery life, and a different carrying system for the device. All of these changes were repeatedly mentioned as positives in the focus groups and would have contributed to increased SUS scores, yet none of these changes could be attributed to increased knowledge of or experience with the device; rather, they were new additions specifically intended to improve usability.

    We are not aware of any other studies that used SUS to evaluate performance in personal air pollution monitors. Thus, it is difficult to put our usability results in the context of other air pollution monitors. We identified another study that built a personal exposure monitor to measure particles as well as activity and location like ours, but it was only used for 6 hours in 1 individual [31]. Thus, once again, it is difficult to contextualize the results. The Air Sensor Guidebook (US Environmental Protection Agency [32]) notes that sensor performance requirements differ according to the specific application, making it difficult to compare our device to others. With respect to the increase in wear time, this is an important finding to us because our larger epidemiologic study will measure context-specific physical activity in addition to air pollution and a “valid monitoring” day in the physical activity literature is generally considered 600 minutes per day [33]. Thus, we are confident that our participants will comply with the wearing time aspect of the protocol and provide adequate physical activity data. The minimum wear time for a valid monitoring day in air pollution exposure studies is unknown, but we are confident that we will exceed the 600 minutes per day threshold, which should provide more than adequate data on personal air pollution exposures on a given day.

    During the third cycle of collection, the Seattle region was experiencing an increase in smoke from wildfires in British Columbia, which was mentioned by the participants in the third focus group. They were curious as to what impact the wildfires would have on the data they were collecting. This speaks to the utility of the device; not only is the device a robust instrument for collecting PM2.5 air pollution data for research but it may also serve as a personal health monitoring tool to assess the impact of current air quality conditions in individuals with compromised health (eg, asthma). An example of the reports sent to participants is included in the Additional File, illustrating PM2.5 outdoor and personal air pollution during the data collection period for 2 members of a twin pair who are discordant for exposure. That the device may serve as a robust instrument for air pollution research is also supported by the participant’s generally favorable view of the research study with an average RAQ score of 24.3 (on a 28-point scale).

    Limitations

    The main limitation of this study is the small sample size. Starting with only 5 pairs, we lost 1 pair in the second wave of the study, and 2 pairs in the third wave of the study. The study required both twins to participate and therefore, the loss of 1 member of a pair resulted in the loss of the full pair because we did not have the singleton twins collect data in the second and third waves. We also had a limited number of devices available for initial testing, which made pair-wise recruitment more difficult.

    Conclusions

    We used the design-feedback iterative cycle to improve the usability of a personal air pollution monitor for subsequent deployment in an epidemiologic cohort study. As attempts are made to decrease overall regional concentrations of PM2.5, identifying hotspots of exposure both in time and space, indoors and outdoors, will become increasingly relevant to protect public health. The availability of a low-cost, validated personal monitor that can measure multiple aspects of exposure, behavior, and context may greatly enhance our future ability to study the health impacts of these policy and planning changes.

    Acknowledgments

    The authors thank the staff and members of the Washington State Twin Registry for their participation and dedication. This work was supported by a grant from the National Institutes of Health (NIH; R21ES024715). The NIH played no role in the design of the study and collection, analysis, and interpretation of data, or in writing the manuscript and the decision to submit it for publication.

    Conflicts of Interest

    None declared.

    References

    1. Dockery DW, Pope CA, Xu X, Spengler JD, Ware JH, Fay ME, et al. An association between air pollution and mortality in six U.S. cities. N Engl J Med 1993 Dec 09;329(24):1753-1759. [CrossRef] [Medline]
    2. Krewski D, Jerrett M, Burnett RT, Ma R, Hughes E, Shi Y, et al. Extended follow-up and spatial analysis of the American Cancer Society study linking particulate air pollution and mortality. Res Rep Health Eff Inst 2009 May(140):5-114; discussion 115. [Medline]
    3. Pope CA, Burnett RT, Thun MJ, Calle EE, Krewski D, Ito K, et al. Lung cancer, cardiopulmonary mortality, and long-term exposure to fine particulate air pollution. JAMA 2002 Mar 06;287(9):1132-1141 [FREE Full text] [Medline]
    4. Pope CA, Turner MC, Burnett RT, Jerrett M, Gapstur SM, Diver WR, et al. Relationships between fine particulate air pollution, cardiometabolic disorders, and cardiovascular mortality. Circ Res 2015 Jan 02;116(1):108-115. [CrossRef] [Medline]
    5. Lipfert F, Wyzga R, Baty J, Miller J. Traffic density as a surrogate measure of environmental exposures in studies of air pollution health effects: Long-term mortality in a cohort of US veterans. Atmospheric Environment 2006 Jan;40(1):154-169. [CrossRef]
    6. Peng RD, Dominici F, Pastor-Barriuso R, Zeger SL, Samet JM. Seasonal analyses of air pollution and mortality in 100 US cities. Am J Epidemiol 2005 Mar 15;161(6):585-594. [CrossRef] [Medline]
    7. Samet JM, Dominici F, Curriero FC, Coursac I, Zeger SL. Fine particulate air pollution and mortality in 20 U.S. cities, 1987-1994. N Engl J Med 2000 Dec 14;343(24):1742-1749. [CrossRef] [Medline]
    8. Zeger SL, Dominici F, McDermott A, Samet JM. Mortality in the Medicare population and chronic exposure to fine particulate air pollution in urban centers (2000-2005). Environ Health Perspect 2008 Dec;116(12):1614-1619 [FREE Full text] [CrossRef] [Medline]
    9. Pope CA, Dockery DW. Health effects of fine particulate air pollution: lines that connect. J Air Waste Manag Assoc 2006 Jun;56(6):709-742. [Medline]
    10. United States Environmental Protection Agency. 2018 Oct 31. PM-2.5 (2012) Designated Area/State Information   URL: https://www3.epa.gov/airquality/greenbook/kbtc.html [accessed 2018-11-13] [WebCite Cache]
    11. Brook R, Rajagopalan S, Pope C, Brook JR, Bhatnagar A, Diez-Roux AV, American Heart Association Council on EpidemiologyPrevention‚ Council on the Kidney in Cardiovascular Disease‚Council on Nutrition‚ Physical ActivityMetabolism. Particulate matter air pollution and cardiovascular disease: An update to the scientific statement from the American Heart Association. Circulation 2010 Jun 01;121(21):2331-2378. [CrossRef] [Medline]
    12. Simkhovich BZ, Kleinman MT, Kloner RA. Air pollution and cardiovascular injury epidemiology, toxicology, and mechanisms. J Am Coll Cardiol 2008 Aug 26;52(9):719-726 [FREE Full text] [CrossRef] [Medline]
    13. Sarnat JA, Wilson WE, Strand M, Brook J, Wyzga R, Lumley T. Panel discussion review: session 1--exposure assessment and related errors in air pollution epidemiologic studies. J Expo Sci Environ Epidemiol 2007 Dec;17 Suppl 2:S75-S82. [CrossRef] [Medline]
    14. Jerrett M, Arain A, Kanaroglou P, Beckerman B, Potoglou D, Sahsuvaroglu T, et al. A review and evaluation of intraurban air pollution exposure models. J Expo Anal Environ Epidemiol 2005 Mar;15(2):185-204. [CrossRef] [Medline]
    15. Sarnat JA, Koutrakis P, Suh HH. Assessing the relationship between personal particulate and gaseous exposures of senior citizens living in Baltimore, MD. J Air Waste Manag Assoc 2000 Jul;50(7):1184-1198. [Medline]
    16. Monn C. Exposure assessment of air pollutants: a review on spatial heterogeneity and indoor/outdoor/personal exposure to suspended particulate matter, nitrogen dioxide and ozone. Atmospheric Environment 2001 Jan;35(1):1-32. [CrossRef]
    17. Klepeis N, Nelson W, Ott W, Robinson J, Tsang A, Switzer P, et al. The National Human Activity Pattern Survey (NHAPS): a resource for assessing exposure to environmental pollutants. J Expo Anal Environ Epidemiol 2001;11(3):231-252 [FREE Full text] [CrossRef] [Medline]
    18. Avery C, Mills K, Williams R, McGraw K, Poole C, Smith R, et al. Estimating error in using ambient PM2.5 concentrations as proxies for personal exposures: a review. Epidemiology 2010 Mar;21(2):215-223 [FREE Full text] [CrossRef] [Medline]
    19. Schmidt HG, Leffringhausen W. [Therapy and results of infections of the large joints without bone injuries using septopal]. Aktuelle Traumatol 1985 Oct;15(5):222-231. [Medline]
    20. Austin E, Novosselov I, Seto E, Yost M. Laboratory Evaluation of the Shinyei PPD42NS Low-Cost Particulate Matter Sensor. PLoS One 2015;10(9):e0137789 [FREE Full text] [CrossRef] [Medline]
    21. Gao M, Cao J, Seto E. A distributed network of low-cost continuous reading sensors to measure spatiotemporal variations of PM2.5 in Xi'an, China. Environ Pollut 2015 Apr;199:56-65 [FREE Full text] [CrossRef] [Medline]
    22. He JY, Novosselov IV. Design and evaluation of an aerodynamic focusing micro-well aerosol collector. Aerosol Science and Technology 2017 May 11;51(9):1016-1026. [CrossRef]
    23. Holstius D, Pillarisetti A, Smith K, Seto E. Field calibrations of a low-cost aerosol sensor at a regulatory monitoring site in California. Atmos. Meas. Tech 2014 Apr 30;7(4):1121-1131. [CrossRef]
    24. Nieuwenhuijsen MJ, Donaire-Gonzalez D, Rivas I, de Castro M, Cirach M, Hoek G, et al. Variability in and agreement between modeled and personal continuously measured black carbon levels using novel smartphone and sensor technologies. Environ Sci Technol 2015 Mar 03;49(5):2977-2982. [CrossRef] [Medline]
    25. He J, Beck NK, Kossik AL, Zhang J, Seto E, Meschke JS, et al. Evaluation of micro-well collector for capture and analysis of aerosolized Bacillus subtilis spores. PLoS One 2018;13(5):e0197783 [FREE Full text] [CrossRef] [Medline]
    26. Brooke J, McClelland I. Usability Evaluation in Industry. In: Jordan P, Thomas B, Weerdmeester BA, McClelland AL, editors. SUS - A quick and dirty usability scale. London: Taylor & Francis; 1996:189-194.
    27. Kim SYH, Kim HM, McCallum C, Tariot PN. What do people at risk for Alzheimer disease think about surrogate consent for research? Neurology 2005 Nov 08;65(9):1395-1401. [CrossRef] [Medline]
    28. Rubright JD, Cary MS, Karlawish JH, Kim SYH. Measuring how people view biomedical research: Reliability and validity analysis of the Research Attitudes Questionnaire. J Empir Res Hum Res Ethics 2011 Mar;6(1):63-68 [FREE Full text] [CrossRef] [Medline]
    29. Hurvitz PM, Moudon AV, Kang B, Saelens BE, Duncan GE. Emerging technologies for assessing physical activity behaviors in space and time. Front Public Health 2014;2:2 [FREE Full text] [CrossRef] [Medline]
    30. Bangor A, Kortum P, Miller J. An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 2008 Jul 30;24(6):574-594. [CrossRef]
    31. Zhuang Y, Lin F, Yoo E, Xu W. AirSense: A portable context-sensing device for personal air quality monitoring. In: AirSense: A portable context-sensing device for personal air quality monitoring. In: MobileHealth '15; Hangzhou, China: ACM; 2015 Presented at: MobileHealth '15; 2015; Hangzhou, China p. A.
    32. Williams R, Vasu K, Snyder E, Kaufman A, Dye T, Rutter A, et al. United States Environmental Protection Agency. 2014. Air Sensor Guidebook   URL: https:/​/cfpub.​epa.gov/​si/​si_public_record_report.​cfm?Lab=NERL&dirEntryId=277996&simpleSearch=1&searchAll=air+sensor+guidebook
    33. Troiano RP, Berrigan D, Dodd KW, Mâsse LC, Tilert T, McDowell M. Physical activity in the United States measured by accelerometer. Med Sci Sports Exerc 2008 Jan;40(1):181-188. [CrossRef] [Medline]


    Abbreviations

    GPS: Global Positioning System
    PM2.5: fine particulate matter
    RAQ: Research Attitudes Questionnaire
    SUS: System Usability Scale


    Edited by G Eysenbach; submitted 23.08.18; peer-reviewed by D Donaire-Gonzalez, N Arora, M Das; comments to author 27.09.18; revised version received 10.10.18; accepted 13.10.18; published 21.12.18

    ©Glen E Duncan, Edmund Seto, Ally R Avery, Mike Oie, Graeme Carvlin, Elena Austin, Jeffry H Shirai, Jiayang He, Byron Ockerman, Igor Novosselov. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 21.12.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.