Using a Mobile App-Based International Classification of Functioning, Disability, and Health Set to Assess the Functioning of Spinal Cord Injury Patients: Rasch Analysis

Introduction

The International Classification of Functioning, Disability, and Health (ICF) is a unified system of terminology for multidisciplinary use issued in 2001 by the World Health Organization (WHO). It provides a consensus framework for defining functioning and disability and their interrelationships with health conditions and contextual factors [1]. As a standard language, the ICF is designed to be easily understood and used among multidisciplinary teams [2]. The comprehensive perspective on functioning and interdisciplinary focus have motivated the development of data collection tools, electronic health records, and assessment instruments [3-5], and it is now sometimes viewed as a third health indicator for monitoring a health system’s performance after mortality and morbidity [6]. Many studies have provided evidence of the ICF’s value in reflecting patient levels of functioning, helping decision making, enhancing collaboration, and planning treatment.

However, challenges have limited practical application of the ICF, including its relatively complex terminology and category numbering. Each ICF category has its own distinct definition, which doesn’t always accord with the prevailing medical terminology. Professionals need to be trained before using the system [5]. Although many ICF core sets with fewer categories have been specifically developed for certain conditions, some studies still report that the application of the ICF is time-consuming [7]. Reducing the number of categories by selecting only the most relevant remains challenging. Additionally, rating using the ICF qualifiers is not easy. There are 5 grades: 0 = no problem, 1 = mild, 2 = moderate, 3 = severe, and 4 = complete problem. But there can be large differences among user assessments because of a lack of clear assessment guidelines, giving the approach poor interrater reliability [8-11]. Some researchers have suggested establishing assessment guidelines, but these would necessarily be complex and require additional user training, further hindering the system’s acceptance [12,13].

A mobile health app is a health-related software program installed on smart mobile devices that provides health information and tracks a user’s health behavior. It might also allow remote consultation [14]. With the popularity of personal mobile devices such as cellphones and tablets, mobile apps have been used in many fields in health care [15,16]. Studies have shown that app-based transitional care enhances patients’ self-perceptions of efficacy [17], helps prevent complications [18,19], improves quality of life [18], and reduces readmissions to hospital [18].

The study team previously developed an ICF-based app called Together for the transitional care of spinal cord injury (SCI) patients in China [20]. SCI is a serious and life-changing disease that can cause paraplegia or quadriplegia. Most SCI patients in China live at home after their acute treatment and rehabilitation [21]. They almost always need professional medical support for further rehabilitation and preventing complications [22,23]. However, community medical resources in China are at present limited and cannot meet SCI patients’ complex long-term health needs [24]. Together was designed to bring professional health care support from medical institutions to SCI patients living with their families in China. The language of the app is Chinese, and the copyright is held by China’s Sun Yat-sen University [25].

Together’s core framework is a set of ICF categories that reflect the levels of functioning typical of SCI patients and help organize online assessment, standardize health guidance, and coordinate interdisciplinary collaboration. The app uses fewer and more specific categories than the normal ICF core set, focusing on the transitional care of SCI patients. Preliminary studies identified 31 ICF categories as the most useful outcome indicators in the transitional care of SCI patients, covering the major physiological, psychological, and social participation problems of SCI patients at home [20,26]. The app provides consistent assessment prompts for the raters. Guidelines are provided for rating each ICF category, which helps to guarantee the consistency of the assessments, minimize the training required, and ease the load on the clinical staff doing the assessments.

This study was part of a research program designed to document the effects of an app-based transitional care model for SCI patients at home. Rasch analysis was applied to examine the suitability of the app’s set of categories. The overall aim was to determine to what extent Together can solve problems related to using the ICF in clinical practice.

The assumption of the Rasch model is that a person with greater ability is more likely than a person with less ability to pass in relation to an item, and that an easy item is more likely to be passed than a difficult one [27]. A Rasch analysis is used to examine whether an instrument makes those distinctions satisfactorily. A person’s performance on an item should be related only to the person’s ability and the difficulty of the item, regardless of gender, age, education, etc. [27]. If an instrument fits the Rasch model well, it can be used to reflect the performance of people with different abilities. This study was designed to test Together’s performance in that regard. A good result could present a new approach in the use of the ICF.

Methods

Study Design

A cross-sectional design was employed involving four research centers in Guangzhou, Chengdu, and Shiyan in China. The study was approved by Sun Yat-sen University’s ethics committee (file 2017ZSLYEC-0620).

Participants

The participants were recruited between May 2018 and October 2019 prior to discharge from the four research centers. The inclusion criteria were age 18 years or older, SCI according to the International Standards for Neurological Classification of Spinal Cord Injury [28] and imaging examination, less than 2 years since injury, conscious and able to communicate, and in possession of an internet-connected mobile device and familiarity with using it. The exclusion criteria were severe heart, brain, lung, liver, or kidney disease; acute-stage spinal cord injury or in the critical period; or spinal cord lesions with a degenerative, genetic, or congenital cause.

App

The Together app was designed to help medical staff provide remote follow-up for home-dwelling SCI patients. Hospital-based nurses, physicians, and therapists responsible for the transitional care for SCI patients at home are the target users. The core functions of the app comprise online assessment, providing standard health guidance, interdisciplinary referral within the team, interaction among health staff and patients, and management of online follow-up. With the help of the app, health care personnel can assess patient performance in terms of ICF categories and remotely provide health education to patients according to the assessment results. The app can be used to refer patients to different professionals on the health care team. Weekly reminders make managing follow-up by medical staff easier.

The app’s development has been reported previously [20,26]. From 51 ICF categories identified as outcome indicators useful in the transitional care of SCI patients by experts in the field via a 3-round Delphi survey, 31 were selected by a panel of 5 experts to form the app’s framework based on the feasibility of use in clinical practice. them. These categories best reflected the dysfunctions and complications most common and most in need of monitoring for SCI patients at home. Categories reflecting actual performance of SCI patients in daily life were preferred.

The 31 ICF categories address physiological functioning, psychological functioning, complications, daily living activities, social participation, adaptation to environmental factors, and personal factors. For each category, guidelines were established for converting routine clinical assessment results to the ICF qualifiers, and a standardized guidance program was formulated by the expert panel based on the knowledge-attitude-practice theory. Together is an Android app developed using the Java language. Most of the app’s functions—online assessment, providing standard health guidance, interdisciplinary referral—apply to all of the categories.

The app’s utility depends heavily on to what extent its assessment results reflect functioning differences among patients with different capabilities. This study used Rasch analysis to test the app’s suitability as an assessment instrument. Figure 1 shows the development and examination process.

‎View this figure

Assessment Guidelines

Medical staff communicate with the patients face to face or by telephone, assess them on each ICF category, and record the results in the app, which offers standard verbal prompts to the clinician to unify different clinicians’ assessments with respect to each ICF category. For example, the verbal prompt of the sensation of pain (b280) category was “If 0 is not painful and 10 is the most painful, how serious is your pain?” Four transformation guidelines were developed to transform the initial clinical assessment results: 0 = no problem, 1 = mild, 2 = moderate, 3 = severe, or 4 = complete problem (Table 1) [29]. The clinician is responsible for the initial assessments and the app automatically transforms their initial assessment results to the ICF qualifiers according to the preset guidelines.

Guideline 1 transforms patient information in the form of percentages to the ICF qualifiers. Using muscle power functions (b730) as an example, the 0 qualifier would indicate that all of the key muscles below the injured neurological level had power grade >3, and the 4 qualifier would indicate that 95% to 100% of key muscles had power grade <3. Guideline 2 transforms the wording of patient reports to the qualifiers. Using mobility of joint functions (b710) as an example, no limitation of joint mobility, slight limitation, moderate limitation, severe limitation, and total immobility would be rated as 0, 1, 2, 3, and 4, respectively. Guideline 3 transforms the frequency with which a problem was observed to the qualifiers. Using increased blood pressure (b4200) as an example, stable blood pressure over the past month would be rated as 0, whereas high blood pressure almost every day would receive a 4. Guideline 4 transforms assessment results generated using routine clinical instruments or standards, such as the 0-10 numerical rating scale (NRS) for pain, to the qualifiers. NRS scores of 0, 1-2, 3-4, 5-9, and 10 would be rated as 0, 1, 2, 3, and 4, respectively (Multimedia Appendix 1).

Instruments

Demographic and Disease Questionnaire

The questionnaire consisted of two parts, including demographic and disease-related data such as name, gender, age, education level, diagnosis, etiology, American Spinal Injury Association Impairment Scale grade, SCI level, and duration of the disability. The information was collected by nurses in face-to-face interviews with the patients and by reviewing their medical records.

App-Based International Classification of Functioning, Disability, and Health Set

The 31 categories came from the three components body functions and body structures (15 categories), activities and participation (10 categories), and contextual factors (6 categories). Although the ICF does not classify personal factors in the contextual factors component, four personal factor items related to the psychology of SCI patients were included based on preliminary testing (acceptance of life in a wheelchair/in bed, knowledge about spinal cord injury, coping with everyday life, and adjustment to new body image) [20,26]. The app was then used to rate the SCI patients’ functioning on a 1-4 scale. In addition, an option 9 (not applicable) was also used. For the activities and participation component, each patient’s performance on tasks in actual life situations (not in a standard environment) was assessed. For the contextual factors component, barriers (but not facilitators) encountered in each patient’s life were assessed. In previous research, the eating (d550) and drinking (d560) categories were found to be strongly interrelated and difficult to assess separately [13]; therefore, those categories were combined into a testlet to be assessed.

Data Collection

All nurses who participated in the study first received half a day of training involving a lecture and workshop, including an introduction to the study and how to use the app. Eligible patients were invited to participate in the study before discharge. After signing the informed consent form, demographic and disease-related data were collected by the trained nurses in face-to-face interviews and by reviewing the patients’ medical records. The nurses then assessed the patients’ performance with respect to each ICF category using the app. They did this face to face referring to the standard verbal prompts for each category in the app. The app system allows submission only after all categories have been evaluated; otherwise, the system indicates to the user that the evaluation is incomplete.

Analysis

Data Processing

SPSS Statistics software version 21.0 (IBM Corporation) was used to analyze the demographic and disease data. RUMM2030 software (RUMM Laboratory Pty) was used to perform the Rasch analysis. For each component of the ICF set, the overall fit to a Rasch model was examined. If the overall fit was not good, poorly fitting categories were identified and deleted. Another round of Rasch analysis was then run until adequate overall fit was attained. The following properties of the ICF set were examined.

Overall Fit to the Rasch Model

A nonsignificant value in a χ² test for item-trait interaction, a mean within ±2.5 (SD <1.5) for the fit residuals of the items and persons indicate good overall fit to the Rasch model [30]. Fit residuals represent the extent to which the observations do not fit a Rasch model. The significance level was adjusted using the Bonferroni correction [31].

Single-Item Fit to the Rasch Model

The good fit of a single category was represented by a nonsignificant χ² test and a mean of the fit residual values within ±2.5 [32].

Person Separation Index

An acceptable person separation index indicates good internal consistency for the instrument and reflects the ability of the instrument to discriminate between people with different abilities. It has a range of 0 to 1, with higher values indicating a better ability (>0.7 indicates good) [33].

Differential Item Functioning

For an ideal Rasch model, no factors should influence a person’s performance regarding an item except the Rasch factors [30]. The differential item functioning shows the influences of the other factors. In this study, a nonsignificant analysis of variance result was taken as indicating no differential item functioning for a specific category based on gender, age, education level, or etiology.

Results

Patient Characteristics

The demographic and disease characteristics of all 112 spinal cord injury patients are shown in Table 2. Their ages ranged from 18 to 65 years (mean 41.7 [SD 12.3]); 82.1% (92/112) of the patients were younger than 60 years, with patients in the 40- to 49-year age group the most numerous; 83.0% (93/112) of the patients were male. A total of 60.8% (68/112) claimed to have had a middle school education and 25.9% (29/112) only primary education or less. The duration of their disability ranged from 1 to 22 months (mean 7.1 [SD 4.2]); 88.4% (99/112) had been injured for less than a year. Most of the injuries (100/112, 89.3%) were caused by trauma. Most of the patients were injured at the thoracic and cervical levels, accounting for 50.0% (56/112) and 27.7% (31/112), respectively. About half (57/112, 50.9%) of the patients had complete injury.

Rasch Analysis Results

The 31 ICF categories belonged to body functions and body structures (15), activities and participation (10), and contextual factors (6). To attain adequate fit to the Rasch model for each component, categories that did not fit were deleted as multiple rounds of Rasch analysis were conducted. Table 3 shows the process and results of the Rasch analysis for each component.

In the first-round Rasch analysis, the body functions and body structures component consisting of 15 categories exhibited a nonsignificant χ² test result for the item-trait interaction (χ²₃₀=41.5, P=.08, Bonferroni-adjusted P=.05/15=.003). Additionally, the means for the item and person fit residuals were within ±2.5 (SD <1.5). These results suggested a good fit to the Rasch model. All of the 15 categories exhibited nonsignificant χ² test results and the means of their fit residual values were also within acceptable limits (Table 4). The person separation index of this component was 0.5. There was no differential item functioning for any of the categories by gender, age, education level, or etiology.

Regarding the initial activities and participation component with 10 categories, in the first-round Rasch analysis, the χ² test for the item-trait interaction yielded a significant result (χ²₂₀=75.7, P<.001, Bonferroni-adjusted P=.05/10=.005). The single-item fit analysis found that four categories, changing basic body position (d410), washing oneself (d510), family relationships (d760), and socializing (d9205), did not fit the Rasch model. The family relationships (d760) and socializing (d9205) categories both exhibited poor fit results in a previous study [13]. Considering the mean fit residual values 3.253 for family relationships (d760) and 2.605 for socializing (d9205) and the category meanings, d760 was deleted first. However, the χ² test result for the item-trait interaction in the second-round Rasch analysis (after deleting d760) remained significant (χ²₁₈=43.3, P<.001, Bonferroni-adjusted P=.05/9=.0056). A third round of Rasch analysis was performed after deleting both family relationships (d760) and socializing (d9205). Although the standard deviation of the overall item fit residuals (SD 1.59) was a little larger than the upper limit, the χ² test result for the item-trait interaction was no longer significant (χ²₁₆=24.7, P=.08, Bonferroni-adjusted P=.05/8=.0063), suggesting good fit to the Rasch model. The single-item fit tests for the remaining 8 categories also yielded good model fit results, with nonsignificant χ² test results (Table 4). The person separation index of the component was excellent (0.89) and no differential item functioning was detected for any of the categories by gender, age, education level, or etiology.

For the contextual factors component, the first-round Rasch analysis starting with 6 categories indicated poor model fit according to the χ² test result for the item-trait interaction (χ²₁₂=32.8, P<.001, Bonferroni-adjusted P=.05/6=.0083). The following single-item fit analysis showed that the personal factor knowledge about spinal cord injury did not fit well (χ²₂=19.1, P<.001). After deleting that item, the component displayed satisfactory overall model fit, with a nonsignificant χ² test result for the item-trait interaction (χ²₁₀=13.6, P=.19, Bonferroni-adjusted P=.05/5=.01). The means and standard deviations of the fit residuals for items and persons were both within the acceptable limits. The single-item fit analyses for the remaining 5 categories were also satisfactory, with nonsignificant χ² test results (Table 4). The person separation index of this component was 0.68, and no differential item functioning was detected for any of the categories by gender, age, education level, or etiology.

Discussion

Principal Findings

The results of the Rasch analysis showed good fit to the Rasch model for the different components of the ICF set as implemented in the app after modification. Both overall and single-item fit were satisfactory. There was no differential item functioning for any of the ICF categories by gender, age, education level, or etiology. These results indicate the suitability of the app-based ICF set as an assessment tool for assessing the functioning of SCI patients.

The app-based ICF set is one of many forms of ICF-based electronic health records. Several previous studies have confirmed the role of ICF-based electronic health records in reflecting patient functioning and facilitating rehabilitation [6,34]. As a unified and standard language originally developed for multidisciplinary use, the original ICF was relatively easily understood by different disciplines and suitable for multidisciplinary teamwork [2]. However, some obstacles still existed, including the complexity of the terminology, lack of operationalization of the ICF qualifiers, and training overload for ICF users. The satisfactory internal construct validity of the set developed in the study is mainly due to the selection of suitable ICF categories and standardized assessment enabled by the app. In this study, the app applied many fewer ICF categories (31) than the comprehensive ICF core set (168 categories) and the brief core set (33 categories) for SCI patients issued by the WHO’s ICF research branch [35]. Additionally, the categories were more specific because they were originally identified as good outcome indicators for SCI patients in China [20,26]. Each ICF category focuses on a specific area and is independent from the others, which helped to ensure the overall fit to the Rasch model based on the assessments’ content.

Together’s verbal prompts standardize assessment and give more consistent assessment results. The transformation guidelines operationalize the 5 ICF qualifiers simply and effectively. With the help of the app, the ICF qualifiers can automatically be matched to the initial clinical assessment results. No additional training on ICF terminology or qualifiers is needed. The process reduces the differences among assessors and makes presentation of the ICF data more convenient and intelligent.

Family relationships (d760), socializing (d9205), and knowledge about spinal cord injury were deleted. In a previous study, d760 and d9205 also exhibited poor model fit [13]. Family relationships are the basis for good functioning in the family, and socializing reflects a patient’s social participation. Both of them are influenced by many factors such as age, severity of the injury, and financial considerations [36,37]. For an item to have perfect fit to the Rasch model, no other factors should influence a person’s performance regarding the item except for the person’s ability and the item’s difficulty, which may explain why the two factors did not fit the Rasch model. The ICF does not classify personal factors because of the large social and cultural variance associated with them [1]. The personal factors assessed in this ICF set were identified by multiround expert surveys [26]. Knowledge about SCI is a broad concept covering many aspects such as injury outcomes, functional rehabilitation, and preventing complications, and it is influenced by multiple factors such as the patient’s level of education, efficacy self-perceptions, and any health education they have received [38]. This may be why the item did not fit the Rasch model well. However, family relationships or support, social participation and the knowledge about SCI are important indicators for the outcomes of rehabilitation. To ensure the completeness of a patient’s health records, it is suggested that they be retained as part of the basic health data collection but not as part of the assessment instrument. Further research should use professional measurements that reflect family relationships, socializing, and the knowledge level for SCI patients.

It is worth noting that, in contrast to the person separation index of the activities and participation component, the person separation index of the other two components was not ideal (0.5 and 0.68), which reflects the poor internal consistency of the two components. This may be related to the different measuring guidelines used. Four guidelines were used in the study to transform the input of the medical staff, based on patient information in the form of percentages, wording from patient reports, frequency with which a problem was observed, and scores on routine clinical instruments or standards. A previous study [29] found that the categories involving guideline 4 (which transforms scores of a routine clinical instrument or standards into the ICF qualifiers) had the best interrater reliability, and the categories involving guideline 2 (which transforms the wording from patient reports into the ICF qualifiers) had the lowest interrater reliability. In this study, all the activity and participation categories used the guideline with the highest reliability (guideline 4), while all the contextual factor categories used the guideline with the lowest reliability (guideline 2). Although the app automatically transforms the initial data into the ICF qualifiers, subjectivity and uncertainty still existed in the patients’ initial reports and recall, which may explain the unsatisfactory person separation index of the two components.

Limitations

In interpreting these results, it is important to keep in mind that the relatively small sample may have influenced the representativeness of the results. Also, although the app was designed to assess SCI patients at home during transitional care via its remote follow-up function, the study data were collected in face-to-face interviews before the participants were discharged. Further validation with larger samples and remote assessment via the app’s communication function are needed.

Conclusions

This study has confirmed the suitability of the Together app as an assessment tool. With relatively fewer but more specific ICF categories, it overcomes some of the limitations related to applying the ICF and makes assessment results more reliable and consistent. The app opens up a new way to use the ICF with SCI and electronic health records. In the future, the app could be used to capture information about the functioning of SCI patients at home remotely. Such assessment results would help to monitor patients’ functional changes and differences, learn their needs, identify their problems, and provide evidences for further interventions if necessary.