Published on in Vol 8, No 11 (2020): November

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/20723, first published .
Using a Mobile App-Based International Classification of Functioning, Disability, and Health Set to Assess the Functioning of Spinal Cord Injury Patients: Rasch Analysis

Using a Mobile App-Based International Classification of Functioning, Disability, and Health Set to Assess the Functioning of Spinal Cord Injury Patients: Rasch Analysis

Using a Mobile App-Based International Classification of Functioning, Disability, and Health Set to Assess the Functioning of Spinal Cord Injury Patients: Rasch Analysis

Original Paper

1School of Nursing, Sun Yat-sen University, Guangzhou, China

2Department of Spinal Cord Injury Rehabilitation, Sichuan Provincial Rehabilitation Hospital, Chengdu, China

3Department of Spinal Cord Injury Rehabilitation, Guangdong Provincial Work Injury Rehabilitation Hospital, Guangzhou, China

4Department of Rehabilitation Medicine, The Fifth Hospital of Xiamen, Xiamen, China

5Department of Rehabilitation Medicine, Sun Yat-sen Memorial Hospital, Guangzhou, China

*these authors contributed equally

Corresponding Author:

Kun Li, PhD

School of Nursing

Sun Yat-sen University

No. 74 Zhong Shan Second Road

Guangzhou,

China

Phone: 86 138 22206519

Fax:86 020 87333043

Email: likun22@mail.sysu.edu.cn


Background: The International Classification of Functioning, Disability, and Health (ICF) is a unified system of functioning terminology that has been used to develop electronic health records and assessment instruments. Its application has been limited, however, by its complex terminology, numerous categories, uncertain operationalization, and the training required to use it well. Together is a mobile health app designed to extend medical support to the families of spinal cord injury (SCI) patients in China. The app’s core framework is a set of only 31 ICF categories. The app also provides rating guidelines and automatically transforms routine assessment results to the terms of the ICF qualifiers.

Objective: The goal of the research is to examine the suitability of the ICF set used in the app Together for use as an instrument for assessing the functioning of SCI patients.

Methods: A cross-sectional study was conducted including 112 SCI patients recruited before discharge from four rehabilitation centers in China between May 2018 and October 2019. Nurses used the app to assess patient functioning in face-to-face interviews. The resulting data were then subjected to Rasch analysis.

Results: After deleting two categories (family relationships and socializing) and one personal factor (knowledge about spinal cord injury) that did not fit the Rasch model, the body functions and body structures, activities and participation, and contextual factors components of the ICF exhibited adequate fit to the Rasch model. All three demonstrated acceptable person separation indices. The 28 categories retained in the set were free of differential item functioning by gender, age, education level, or etiology.

Conclusions: Together overcomes some of the obstacles to practical application of the ICF. The app is a reliable assessment tool for assessing functioning after spinal cord injury.

JMIR Mhealth Uhealth 2020;8(11):e20723

doi:10.2196/20723

Keywords



The International Classification of Functioning, Disability, and Health (ICF) is a unified system of terminology for multidisciplinary use issued in 2001 by the World Health Organization (WHO). It provides a consensus framework for defining functioning and disability and their interrelationships with health conditions and contextual factors [1]. As a standard language, the ICF is designed to be easily understood and used among multidisciplinary teams [2]. The comprehensive perspective on functioning and interdisciplinary focus have motivated the development of data collection tools, electronic health records, and assessment instruments [3-5], and it is now sometimes viewed as a third health indicator for monitoring a health system’s performance after mortality and morbidity [6]. Many studies have provided evidence of the ICF’s value in reflecting patient levels of functioning, helping decision making, enhancing collaboration, and planning treatment.

However, challenges have limited practical application of the ICF, including its relatively complex terminology and category numbering. Each ICF category has its own distinct definition, which doesn’t always accord with the prevailing medical terminology. Professionals need to be trained before using the system [5]. Although many ICF core sets with fewer categories have been specifically developed for certain conditions, some studies still report that the application of the ICF is time-consuming [7]. Reducing the number of categories by selecting only the most relevant remains challenging. Additionally, rating using the ICF qualifiers is not easy. There are 5 grades: 0 = no problem, 1 = mild, 2 = moderate, 3 = severe, and 4 = complete problem. But there can be large differences among user assessments because of a lack of clear assessment guidelines, giving the approach poor interrater reliability [8-11]. Some researchers have suggested establishing assessment guidelines, but these would necessarily be complex and require additional user training, further hindering the system’s acceptance [12,13].

A mobile health app is a health-related software program installed on smart mobile devices that provides health information and tracks a user’s health behavior. It might also allow remote consultation [14]. With the popularity of personal mobile devices such as cellphones and tablets, mobile apps have been used in many fields in health care [15,16]. Studies have shown that app-based transitional care enhances patients’ self-perceptions of efficacy [17], helps prevent complications [18,19], improves quality of life [18], and reduces readmissions to hospital [18].

The study team previously developed an ICF-based app called Together for the transitional care of spinal cord injury (SCI) patients in China [20]. SCI is a serious and life-changing disease that can cause paraplegia or quadriplegia. Most SCI patients in China live at home after their acute treatment and rehabilitation [21]. They almost always need professional medical support for further rehabilitation and preventing complications [22,23]. However, community medical resources in China are at present limited and cannot meet SCI patients’ complex long-term health needs [24]. Together was designed to bring professional health care support from medical institutions to SCI patients living with their families in China. The language of the app is Chinese, and the copyright is held by China’s Sun Yat-sen University [25].

Together’s core framework is a set of ICF categories that reflect the levels of functioning typical of SCI patients and help organize online assessment, standardize health guidance, and coordinate interdisciplinary collaboration. The app uses fewer and more specific categories than the normal ICF core set, focusing on the transitional care of SCI patients. Preliminary studies identified 31 ICF categories as the most useful outcome indicators in the transitional care of SCI patients, covering the major physiological, psychological, and social participation problems of SCI patients at home [20,26]. The app provides consistent assessment prompts for the raters. Guidelines are provided for rating each ICF category, which helps to guarantee the consistency of the assessments, minimize the training required, and ease the load on the clinical staff doing the assessments.

This study was part of a research program designed to document the effects of an app-based transitional care model for SCI patients at home. Rasch analysis was applied to examine the suitability of the app’s set of categories. The overall aim was to determine to what extent Together can solve problems related to using the ICF in clinical practice.

The assumption of the Rasch model is that a person with greater ability is more likely than a person with less ability to pass in relation to an item, and that an easy item is more likely to be passed than a difficult one [27]. A Rasch analysis is used to examine whether an instrument makes those distinctions satisfactorily. A person’s performance on an item should be related only to the person’s ability and the difficulty of the item, regardless of gender, age, education, etc. [27]. If an instrument fits the Rasch model well, it can be used to reflect the performance of people with different abilities. This study was designed to test Together’s performance in that regard. A good result could present a new approach in the use of the ICF.


Study Design

A cross-sectional design was employed involving four research centers in Guangzhou, Chengdu, and Shiyan in China. The study was approved by Sun Yat-sen University’s ethics committee (file 2017ZSLYEC-0620).

Participants

The participants were recruited between May 2018 and October 2019 prior to discharge from the four research centers. The inclusion criteria were age 18 years or older, SCI according to the International Standards for Neurological Classification of Spinal Cord Injury [28] and imaging examination, less than 2 years since injury, conscious and able to communicate, and in possession of an internet-connected mobile device and familiarity with using it. The exclusion criteria were severe heart, brain, lung, liver, or kidney disease; acute-stage spinal cord injury or in the critical period; or spinal cord lesions with a degenerative, genetic, or congenital cause.

App

The Together app was designed to help medical staff provide remote follow-up for home-dwelling SCI patients. Hospital-based nurses, physicians, and therapists responsible for the transitional care for SCI patients at home are the target users. The core functions of the app comprise online assessment, providing standard health guidance, interdisciplinary referral within the team, interaction among health staff and patients, and management of online follow-up. With the help of the app, health care personnel can assess patient performance in terms of ICF categories and remotely provide health education to patients according to the assessment results. The app can be used to refer patients to different professionals on the health care team. Weekly reminders make managing follow-up by medical staff easier.

The app’s development has been reported previously [20,26]. From 51 ICF categories identified as outcome indicators useful in the transitional care of SCI patients by experts in the field via a 3-round Delphi survey, 31 were selected by a panel of 5 experts to form the app’s framework based on the feasibility of use in clinical practice. them. These categories best reflected the dysfunctions and complications most common and most in need of monitoring for SCI patients at home. Categories reflecting actual performance of SCI patients in daily life were preferred.

The 31 ICF categories address physiological functioning, psychological functioning, complications, daily living activities, social participation, adaptation to environmental factors, and personal factors. For each category, guidelines were established for converting routine clinical assessment results to the ICF qualifiers, and a standardized guidance program was formulated by the expert panel based on the knowledge-attitude-practice theory. Together is an Android app developed using the Java language. Most of the app’s functions—online assessment, providing standard health guidance, interdisciplinary referral—apply to all of the categories.

The app’s utility depends heavily on to what extent its assessment results reflect functioning differences among patients with different capabilities. This study used Rasch analysis to test the app’s suitability as an assessment instrument. Figure 1 shows the development and examination process.

Figure 1. Development process of the Together app.
View this figure

Assessment Guidelines

Medical staff communicate with the patients face to face or by telephone, assess them on each ICF category, and record the results in the app, which offers standard verbal prompts to the clinician to unify different clinicians’ assessments with respect to each ICF category. For example, the verbal prompt of the sensation of pain (b280) category was “If 0 is not painful and 10 is the most painful, how serious is your pain?” Four transformation guidelines were developed to transform the initial clinical assessment results: 0 = no problem, 1 = mild, 2 = moderate, 3 = severe, or 4 = complete problem (Table 1) [29]. The clinician is responsible for the initial assessments and the app automatically transforms their initial assessment results to the ICF qualifiers according to the preset guidelines.

Guideline 1 transforms patient information in the form of percentages to the ICF qualifiers. Using muscle power functions (b730) as an example, the 0 qualifier would indicate that all of the key muscles below the injured neurological level had power grade >3, and the 4 qualifier would indicate that 95% to 100% of key muscles had power grade <3. Guideline 2 transforms the wording of patient reports to the qualifiers. Using mobility of joint functions (b710) as an example, no limitation of joint mobility, slight limitation, moderate limitation, severe limitation, and total immobility would be rated as 0, 1, 2, 3, and 4, respectively. Guideline 3 transforms the frequency with which a problem was observed to the qualifiers. Using increased blood pressure (b4200) as an example, stable blood pressure over the past month would be rated as 0, whereas high blood pressure almost every day would receive a 4. Guideline 4 transforms assessment results generated using routine clinical instruments or standards, such as the 0-10 numerical rating scale (NRS) for pain, to the qualifiers. NRS scores of 0, 1-2, 3-4, 5-9, and 10 would be rated as 0, 1, 2, 3, and 4, respectively (Multimedia Appendix 1).

Table 1. Guidelines for transforming routine assessment results to the International Classification of Functioning, Disability, and Health qualifiers.
ICFa qualifierGuideline 1bGuideline 2cGuideline 3dGuideline 4e
0 (no problem)0%-4%No, none, absent, negligible...The person has no such problem.NRSf for pain, MASg, NPIAPh stage, FIMi, WHOQoL-BREFj, SF-36k
1 (mild problem)5%-24%Mild, slight, low...The problem rarely happened in the last month (<25% of the time). NRSf for pain, MASg, NPIAPh stage, FIMi, WHOQoL-BREFj, SF-36k
2 (moderate problem)25%-49%Moderate, medium, fair...The problem happened occasionally in the last month (<50% of the time). NRSf for pain, MASg, NPIAPh stage, FIMi, WHOQoL-BREFj, SF-36k
3 (severe problem)50%-95%Severe, high, extreme...The problem happened frequently in the last month (>50% of the time). NRSf for pain, MASg, NPIAPh stage, FIMi, WHOQoL-BREFj, SF-36k
4 (complete problem)96%-100%Complete, total...The problem happened almost every day in the last month (>95% of the time). NRSf for pain, MASg, NPIAPh stage, FIMi, WHOQoL-BREFj, SF-36k

aICF: International Classification of Functioning, Disability, and Health.

bGuideline 1: transforms patient information in the form of percentages to the ICF qualifiers.

cGuideline 2: transforms wording from patient reports to the ICF qualifiers.

dGuideline 3: transforms the frequency with which a problem was observed during the previous month to the ICF qualifiers.

eGuideline 4: transforms the scores of a routine clinical instrument or standards to the ICF qualifiers.

fNRS: numeric rating scale.

gMAS: Modified Ashworth Scale.

hNPIAP: National Pressure Injury Advisory Panel.

iFIM: Functional Independence Measure.

jWHOQoL-BREF: World Health Organization Quality of Life Assessment–Abbreviated.

kSF-36: 36-Item Short Form Health Survey.

Instruments

Demographic and Disease Questionnaire

The questionnaire consisted of two parts, including demographic and disease-related data such as name, gender, age, education level, diagnosis, etiology, American Spinal Injury Association Impairment Scale grade, SCI level, and duration of the disability. The information was collected by nurses in face-to-face interviews with the patients and by reviewing their medical records.

App-Based International Classification of Functioning, Disability, and Health Set

The 31 categories came from the three components body functions and body structures (15 categories), activities and participation (10 categories), and contextual factors (6 categories). Although the ICF does not classify personal factors in the contextual factors component, four personal factor items related to the psychology of SCI patients were included based on preliminary testing (acceptance of life in a wheelchair/in bed, knowledge about spinal cord injury, coping with everyday life, and adjustment to new body image) [20,26]. The app was then used to rate the SCI patients’ functioning on a 1-4 scale. In addition, an option 9 (not applicable) was also used. For the activities and participation component, each patient’s performance on tasks in actual life situations (not in a standard environment) was assessed. For the contextual factors component, barriers (but not facilitators) encountered in each patient’s life were assessed. In previous research, the eating (d550) and drinking (d560) categories were found to be strongly interrelated and difficult to assess separately [13]; therefore, those categories were combined into a testlet to be assessed.

Data Collection

All nurses who participated in the study first received half a day of training involving a lecture and workshop, including an introduction to the study and how to use the app. Eligible patients were invited to participate in the study before discharge. After signing the informed consent form, demographic and disease-related data were collected by the trained nurses in face-to-face interviews and by reviewing the patients’ medical records. The nurses then assessed the patients’ performance with respect to each ICF category using the app. They did this face to face referring to the standard verbal prompts for each category in the app. The app system allows submission only after all categories have been evaluated; otherwise, the system indicates to the user that the evaluation is incomplete.

Analysis

Data Processing

SPSS Statistics software version 21.0 (IBM Corporation) was used to analyze the demographic and disease data. RUMM2030 software (RUMM Laboratory Pty) was used to perform the Rasch analysis. For each component of the ICF set, the overall fit to a Rasch model was examined. If the overall fit was not good, poorly fitting categories were identified and deleted. Another round of Rasch analysis was then run until adequate overall fit was attained. The following properties of the ICF set were examined.

Overall Fit to the Rasch Model

A nonsignificant value in a χ2 test for item-trait interaction, a mean within ±2.5 (SD <1.5) for the fit residuals of the items and persons indicate good overall fit to the Rasch model [30]. Fit residuals represent the extent to which the observations do not fit a Rasch model. The significance level was adjusted using the Bonferroni correction [31].

Single-Item Fit to the Rasch Model

The good fit of a single category was represented by a nonsignificant χ2 test and a mean of the fit residual values within ±2.5 [32].

Person Separation Index

An acceptable person separation index indicates good internal consistency for the instrument and reflects the ability of the instrument to discriminate between people with different abilities. It has a range of 0 to 1, with higher values indicating a better ability (>0.7 indicates good) [33].

Differential Item Functioning

For an ideal Rasch model, no factors should influence a person’s performance regarding an item except the Rasch factors [30]. The differential item functioning shows the influences of the other factors. In this study, a nonsignificant analysis of variance result was taken as indicating no differential item functioning for a specific category based on gender, age, education level, or etiology.


Patient Characteristics

The demographic and disease characteristics of all 112 spinal cord injury patients are shown in Table 2. Their ages ranged from 18 to 65 years (mean 41.7 [SD 12.3]); 82.1% (92/112) of the patients were younger than 60 years, with patients in the 40- to 49-year age group the most numerous; 83.0% (93/112) of the patients were male. A total of 60.8% (68/112) claimed to have had a middle school education and 25.9% (29/112) only primary education or less. The duration of their disability ranged from 1 to 22 months (mean 7.1 [SD 4.2]); 88.4% (99/112) had been injured for less than a year. Most of the injuries (100/112, 89.3%) were caused by trauma. Most of the patients were injured at the thoracic and cervical levels, accounting for 50.0% (56/112) and 27.7% (31/112), respectively. About half (57/112, 50.9%) of the patients had complete injury.

Table 2. Characteristics of the study sample (n=112).
CharacteristicValue, n (%)
Gender

Male93 (83.0)

Female19 (17.0)
Age in years

18-2927 (24.1)

30-3919 (17.0)

40-4936 (32.1)

50-5920 (17.9)

60-6520 (17.9)
Education

Primary school and below29 (25.9)

Junior high school47 (42.0)

Senior high school21 (18.8)

College and above15 (13.4)
Etiology

Trauma100 (89.3)

Nontrauma12 (10.7)
Duration of disease in months

1-659 (52.7)

7-1240 (35.7)

13-1812 (10.7)

19-221 (0.9)
American Spinal Injury AssociationImpairment scale

Complete injury57 (50.9)

Incomplete injury55 (49.1)
Spinal cord injurylevel

Cervical31 (27.7)

Thoracic56 (50.0)

Lumbar sacral25 (22.3)

Rasch Analysis Results

The 31 ICF categories belonged to body functions and body structures (15), activities and participation (10), and contextual factors (6). To attain adequate fit to the Rasch model for each component, categories that did not fit were deleted as multiple rounds of Rasch analysis were conducted. Table 3 shows the process and results of the Rasch analysis for each component.

Table 3. Summary of results of the Rasch analyses (n=112).
Analysis and actionItem fit residual, mean (SD)Person fit residual, mean (SD)Overall model fitaPerson separation index



χ2P value
Body functions and body structures

1Original categories–0.41 (0.86)–0.32 (0.62)41.5.080.50
Activities and participation

2Original categories–0.48 (2.27)–0.38 (0.80)75.7<.001b0.89

3Deleted family relationships (d760)–0.18 (2.37)–0.30 (0.79)43.3<.001b0.89

6Deleted family relationships (d760) and socializing (d9205)–0.06 (1.59)–0.23 (0.73)24.7.080.89
Environmental factors and personal factors

7Original categories0.25 (1.63)–0.45 (1.30)32.8.001b0.65

8Deleted knowledge about spinal cord injury–0.33 (1.32)–0.43 (1.22)13.6.190.68

aOverall model fit was tested using a χ2 test with a Bonferroni-adjusted P value. The values were all P<.01.

bSignificant according to the Bonferroni-adjusted P value.

In the first-round Rasch analysis, the body functions and body structures component consisting of 15 categories exhibited a nonsignificant χ2 test result for the item-trait interaction (χ230=41.5, P=.08, Bonferroni-adjusted P=.05/15=.003). Additionally, the means for the item and person fit residuals were within ±2.5 (SD <1.5). These results suggested a good fit to the Rasch model. All of the 15 categories exhibited nonsignificant χ2 test results and the means of their fit residual values were also within acceptable limits (Table 4). The person separation index of this component was 0.5. There was no differential item functioning for any of the categories by gender, age, education level, or etiology.

Regarding the initial activities and participation component with 10 categories, in the first-round Rasch analysis, the χ2 test for the item-trait interaction yielded a significant result (χ220=75.7, P<.001, Bonferroni-adjusted P=.05/10=.005). The single-item fit analysis found that four categories, changing basic body position (d410), washing oneself (d510), family relationships (d760), and socializing (d9205), did not fit the Rasch model. The family relationships (d760) and socializing (d9205) categories both exhibited poor fit results in a previous study [13]. Considering the mean fit residual values 3.253 for family relationships (d760) and 2.605 for socializing (d9205) and the category meanings, d760 was deleted first. However, the χ2 test result for the item-trait interaction in the second-round Rasch analysis (after deleting d760) remained significant (χ218=43.3, P<.001, Bonferroni-adjusted P=.05/9=.0056). A third round of Rasch analysis was performed after deleting both family relationships (d760) and socializing (d9205). Although the standard deviation of the overall item fit residuals (SD 1.59) was a little larger than the upper limit, the χ2 test result for the item-trait interaction was no longer significant (χ216=24.7, P=.08, Bonferroni-adjusted P=.05/8=.0063), suggesting good fit to the Rasch model. The single-item fit tests for the remaining 8 categories also yielded good model fit results, with nonsignificant χ2 test results (Table 4). The person separation index of the component was excellent (0.89) and no differential item functioning was detected for any of the categories by gender, age, education level, or etiology.

For the contextual factors component, the first-round Rasch analysis starting with 6 categories indicated poor model fit according to the χ2 test result for the item-trait interaction (χ212=32.8, P<.001, Bonferroni-adjusted P=.05/6=.0083). The following single-item fit analysis showed that the personal factor knowledge about spinal cord injury did not fit well (χ22=19.1, P<.001). After deleting that item, the component displayed satisfactory overall model fit, with a nonsignificant χ2 test result for the item-trait interaction (χ210=13.6, P=.19, Bonferroni-adjusted P=.05/5=.01). The means and standard deviations of the fit residuals for items and persons were both within the acceptable limits. The single-item fit analyses for the remaining 5 categories were also satisfactory, with nonsignificant χ2 test results (Table 4). The person separation index of this component was 0.68, and no differential item functioning was detected for any of the categories by gender, age, education level, or etiology.

Table 4. International Classification of Functioning, Disability, and Health categories retained after multiple rounds of Rasch analysis.
ICFa categoryLocationFit residualχ2bP valueTransformation
Body functions and body structures

1Sleep functions (b134)–1.450–0.7621.0.614

2Emotional functions (b152)–1.281–1.4147.9.023

3Sensation of pain (b280)0.3081.0472.4.304

4Blood vessel functions (b415)2.0340.3953.7.162

5Increased blood pressure (b4200)1.9320.7893.6.163

6Decreased blood pressure (b4201)0.757–1.1061.6.463

7Immunological system functions (b435)1.625–0.9221.8.402

8Respiration functions (b440)1.767–0.1560.8.683

9Weight maintenance functions (b530)0.7290.5232.4.301

10Sexual functions (b640)–2.588–2.0865.5.072

11Procreation functions (b660)–1.640–0.8621.6.442

12Mobility of joint functions (b710)–0.895–0.7911.7.432

13Muscle power functions (b730)–2.610–0.2163.5.171

14Muscle tone functions (b735)–1.026–0.5862.7.254

15Structure of areas of skin (s810)2.3390.0461.2.544
Activities and participationc

16Changing basic body position (d410)0.252–1.5435.6.064

17Moving around using equipment (d465)–0.614–0.2742.2.334

18Washing oneself (d510)–1.697–1.5945.2.084

19Caring for body parts (d520)1.147–0.6370.8.664

20Regulating urination (d5300)–0.8663.3538.5.014

21Regulating defecation (d5301)–1.8410.7130.4.804

22Dressing (d540)0.1270.260.7.724

23Eating (d550) and drinking (d560)3.493–0.7411.3.524
Contextual factorsd

24Assistive products and technology for personal indoor and outdoor mobility and transportation (e1201)1.8050.5852.4.302

25Design, construction, and building products and technology of buildings for private use (e155)–0.1771.7932.7.262

26Acceptance of life in a wheelchair/in bed–0.237–1.2643.3.192

27Coping with everyday life–1.1271.3051.9.392

28Adjustment to new body image–0.263–0.7873.3.192

aICF: International Classification of Functioning, Disability, and Health.

bGoodness of fit of each category was tested using a χ2 test with a Bonferroni-adjusted P value. All were <.01.

cFamily relationships (d760) and socializing (d9205) were deleted because of poor fit.

dKnowledge about spinal cord injury was deleted because of poor fit.


Principal Findings

The results of the Rasch analysis showed good fit to the Rasch model for the different components of the ICF set as implemented in the app after modification. Both overall and single-item fit were satisfactory. There was no differential item functioning for any of the ICF categories by gender, age, education level, or etiology. These results indicate the suitability of the app-based ICF set as an assessment tool for assessing the functioning of SCI patients.

The app-based ICF set is one of many forms of ICF-based electronic health records. Several previous studies have confirmed the role of ICF-based electronic health records in reflecting patient functioning and facilitating rehabilitation [6,34]. As a unified and standard language originally developed for multidisciplinary use, the original ICF was relatively easily understood by different disciplines and suitable for multidisciplinary teamwork [2]. However, some obstacles still existed, including the complexity of the terminology, lack of operationalization of the ICF qualifiers, and training overload for ICF users. The satisfactory internal construct validity of the set developed in the study is mainly due to the selection of suitable ICF categories and standardized assessment enabled by the app. In this study, the app applied many fewer ICF categories (31) than the comprehensive ICF core set (168 categories) and the brief core set (33 categories) for SCI patients issued by the WHO’s ICF research branch [35]. Additionally, the categories were more specific because they were originally identified as good outcome indicators for SCI patients in China [20,26]. Each ICF category focuses on a specific area and is independent from the others, which helped to ensure the overall fit to the Rasch model based on the assessments’ content.

Together’s verbal prompts standardize assessment and give more consistent assessment results. The transformation guidelines operationalize the 5 ICF qualifiers simply and effectively. With the help of the app, the ICF qualifiers can automatically be matched to the initial clinical assessment results. No additional training on ICF terminology or qualifiers is needed. The process reduces the differences among assessors and makes presentation of the ICF data more convenient and intelligent.

Family relationships (d760), socializing (d9205), and knowledge about spinal cord injury were deleted. In a previous study, d760 and d9205 also exhibited poor model fit [13]. Family relationships are the basis for good functioning in the family, and socializing reflects a patient’s social participation. Both of them are influenced by many factors such as age, severity of the injury, and financial considerations [36,37]. For an item to have perfect fit to the Rasch model, no other factors should influence a person’s performance regarding the item except for the person’s ability and the item’s difficulty, which may explain why the two factors did not fit the Rasch model. The ICF does not classify personal factors because of the large social and cultural variance associated with them [1]. The personal factors assessed in this ICF set were identified by multiround expert surveys [26]. Knowledge about SCI is a broad concept covering many aspects such as injury outcomes, functional rehabilitation, and preventing complications, and it is influenced by multiple factors such as the patient’s level of education, efficacy self-perceptions, and any health education they have received [38]. This may be why the item did not fit the Rasch model well. However, family relationships or support, social participation and the knowledge about SCI are important indicators for the outcomes of rehabilitation. To ensure the completeness of a patient’s health records, it is suggested that they be retained as part of the basic health data collection but not as part of the assessment instrument. Further research should use professional measurements that reflect family relationships, socializing, and the knowledge level for SCI patients.

It is worth noting that, in contrast to the person separation index of the activities and participation component, the person separation index of the other two components was not ideal (0.5 and 0.68), which reflects the poor internal consistency of the two components. This may be related to the different measuring guidelines used. Four guidelines were used in the study to transform the input of the medical staff, based on patient information in the form of percentages, wording from patient reports, frequency with which a problem was observed, and scores on routine clinical instruments or standards. A previous study [29] found that the categories involving guideline 4 (which transforms scores of a routine clinical instrument or standards into the ICF qualifiers) had the best interrater reliability, and the categories involving guideline 2 (which transforms the wording from patient reports into the ICF qualifiers) had the lowest interrater reliability. In this study, all the activity and participation categories used the guideline with the highest reliability (guideline 4), while all the contextual factor categories used the guideline with the lowest reliability (guideline 2). Although the app automatically transforms the initial data into the ICF qualifiers, subjectivity and uncertainty still existed in the patients’ initial reports and recall, which may explain the unsatisfactory person separation index of the two components.

Limitations

In interpreting these results, it is important to keep in mind that the relatively small sample may have influenced the representativeness of the results. Also, although the app was designed to assess SCI patients at home during transitional care via its remote follow-up function, the study data were collected in face-to-face interviews before the participants were discharged. Further validation with larger samples and remote assessment via the app’s communication function are needed.

Conclusions

This study has confirmed the suitability of the Together app as an assessment tool. With relatively fewer but more specific ICF categories, it overcomes some of the limitations related to applying the ICF and makes assessment results more reliable and consistent. The app opens up a new way to use the ICF with SCI and electronic health records. In the future, the app could be used to capture information about the functioning of SCI patients at home remotely. Such assessment results would help to monitor patients’ functional changes and differences, learn their needs, identify their problems, and provide evidences for further interventions if necessary.

Acknowledgments

The authors would like to express special thanks to Bing Xie and Haishan Pan, the engineers who provided technical support for the development of the app. This study was supported by China’s National Natural Science Foundation (grant number 71603293) and the Fundamental Research Funds for the Central Universities (grant number 20ykpy88). The sponsors had no role in the study’s design; in the collection, analysis or interpretation of data; in writing of the report; or in the decision to submit the article for publication.

Authors' Contributions

MJ drafted the manuscript and assisted with managing the app. JT was responsible for conducting the study and revising the manuscript. SX, XH, and YW helped collect the data. TL was responsible for the management of the app. TY participated in the study’s design. KL was responsible for the project design, implementation, quality control, app management, and manuscript revision. All the authors reviewed the submitted manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Together tutorial.

MP4 File (MP4 Video), 14058 KB

References

  1. International Classification of Functioning, Disability and Health. Geneva: World Health Organization; 2001.   URL: https://www.who.int/classifications/drafticfpracticalmanual2.pdf?ua=1 [accessed 2020-10-05]
  2. Tempest S, Harries P, Kilbride C, De Souza L. Enhanced clarity and holism: the outcome of implementing the ICF with an acute stroke multidisciplinary team in England. Disabil Rehabil 2013;35(22):1921-1925 [FREE Full text] [CrossRef] [Medline]
  3. Nilmart P, Vongsirinavarat M, Somprasong S, Apinonkul B. Development of an extensive assessment list for knee osteoarthritis based on the International Classification of Functioning, Disability and Health: a Delphi study. Int J Rehabil Res 2019 Sep;42(3):240-248. [CrossRef] [Medline]
  4. Kisser U, Adderson-Kisser C, Coenen M, Stier-Jarmer M, Becker S, Sabariego C, et al. The development of an ICF-based clinical guideline and screening tool for the standardized assessment and evaluation of functioning after head and neck cancer treatment. Eur Arch Otorhinolaryngol 2017 Feb;274(2):1035-1043. [CrossRef] [Medline]
  5. Vreeman DJ, Richoz C. Possibilities and implications of using the ICF and other vocabulary standards in electronic health records. Physiother Res Int 2015 Dec;20(4):210-219 [FREE Full text] [CrossRef] [Medline]
  6. Stucki G, Bickenbach J. Functioning: the third health indicator in the health system and the key indicator for rehabilitation. Eur J Phys Rehabil Med 2017 Feb;53(1):134-138 [FREE Full text] [CrossRef] [Medline]
  7. Bautz-Holter E, Sveen U, Cieza A, Geyh S, Røe C. Does the International Classification of Functioning, Disability and Health (ICF) core set for low back pain cover the patients' problems? A cross-sectional content-validity study with a Norwegian population. Eur J Phys Rehabil Med 2008 Dec;44(4):387-397 [FREE Full text] [Medline]
  8. Hilfiker R, Obrist S, Christen G, Lorenz T, Cieza A. The use of the comprehensive International Classification of Functioning, Disability and Health Core Set for low back pain in clinical practice: a reliability study. Physiother Res Int 2009 Sep;14(3):147-166. [CrossRef] [Medline]
  9. Starrost K, Geyh S, Trautwein A, Grunow J, Ceballos-Baumann A, Prosiegel M, et al. Interrater reliability of the extended ICF core set for stroke applied by physical therapists. Phys Ther 2008 Jul;88(7):841-851. [CrossRef] [Medline]
  10. Uhlig T, Lillemo S, Moe RH, Stamm T, Cieza A, Boonen A, et al. Reliability of the ICF Core Set for rheumatoid arthritis. Ann Rheum Dis 2007 Aug;66(8):1078-1084 [FREE Full text] [CrossRef] [Medline]
  11. Cao R, Xu G, Ding X, Lin F, Li J. Study on the reliability and validity of international classification of function, disability and health brief core sets for stroke. Chin J Rehab Med 2011;26:715-719. [CrossRef]
  12. Gao Y, Yan T, You L, Li K. Developing operational items for the International Classification of Functioning, Disability and Health Rehabilitation Set: the experience from China. Int J Rehabil Res 2018 Mar;41(1):20-27. [CrossRef] [Medline]
  13. Li K, Yan T, You L, Xie S, Li Y, Tang J, et al. Psychometric properties of the International Classification of Functioning, Disability and Health set for spinal cord injury nursing based on Rasch analysis. Disabil Rehabil 2018 Feb;40(3):338-345. [CrossRef] [Medline]
  14. Barton AJ. The regulation of mobile health applications. BMC Med 2012;10:46 [FREE Full text] [CrossRef] [Medline]
  15. Kaplan AL, Cohen ER, Zimlichman E. Improving patient engagement in self-measured blood pressure monitoring using a mobile health technology. Health Inf Sci Syst 2017 Dec;5(1):4 [FREE Full text] [CrossRef] [Medline]
  16. Semple JL, Sharpe S, Murnaghan ML, Theodoropoulos J, Metcalfe KA. Using a mobile app for monitoring post-operative quality of recovery of patients at home: a feasibility study. JMIR Mhealth Uhealth 2015;3(1):e18 [FREE Full text] [CrossRef] [Medline]
  17. Wang Q, Zhao J, Huo X, Wu L, Yang L, Li J, et al. Effects of a home care mobile app on the outcomes of discharged patients with a stoma: a randomised controlled trial. J Clin Nurs 2018 Oct;27(19-20):3592-3602. [CrossRef] [Medline]
  18. Dahlberg K, Jaensson M, Nilsson U. “Let the patient decide”: person-centered postoperative follow-up contacts, initiated via a phone app after day surgerycondary analysis of a randomized controlled trial. Int J Surg 2019 Jan;61:33-37 [FREE Full text] [CrossRef] [Medline]
  19. De La Cruz Monroy MFI, Mosahebi A. The use of smartphone applications (apps) for enhancing communication with surgical patients: a systematic review of the literature. Surg Innov 2019 Apr;26(2):244-259. [CrossRef] [Medline]
  20. Liu T, Li K, Xie S, Wang Y, Tang J, He X, et al. Development of the APP for transitional care of people with spinal cord injury based on the ICF. Chinese J Rehab Med 2019;34:154-158. [CrossRef]
  21. Zhang N, Zhou M, Liu N, Liu X, Zhang Y. An investigation study on quality management of spinal cord injury rehabilitation in China 2016. Chinese J Rehab Med 2018;33:1137-1141. [CrossRef]
  22. Zeng X, Wang X, Xu J, Chen T, Xu J. Investigation on the rehabilitation of patients with amputees, spinal cord injuries and craniocerebral injuries after the earthquake in Shifang, Sichuan province. Chinese J Rehab Med 2010;25:771-774. [CrossRef]
  23. Chen B, Jiang C, Niu W. Research status of rehabilitation needs of patients with spinal cord injury in Wenchuan earthquake. Chinese J Hygiene Rescue 2017;3:72-74. [CrossRef]
  24. Xie H, Yang Y, Wu A, Shen C, Hu L, Zhang J, et al. Status quo and development of hope houses for individuals with spinal cord injury in Shanghai from perspective of managers and organizers. Chinese J Rehab Theory Pract 2019;25:1221-1227. [CrossRef]
  25. SunYat-Sen University. Distance transitional care system for spinal cord injury patients. Copyright Protection Center of China Nov2018SR904705. 2018.   URL: www.ccopyright.com [accessed 2020-10-09]
  26. Li K, Xie S, Wang Y, Tang J, He X, Liu T, et al. Outcome indicators in the transitional care of people with spinal cord injury in China: a Delphi survey based on the International Classification of Functioning, Disability and Health. Disabil Rehabil 2019 Feb 06:1-9. [CrossRef] [Medline]
  27. Tesio L. Measuring behaviours and perceptions: Rasch analysis as a tool for rehabilitation research. J Rehabil Med 2003 May;35(3):105-115 [FREE Full text] [CrossRef] [Medline]
  28. Schuld C, Franz S, Brüggemann K, Heutehaus L, Weidner N, Kirshblum SC, EMSCI study group. International standards for neurological classification of spinal cord injury: impact of the revised worksheet (revision 02/13) on classification performance. J Spinal Cord Med 2016 Sep;39(5):504-512 [FREE Full text] [CrossRef] [Medline]
  29. Li K, Yan T, You L, Xie S, Li Y, Tang J, et al. The inter-rater reliability of the International Classification of Functioning, Disability and Health set for spinal cord injury nursing. Int J Rehabil Res 2016 Sep;39(3):240-248. [CrossRef] [Medline]
  30. Grill E, Stucki G. Scales could be developed based on simple clinical ratings of International Classification of Functioning, Disability and Health Core Set categories. J Clin Epidemiol 2009 Sep;62(9):891-898. [CrossRef] [Medline]
  31. Bland JM, Altman DG. Multiple significance tests: the Bonferroni method. BMJ 1995 Jan 21;310(6973):170 [FREE Full text] [CrossRef] [Medline]
  32. Cieza A, Hilfiker R, Boonen A, Chatterji S, Kostanjsek N, Ustün BT, et al. Items from patient-oriented instruments can be integrated into interval scales to operationalize categories of the International Classification of Functioning, Disability and Health. J Clin Epidemiol 2009 Sep;62(9):912-921. [CrossRef] [Medline]
  33. Tennant A, Conaghan PG. The Rasch measurement model in rheumatology: what is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Rheum 2007 Dec 15;57(8):1358-1362 [FREE Full text] [CrossRef] [Medline]
  34. Maritz R, Aronsky D, Prodinger B. The International Classification of Functioning, Disability and Health (ICF) in electronic health records. a systematic literature review. Appl Clin Inform 2017 Sep 20;8(3):964-980. [CrossRef] [Medline]
  35. ICF-based documentation tool: ICF Core Sets in clinical practice. Geneva: World Health Organization; 2012.   URL: https://www.icf-core-sets.org/ [accessed 2020-10-05]
  36. Noreau L, Fougeyrollas P. Long-term consequences of spinal cord injury on social participation: the occurrence of handicap situations. Disabil Rehabil 2000 Mar 10;22(4):170-180. [CrossRef] [Medline]
  37. Kennedy P, Lude P, Taylor N. Quality of life, social participation, appraisals and coping post spinal cord injury: a review of four community samples. Spinal Cord 2006 Feb;44(2):95-105. [CrossRef] [Medline]
  38. Zhang W, Li S, Li X, Chen C, Tang L, Hu J, et al. [Currency survey and influencing factors analysis on the knowledge and behavior related to brucellosis among occupational workers in Jianyang City]. Sichuan Da Xue Xue Bao Yi Xue Ban 2019 Jul;50(4):551-555. [Medline]


ICF: International Classification of Functioning, Disability, and Health
NRS: numerical rating scale
SCI: spinal cord injury
WHO: World Health Organization


Edited by G Eysenbach; submitted 27.05.20; peer-reviewed by C Jacob, L Grepo; comments to author 15.08.20; revised version received 26.09.20; accepted 29.09.20; published 11.11.20

Copyright

©Mengmeng Jia, Jie Tang, Sumei Xie, Xiaokuo He, Yingmin Wang, Ting Liu, Tiebin Yan, Kun Li. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 11.11.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mHealth and uHealth, is properly cited. The complete bibliographic information, a link to the original publication on http://mhealth.jmir.org/, as well as this copyright and license information must be included.