Systematic literature review and validity evaluation of the Expanded Disability Status Scale (EDSS) and the Multiple Sclerosis Functional Composite (MSFC) in patients with multiple sclerosis

1Multiple sclerosis (MS) is a chronic inflammatory disease of the central nervous system that mainly affects young adults. The disease is characterized by the occurrence of relapsing neurological deficits that affect different functional systems of the central nervous system. The majority of patients initially present with a relapsing remitting disease course (RRMS > 80%), however after 10 – 15 years about 60% of these patients show a transition into a secondary progressive disease course (SPMS) that is characterized by a gradual decline of neurological function. In less than 15% of cases, the disease course is progressive from the onset (primary progressive MS, PPMS).

2In recent decades, a number of instruments have been developed that describe the clinical severity and the functional deficits in multiple sclerosis. These instruments are increasingly used as an endpoint in clinical trials to assess the effectiveness of therapeutic interventions.

3The most popular and widely used instrument is the Expanded Disability Status Scale (EDSS) of Kurtzke. The EDSS is a clinician-administered assessment scale evaluating the functional systems of the central nervous system. The EDSS is used to describe disease progression in patients with MS and to assess the effectiveness of therapeutic interventions in clinical trials. It consists of ordinal rating system ranging from 0 (normal neurological status) to 10 (death due to MS) in 0.5 increments interval (when reaching EDSS 1). The lower scale values of the EDSS measure impairments based on the neurological examination, while the upper range of the scale (> EDSS 6) measures handicaps of patients with MS. The determination of EDSS 4 – 6 is heavily dependent on aspects of walking ability.

4Another important instrument is the Multiple Sclerosis Functional Composite (MSFC), which was developed by the MS Society’s Clinical Assessment Task Force as an additional clinical measure of MS disability progression. The primary goal for creating the MSFC was to improve the standard measure of MS disability for clinical trials and to develop a multidimensional metric of overall MS clinical status. The MSFC is a three-part performance scale for evaluating the degree of impairment in MS patients. It includes the assessment of leg function by moving a short walking distance (“Timed 25-Foot Walk”, T25FT), the assessment of arm function using breadboard test (“9-Hole Peg Test”, 9HPT) and an attention/concentration test to assess cognitive functions (“Paced Auditory Serial Addition test”, PASAT). An integrated MSFC score is calculated using z-scores. There is an ongoing debate about which dimensions to include in the MSFC (e.g. the inclusion of a vision testing) as well as how the reference population affect the standardized scoring (z-scores) of the MSFC. In recent years, the MSFC is increasingly used in clinical trials.

5A number of other instruments are available to assess MS: the Ambulation Index (AI), the Scripps Neurological Rating Scale (SNRS) and the Illness Severity Scale (ISS), the Guy’s Neurological Disability Scale (GNDS) and the Multiple Sclerosis Impairment Scale (MSIS). Furthermore, the overall Functional Independence Measure (FIM) and the Cambridge Multiple Sclerosis Basic Score (CAMBS) are clinical assessment instruments that could be used to assess MS. Specific instruments for measuring health-related quality of life in MS patients are the Multiple Sclerosis Quality of Life-54 (MSQOL-54) and the Multiple Sclerosis Quality of Life Inventory (MSQLI). Only a few of these instruments meet the requirements of methodological standards (e.g. validity, reliability, responsiveness), particularly for use in clinical trials. None of these instruments is recognized to use in clinical trials without any restrictions.

6The aim of this study was to identify strengths and weaknesses of EDSS and MSFC. We investigated to what extent the psychometric properties of the two most important instruments – EDSS and MSFC – meet the methodological standards and what value they have in clinical trials. Although different methodological characteristics of the EDSS are the focus of many studies, a summary and synthesis of the study results is missing. In addition, the importance the MSFC is still a matter of debate and a summary of its methodological characteristics is also missing. We addressed this gap by conducting a systematic literature review to specifically answer the following questions:

7• Are the instruments suitable to detect the efficacy and/or effectiveness of clinical interventions and to monitor the disease progress?

8• Do the methodological properties of EDSS and MSFC meet the required standards of objectivity, reliability and validity as applied to instruments for assessment of disease impact in MS?

9• What is the evidence of the practicality and sensitivity to change of these instruments?

10• Which methodological shortcomings of EDSS and MSFC must be considered in interpretation of study results?

11• Which minimally important difference (minimal clinically important difference, MCID) is reported for EDSS and MSFC?

12The study was conducted in two phases. First, an exploratory electronic literature search was undertaken to (a) develop and optimize a systematic search strategy and (b) estimate the extent and quality of available studies. Then, a systematic literature search was conducted using the developed search algorithms. The databases MEDLINE (PubMed), ISI Web of Science, EMBASE, PsycINFO and CINAHL & PSYNDEX were systematically searched. Search terms were: Multiple Sclerosis + Expanded Disability Status Scale OR EDSS (for EDSS) and Multiple Sclerosis + Multiple Sclerosis Functional Composite OR MSFC (for MSFC) (Table 1).

13Overall, 3680 results were identified (4272 of the initially generated 8132 hits were excluded as duplicates); the abstracts were reviewed by two independent scientists and structured according to the following criteria:

141) Psychometric studies: validity, reliability, sensitivity of change, head-to-head comparisons, further developments of EDSS or MSFC and reviews.

152) Publications on clinical trials in which the EDSS and/or the MSFC is used as outcome measure or part of it.

163) Other types of studies, if these relate to the central question (for example, review articles).

17When the theme did not meet the research question, the abstract was excluded. A total of 684 publications were classified as relevant and ordered as full-text publications in electronic copy or selected in lending (psychometric publications n = 295; papers on clinical studies n = 364; reviews, summaries etc. n = 25).

18In a further selection process the full-text psychometric papers (n = 295) were reviewed and subdivided into:

191) Studies that include information about the psychometric properties of the investigated scores (validity, reliability, sensitivity to change, and head-to-head comparisons).

202) Studies that focus on a different instrument than EDSS and/or MSFC (without acceptable psychometric information about the EDSS or MSFC).

213) Studies that are not relevant to the central study questions.

22The categories 2 and 3 were excluded. Ultimately, 120 papers were identified as relevant methodological publications. Figure 1 gives an overview of the selection process.

23In addition, the publications on clinical trials were reviewed. Furthermore, the public register of clinical trials ClinicalTrials.gov by the U.S. National Institutes of Health was systematically searched and an overview of relevant clinical studies generated.

24We identified 120 publications with methodological content for EDSS and/or MSFC. Table 2 gives an overview of scores distribution and period of publication.

25Numerous relevant studies of EDSS were published before 2000. The majority of the MSFC studies as well as the head-to-head comparisons were published between the years 2000 and 2009.

26In the early clinical studies evaluating therapeutic interventions for MS, the effectiveness of treatment was exclusively measured by reducing the number of relapses, due to the importance of relapses for disease progression. However disease progression measured by EDSS was introduced as a primary endpoint in 1996 when the therapeutic effect of Interferon-beta 1a i.m. was evaluated. Currently, progression of disability measured by EDSS is the most important secondary endpoint in MS trials addressing RRMS patients and the most important primary endpoint in all studies dealing with therapeutic interventions in patient with a progressive disease course. Recent studies usually also include the MSFC-score as an endpoint in clinical trials.

27Therefore the current guidelines of the European Medicines Agency (EMA) recommend prevention and delay of disability progression for SPMS and PPMS studies and relapses, disease progression and the increase in disability for studies on RRMS or SPMS (with superimposed relapses) as primary or secondary endpoints.

28Overall, we identified 54 full-text publications in which the psychometric properties of the EDSS (without comparison to the MSFC) were examined. One of the studies is the original work of Kurtzke, 9 were summary papers or reviews, 21 investigated the validity, 13 the reliability, 8 the sensitivity of change and 10 included modifications of the EDSS, especially versions for self-reporting. In addition, there is a simulation study on power (overlaps are possible). (The 40 studies which consider the MSFC as well are not listed here; for more information see MSFC.)

29Kurtzke himself did not test the psychometric properties of EDSS. A series of studies on the properties and the functional systems of the EDSS was conducted as awareness of psychometric methods increased. Most of the early studies concentrated on reliability. A first study considering reliability, validity and sensitivity to change is the work of Sharrack and colleagues. A detailed study on the validity, reliability (inter-and intrarater reliability), sensitivity to change, and floor and ceiling effects on a sample of patients with MS is published from Hobart and colleagues in 2002.

30The validity of the EDSS has been established in numerous studies: Amato and colleagues certify a good validity of the EDSS; Ebers described the EDSS as “well validated”. In early studies, the EDSS was correlated with the Barthel Index (BI), the London Handicap Scale (LHS), the Scripps Neurological Rating Scale (SNRS), the Functional Independence Measure (FIM) and physical function ability of the SF-36; strong to very strong correlation could be shown, weaker results were reported between the EDSS and the ambulation index (AI). In addition, the EDSS correlated with neuropsychological impairment and brain changes measured by MRI – with no or weak correlations. Results of correlations of EDSS with patient-reported outcomes were heterogeneous, from moderate to no correlation.

31In studies checking the validity of the MSFC, the EDSS is often used as gold standard reference (see the validity of the MSFC in the next section).

32A criticism of several studies is the limited reliability of EDSS measurements. For inter-rater reliability kappa values between 0.32 to 0.76 for the EDSS and between 0.23 to 0.58 for the individual functional systems were reported. The intra-rater agreement is slightly higher than the inter-rater, they both show greater variability for lower EDSS scores (1.0-3.5) than for higher score values.

33Different intervals of agreement (exact match, one step (0.5 points), two steps (1.0 points), etc. on the EDSS scale) affect the degree of reliability; studies that attest good reproducibility to the EDSS allow higher deviations. From a deviation of 3 steps on the EDSS scale (1.5 points) almost all studies report good to perfect inter-rater reliability.

34In the literature review, the EDSS was found somewhat sensitive to changes in disease progression. Schwid and colleagues compared limitations of walking ability in MS patients, the EDSS and Ambulation Index and found them to be less sensitive to change than the Dmax (the maximum distance that a person can go) and T8 (time to walk 8 m). Similarly, Vaney and colleagues reported lesser changes in EDSS than in the Rivermead Mobility Index (RMI), the AI and the 10 m walking time test. Hohol and colleagues reported a lower sensitivity to change in EDSS compared with Disease Steps. In turn, Koziol and colleagues observed in a clinical study that the EDSS and the Scripps Neurological Rating Scale could reflect improvements in the cladribine group and deteriorations in the placebo group appropriately.

35Moreover, some studies showed that the rates of change varied depending on the initial value. Greater rates of change were observed for minor severity of disease, but from an EDSS score of 6, the EDSS showed very little change in sensitivity. Ravnborg and colleagues reported higher annual rates of change (worsening health) at low EDSS baseline.

36The EDSS shows a bimodal frequency distribution in a number of studies with only a few values in the central region and two peaks around the values 3 and 6.

37In numerous studies the unequal interval distances between the EDSS have been criticized,, particularly in the context of clinical trials. This leads to different meanings of change depending on the position on the scale: a difference between the values 1.0 and 2.0 has a different relevance as between 6.0 and 7.0. Amato and Ponziani as well as Drulovic and colleagues criticized that the use of parametric statistical methods is not possible. The EDSS is also not suitable for use in economic analyses. The non-linearity was demonstrated in studies comparing the EDSS with the health-related quality of life.

38As reported in the literature, the lower scale values (0-4.0) are influenced by impairments detected by the neurological exam of eight functional systems, while the values above 4.0 are mainly based on the walking ability, and values above 6 mainly on patients handicaps.

39A clear recommendation how to interpret changes in EDSS-values cannot be found in the literature. Noseworthy et al. recommend a progression of 1.0 as a meaningful change in clinical trials. Similarly, Healy and colleagues recommended using a continuing progression by one unit on the scale for at least 6 or 12 months. On the other hand, Francis et al. suggested a change of 1.5 points to be more appropriate. For patients with an EDSS of 5.5 or higher at inclusion in the study, progression by 0.5 was deemed sufficient, since changes in this range of the scale are easily perceptible.

40The guidelines of the European Medicines Agency (EMA) suggest that an average change from baseline is not an adequate efficacy parameter. Instead, they recommend defining treatment success or treatment failure of either reaching a certain EDSS score or a sustained change in sufficient volume. A separate consideration of the lower and upper value range of the EDSS is recommended: 1 point on the EDSS scale with a baseline EDSS score less than or equal to 5.5 and 0.5 points in an EDSS score over 5.5.

41A commonly recognized benefit of the EDSS is its good comparability of results from different studies. The EDSS is fairly robust for measurements over a long period.

42Overall, we identified 66 full-text publications in which the psychometric properties of the MSFC are reported. Of these, 10 were reviews or summaries, 14 examined the validity, 18 the sensitivity to change and 4 the reliability. Seven publications included modifications to the MSFC. A total of 40 publications also contain information on the EDSS with two publications assessing the superiority of the MSFC over the EDSS (overlaps are possible).

43MSFC has been shown to correlate with other indicators of disease in MS, including the EDSS, MRI measures (although correlation here is weak), patient reported health and employment. The predictive validity of MSFC has also been shown for EDSS, brain atrophy, SPMS disease and self-reported health.

44The EDSS is used most often to test the validity of the MSFC – studies found strong correlations between the two scores, ranging from -0.41 to -0.83. The PASAT is the weakest correlate. Kalkers and collegues found that while overall MSFC scores distinguished across MS disabilities, the PASAT could not distinguish as well. In fact, Brochet et al. found no correlation between PASAT and EDSS. Rudick and collogues found that MSFC progression seem to be driven by one component of the MSFC, the Timed 25-Foot Walk (T25FWT).

45Correlation with MRI measures and patient reported outcomes were inconsistent, showing moderate to no correlations. For example, MSFC was correlated with the physical component of the SF-36 scale and the sickness impact profile, but not with the mental component of the SF-36. Change in MSFC score also is shown to have a very weak correlation with change in SF-36 score.

46The source publication found MSFC reliability to be high (intra-rater ICC = 0.98, inter-rater ICC = 0.96). Inter-rater and intra-rater reliability were shown to be high for MSFC overall score as well as its component measures. The intraclass correlation (ICCs) were above 0.8 across studies. Schwid and colleges found that MSFC scores varied over measurement periods (over 5 days) but all within 20% change in raw scores.

47However, the 9HPT and PASAT have well demonstrated practice effects, meaning that as participants learn about the test, their scores improve. This is a general problem of all studies that follow patients over time. The MSFC manual suggests administering the PASAT at least three times prior to baseline measurement to address this practice effect as there is some evidence that practice effects stabilized by the 4th observation. However, some evidence point to practice effect persisting beyond fourth testing repeats.

48Some studies did not detect notable practice effects. However, due to potential artifacts that may lead to overestimations of improvement in MSFC scores, precautions should be taken during data collection.

49In the source database, MSFC was found to significantly decline from baseline (z = -0.14) to 2 years (z = -0.14) for patients in placebo groups of clinical trials. They also found patients with MSFC deteriorating by at least one standard deviation to have an odds ratio of 2.1 for sustained EDSS deterioration.

50Some studies indicate the MSFC to be more sensitive than EDSS to detecting changes in disease. However, due to missing dimensions (e.g visual function) there is also evidence that the MSFC is not more sensitive than the EDSS. Others found the PASAT does not reflect disease progression in certain MS populations [11 for PPMS].

51Fischer et al. found that it took 15 minutes for trained personnel to administer the MSFC to patients (using a small sample of 10 patients). The PASAT, however, is notoriously disliked by patients. From our review, some studies found patients not willing to undergo the PASAT.

52A major drawback of the MSFC is its interpretability, especially in the scoring mechanism. Although choice of reference population does not affect statistical significance, it does change the value of the z-scores, making cross-trial comparison problematic. Some studies examined reference population choice on MSFC scores and found the question of reference population problematic.

53A further objection to z-score interpretability is its abstractness from clinically meaningful values. A solution involves using clinically relevant cut-offs in the raw scores of MSFC dimensions as opposed to z-scores. Bosma et al. found optimal cutoff values to be 20% change for the T25FW and 20% for the 9HPT but could not determine a relevant cutoff for PASAT. Hoogervorst et al. established that 20% change in all three MSFC measures were meaningful to patient’s own perception of health (using the GNDS as a reference measure). Similarly, other studies found the 20% change in T25FW and 9HPT reflect disease status in MS patients. Rudick et al. further found, using data from the AFFRIM and SENTINEL trials, that a 15 or 20% change, sustained for 3 months, seems to be an adequate way to quantify the MSFC dimensions. However, others found the 20% not sufficient to detect neurological worsening.

54In addition to the review, a systematic search in the public register of clinical trials ClinicalTrials.gov by the U.S. National Institutes of Health was conducted to identify relevant clinical trials for MS. Under the topic multiple sclerosis (search terms) 904 studies were identified. In a second step, clinical phase III or IV studies on interferon beta-1b, interferon beta-1a, glatiramer acetate, fingolimod, natalizumab, mitoxantrone and cyclophosphamide as well as – due to their current importance – studies on laquinimod, teriflunomide, BG-12 and alemtuzumab were extracted. Finally, 66 clinical phase III or IV studies to a relevant agent of MS therapy could be identified. In 50 of those studies EDSS and/or MSFC were used as primary and/or secondary outcome measurement method (in 15 studies EDSS/MSFC were not reported, in 1 study EDSS was named as a tertiary endpoint). In 9 of the 50 relevant studies the MSFC was used in addition to the EDSS, but there is no study that used only the MSFC (Table 3). It was found that the EDSS has been used preferentially as primary and secondary outcome measurement.

55This study examined the extent to which the psychometric properties of the most important instruments to assess disease progression in MS – EDSS and MSFC – meet the methodological standards and what value they have as outcome measures in clinical trials.

56In a systematic literature review of 120 relevant full-text publications, we found that while the EDSS is the most widely used instrument in clinical trials assessing the effectiveness of therapeutic interventions, the MSFC is becoming increasingly important.

57Single methodological characteristics of the two instruments have been investigated in numerous studies. The EDSS has some documented weaknesses in reliability and sensitivity to change. Although the MSFC was developed rigorously, its weaknesses include interpreting the z-scores (which are used to calculate the summary score from the three components), the learning effects of the PASAT, low acceptance by patients and lack of a visual dimension. However, the MSFC has better sensitivity and reliability compared to EDSS. The methodological quality criterion of validity is sufficiently met by both instruments.

58Despite the criticisms, in particular the EDSS could be classified as an important endpoint. In neither publication, the EDSS (or MSFC) are discussed as a surrogate parameter (in contrast to MRI measurements). Cohen et al. emphasized in their review of clinical outcome parameters, the general acceptance of the EDSS as an endpoint by different authorities. In contrast, the MSFC is usually used as a secondary endpoint.

59The importance of the instruments is also reflected in our evaluation of clinical trials: in 50 of 66 identified relevant phase-III and phase-IV studies, disease progression as measured by EDSS was used as primary (27 studies) or secondary outcome (23 studies). The MSFC was used in eight studies as a secondary endpoint and only one study as a primary endpoint.

60In summary, both instruments are acceptable outcome criteria to assess the effectiveness of clinical interventions and to monitor disease progression. When using the EDSS, its limited inter-and intra-rater reliability should be considered. All possibilities to increase reliability should be used, including training of investigators, assessment by the same doctor/neurologist during the study, specified times of detection, standardized protocols for neurological examination and a precise definition of all requirements. When using the MSFC, the learning effects of the PASAT and 9HPT should be considered and controlled (e.g. using control groups).

61Concerning the natural history of MS, Weinshenker and colleagues observed an average change of 0.5 points on the EDSS scale in a year. A clear recommendation on interpreting changes in EDSS and MSFC values does not yet exist. EDSS changes by 1.0 points from a baseline EDSS less than or equal to 5.5 and 0.5 points over a baseline 5.5 are commonly recognized as a clinically increase in disability. However, it is now understood that it is more accurate to define disability change as a sustained change for 12 weeks or, even more reliably, for 24 weeks.

62For the MSFC, changes by 20% in the individual components are considered to be clinically relevant. However, it is difficult for a clinician to understand what a 20% change in MSFC subscores is while a significant EDSS change is more intuitively understood.

63Overall, the literature reveals that the EDSS is the most widely used and best-known instrument to assess disease progression in MS. The advantages and disadvantages of EDSS and MSFC have been investigated frequently. The great advantage of the EDSS is its international acceptance (including the EMA) as a primary endpoint in clinical trials. Because it is so commonly used, studies that use the EDSS can easily compare results to other findings. Following the current literature, we can conclude that the importance of the EDSS will be unabated in future. Major changes of the EDSS are not recommended, as not to jeopardize its advantage.

64In addition, the use of other measurements as clinical trial endpoints (e.g. the MSFC) is recommended to provide information on dimensions no covered in the EDSS, such as upper limb function or cognitive skills.

65The assessment instruments we examined, EDSS and MSFC, both are frequently and internationally used in clinical studies – despite their well know methodological limitations. They are suitable to detect patient-relevant endpoints in MS with reasonable validity in spite of some methodological concerns.

66Written informed consent was obtained from the patients for the publication of the reviewed studies and accompanying images.

67The authors declare that they have no competing interests.

68SMM conducted the literature review and drafted the manuscript. YSF conducted the literature review and drafted the manuscript. MM drafted the manuscript and gave clinical input. FWD contributed to the clinical studies section and drafted the manuscript. TK conceptualized the study and critically reviewed the manuscript. All authors read and approved the final manuscript.

69The pre-publication history for this paper can be accessed here:

70http://www.biomedcentral.com/1471-2377/14/58/prepub