The Harris hip score: Do ceiling effects limit its usefulness in orthopedics?

1In evidence-based medicine, the use of clinically important patient outcomes in clinical research is paramount. Quality of life and function are usually measures that are important for patients and healthcare providers. The Harris hip score (HHS) is one such measure that has frequently been used to measure outcome after total hip arthroplasty.

2developed this score with a rating scale of 100 points and with domains of pain, function, activity, deformity, and motion. It was designed for use in young men with often long-standing severe secondary osteoarthritis after a fracture of the acetabulum that was operated on with a Smith-Petersen vitallium mold arthroplasty. Although not originally designed for hip arthroplasty (THA) patients, it is widely used for this population. Since its introduction, several authors have reported the score to be a valid outcome measure for THA based on good construct validity alone. While construct validity is important, it is not the sole factor in evaluating the overall validity of an outcome questionnaire. Reliability, internal consistency, content validity, and responsiveness are also important. A questionnaire is suitable only when all 5 psychometric properties are of sufficient quality.

3Content validity assesses the extent to which a metric measures all aspects of a certain phenomenon. The amount of ceiling and floor effects present determine the quality of the content validity. A floor effect occurs when several of the patients score the lowest possible score, whereas a ceiling effect occurs when several of the patients score the highest possible score.

4Given the persistent use of the HHS in clinical research, we systemically reviewed clinical trials of primary hip arthroplasty using the HHS as an outcome measure. We hypothesized that ceiling effects are common, thereby limiting the validity of the HHS in arthroplasty outcome research.

5Our systematic review conformed to the PRISMA guidelines for reporting of meta-analyses and systematic reviews.

6We performed a systematic review of the literature by performing a computerized literature search in PubMed, Embase, and the Cochrane Library for studies published in 2007, searching for “hip” and “arthroplasty” both as free text and as MESH terms.

7We included all descriptive trials, both prospective and retrospective, reporting on the outcome of primary total hip arthroplasty. Our inclusion criteria for further analysis were: articles published in English, patients undergoing primary total hip arthroplasty, and a range (standard deviation) of the HHS score reported in the article.

82 of the authors (DH and KW) independently selected titles and abstracts for possible inclusion. Full-text manuscripts were retrieved for any abstracts that appeared potentially eligible. The final decision to include a paper was based on a consensus between the 2 reviewers.

9Data extraction was performed independently by the same 2 authors (KW and DH), after which the data were compared and a consensus obtained.

10Harris hip scores were reported as averages with SD or range. If no SD was given in the study, it was calculated from the range to estimate the percentage ceiling effect. A ceiling effect means that several patients score the highest possible score, thus they “reach the ceiling”. A ceiling effect is caused when the test items are not challenging enough for a group of individuals because the test has a limited number of difficult items or even an inappropriate item selection. It will lead to a shortcoming in the discriminative ability of the test to detect clinically relevant changes; a person may continue to improve, but the test does not capture that improvement. A floor or ceiling effect of 15% is considered the maximum acceptable.

11As described by, it is possible to estimate the SD from a study when the mean and range and the size of the population is known. This method is widely used and accepted. We used it to estimate the SD for those studies where only the range and size (n) were given. In this data calculation, we assumed a normal distribution in the patient populations. This allowed us to estimate the percentage of ceiling effect present. This was calculated for each separate study, and data were also pooled to calculate a weighted overall percentage for all patients.

12We performed a subgroup analysis based on the indication for THA to assess whether the HHS would be suitable for any subgroup. Also, a subanalysis for the influence of age and length of follow-up on the content validity was performed. Finally, we conducted a subgroup analysis evaluating ceiling effect in patients who underwent hip resurfacing arthroplasties.

13We report a descriptive analysis of the ceiling effects, as percentage with 95% confidence interval (CI). For analysis, we used the HHS score from the latest follow-up reported. A linear logistic regression was performed to search for factors influencing the percentage ceiling effect (indication, retrospective or prospective trial, average age, length of follow-up period, or type of procedure (hip resurfacing, minimally invasive techniques, or normal THA). Statistical analyses were performed using SPSS software version 15.0.

14Of 764 potential studies, 54 studies—of which 59 groups could be reviewed (5 trials were comparative, describing 2 populations; altogether, 6,667 patients)—were suitable for inclusion. Figure shows the reasons for exclusion of certain studies. 45 studies reported primary THA for several indications, 13 of which focused on specific patient populations (4 osteonecrosis, 9 dysplasia). Hip resurfacing was reported in 4 studies. Details of each study included are given in the Table.

1531/59 patient groups showed a ceiling effect greater than 15%. Pooling across the 59 patient groups showed a ceiling effect of 20% (CI: 18–22).

16When studies needing a calculation of SD were excluded and only the 14 studies in which an SD was given were included, we found an average ceiling effect of 15.8% (4–30); 7 of the 14 studies had a ceiling effect of more than 15%.

17In the studies evaluating total hip arthroplasty in patients with avascular necrosis of the femoral head, the ceiling effect averaged 16% (CI: 8–24). Similarly, in patients with total hip arthroplasty for osteoarthritis secondary to dysplasia, the ceiling effect averaged 24% (CI: 18–31). In patients treated with a hip resurfacing, the mean ceiling effect was 32% (CI: 12–52).

18Indication, study design, patient age, length of follow-up, and type of procedure had no statistically significant influence on the magnitude of the ceiling effect (p-values all > 0.05).

19Our review shows that the Harris hip score has frequent ceiling effects in trials evaluating outcomes of primary hip arthroplasty, which indicates that it has limited application in exploration of treatment differences using newer techniques.

20Our meta-analysis has several strengths. We used a comprehensive systematic approach to identify relevant papers, we assessed the reliability of our assessments, and included a sufficient number of trials to be able to reach a conclusion. Finally, our systematic review followed the international PRISMA guidelines for reporting.

21Our review does have some limitations. We calculated ceiling effect based on the assumption that scores in every population had a normal distribution. It is possible that normality was not met due to insufficient sample size, or merely because of the existence of the ceiling effect we were trying to investigate. However, in half of studies in which the SD was given rather than calculated by us, a ceiling effect was found. We determined only the ceiling effects and not floor effects because we believed that ceiling effects are the main limitations of the Harris hip score. None of the studies that investigated the reliability included the floor score in the range of distribution, while most included the ceiling value of 100.

22In 1969, when the HHS was developed, it probably had excellent content validity due to the nature of the patient population and type of implant at that time. However, indications for joint replacement have expanded over time and improvements in implant designs and techniques have led to improved outcomes. The ability of a functional outcome measure to distinguish clinically relevant improvements in outcomes with changes in prosthetic design is important. Ceiling effects in an instrument can hide these differences when patients already score the maximum possible score and cannot improve on that score. For example, a 75-year-old patient just able to walk 2 hours at a normal pace would have the same score as a 45-year-old patient who has returned to running marathons.

23There are plenty of alternative scoring systems, including the WOMAC score, the Oxford 12-item questionnaire, and the HOOS. evaluated the WOMAC score and the Oxford 12-item questionnaire for validity. The Oxford hip score did well in their study on all validity items. evaluated the content validity of the Dutch version of the hip disability and osteoarthritis outcome score (HOOS) and reported no ceiling effect and good validity.

24Besides having a problematic ceiling effect, HHS includes a physician's physical examination component. Previous studies have shown that physical examination has a high intraobserver variablility. Consequently, investigators have commonly used a modified Harris hip score without the physical examination part. Thus, the modified HHS suffers the drawbacks of ceiling effects as well as the problems of a non-validated modification of the original HHS score.

25Based on our systematic review and meta-analysis, we conclude that the Harris hip score commonly shows ceiling effects, which limit its usefulness in trials evaluating the efficacy of primary total hip arthroplasty.