Development and Initial Validation of a Self-Scored COPD Population Screener Questionnaire (COPD-PS)

1Chronic obstructive pulmonary disease (COPD) is a common disease with a profound impact on a patient's functioning and is underdiagnosed. A major objective of the Global Initiative for Chronic Obstructive Lung Disease (GOLD) is to increase awareness among healthcare providers and the general public of the significance of COPD symptoms. Despite the existence of evidence-based diagnostic and treatment guidelines, many patients with COPD continue to be undiagnosed or mis-diagnosed. COPD may remain undiagnosed for several reasons. Patients may not visit a physician because they have become accustomed to symptoms, may be concerned that their respiratory symptoms are self-inflicted, or are unsure whether COPD can be treated. Additionally, spirometry is underutilized and may be difficult to interpret by some healthcare providers. Furthermore, asthma and COPD may be confused as they can both exhibit fixed airflow obstruction (AO).

2COPD screening that focuses only on patients meeting a limited set of characteristics, such as older smokers, may fail to detect COPD among individuals in the general population (e.g., younger adults with early disease) or those having another etiology (e.g., occupational exposure). Similarly, identifying only individuals who are current smokers may miss those with COPD symptoms who have already quit smoking.

3Hence, there is a need for a simple method to help identify persons who might have COPD. A simple, self-administered, self-scored tool to screen individuals for the disease may lead to increased awareness, earlier symptom recognition, and the use of spirometry for accurate diagnosis. Dissemination of such a tool in the general population might encourage individuals to discuss their respiratory symptoms with a healthcare provider. This tool could also assist physicians in identifying those individuals who would need spirometric assessment. Although several investigators have developed surveys for describing the impact of COPD, measuring the outcomes of treatment, or assessing disease severity, there are few tools designed to identify patients with COPD in a nonclinical setting.

4We developed a simple, self-administered screening tool for the identification of patients with possible COPD in the general population. The COPD Population Screener™(COPD-PS™; QualityMetric Incorporated, Lincoln, Rhode Island, USA) is a 5-item questionnaire, designed to meet the following criteria: (1) a high correct classification rate for AO diagnosis while retaining a good trade-off between sensitivity and specificity; (2) self-scored, brief, and simple to complete; and (3) demonstrated adequate levels of reliability and preliminary evidence for validity. Of particular interest was the development of a tool appropriate for the general population rather than patients already seeking treatment for respiratory difficulties.

5A clinician working group comprised of 5 pulmonary specialists, 4 primary care physicians, and 1 respiratory therapy professor was assembled to assist in the development of the COPD-PS questionnaire. The working group guided development of specific survey-question content, participated in designing a clinical development and validation study, and assisted in the evaluation of the study results.

6Based on clinical experience, this working group first identified 7 conceptual domains relevant to the detection of COPD and easily identified by patients. A development survey included items that measured the conceptual domains in terms of presence, frequency, duration, or quality. An abbreviated description of these items is contained in Table 1, while a complete list of the items is enumerated in the Appendix. Additionally, a 12-item, short-form, general health-status assessment (SF-12®v2 Health Survey™ [QualityMetric Incorporated, Lincoln, Rhode Island, USA]), demographic items, questions about resource use, and a chronic condition checklist also were employed.

7To identify an optimal set of questions for inclusion in the COPD-PS questionnaire, a national cross-sectional survey of patients receiving care from 4 pulmonary specialists and 8 general practice sites was conducted during an 8-week period in 2004. Selected clinical sites were required to have experience performing spirometry following American Thoracic Society (ATS) standards. Approval of the study protocol was obtained by each site using a central Institutional Review Board (Chesapeake Research Review, Inc.), and all patients provided written consent.

8Patients aged 35 years and older with a previously scheduled office visit were recruited for study participation. Patients were excluded from the study for the following reasons: being allergic to bronchodilators, having a condition contraindicating study participation, participating in another clinical study, or currently seeking care for an acute respiratory problem. Based on a priori power calculations for logistic regression and sensitivity/specificity analyses, approximately 100 patients with COPD were sought.

9Patients completed the development survey and physicians recorded patient demographics and medical history. Spirometry was performed before and 10 minutes after albuterol administration via metered-dose inhaler (90 μg, 2 puffs), according to ATS standards current at the time of the study using predicted values from Morris et al.. For the purposes of this study, AO was defined based on GOLD guidelines (postbronchodilator forced expiratory volume in 1 second/forced vital capacity [FEV1/FVC] < 70%) as airway obstruction that is not fully reversible. Patients with AO, identified by spirometry, were further characterized as having mild AO or moderate-to-very-severe AO.

10Spirometry tracings were reviewed post hoc by an independent pulmonary specialist auditor to evaluate adherence to ATS standards for acceptability and reproducibility. Patients with spirometry tracings meeting the strict ATS criteria were included in the analysis. A random subsample of patients (20%) completed the development survey (but not spirometry) again 2 weeks later to generate test-retest reliability estimates; survey methods to optimize return rates were implemented.

11Analyses were performed using SAS (Version 8.2 for Windows, SAS Institute Inc., Cary, NC, USA) and STATA (version 8.0; StataCorp LP, College Station, TX, USA).

12Based on spirometry results, patients were categorized as “No AO” and “AO.” Point-biserial correlations of survey items with dichotomous AO diagnosis were generated. Forward stepwise logistic regression models were used to test blocks of items with similar content to identify the best predictors for AO status within each unidimensional set. Items within content blocks that met the criterion for significant model contribution (p < 0.10) were retained for inclusion in a second stage of modeling to identify the best multidimensional set of items (this significance level was selected to ensure items with any possible contribution to prediction were identified for further study). Items were evaluated as both continuous variables and as sets of dichotomized response options to study the contribution of the total item and specific response options.

13Age and smoking variables (including pack-years, categorical smoking status, and other variables regarding smoking experience) were also tested to identify optimal questions. Based on overall model fit and significant contribution of individual items, a final multivariate logistic model was constructed. Consistency of these results was examined across specialist and general practice sites. The final model was applied to the total study sample (N = 697) to study consistency across different sample definitions, although the accuracy of the AO outcome in the full sample cannot be confirmed. To estimate the robustness of final model results, 1000 bootstrap samples were drawn with replacement from the original audited sample and analyzed.

14Three scoring options for combining item responses in the final model were evaluated: 1) total sum of item responses, 2) dichotomous variable for each item, and 3) weighted sum of item responses. Receiver operating characteristic (ROC) analyses were conducted to evaluate the COPD-PS score in screening for AO. The area under an ROC curve (AUC), odds ratios, sensitivity and specificity, positive and negative predictive values, and percent correctly classified were estimated for the continuous screener score and at each scoring level or “cut-point.”

15Test-retest reliability for the COPD-PS score was assessed with product-moment and intraclass correlations between scores at study entry and 2-week follow-up.

16Clinician-working-group input assured a degree of content validity by defining conceptual domains and specific content items based on clinical experience. COPD is associated with decrements in quality of life; hence, convergent and discriminant validity was assessed by correlating COPD-PS scores and results from the SF-12v2 Health Survey. A series of known-groups construct validation analyses were conducted to examine the COPD-PS mean score differences between groups differing in the construct being measured. We hypothesized that patients with more respiratory problems would score significantly higher on the COPD-PS than those without for the categories of AO severity groups, site type, physician-reported COPD status, self-reported COPD status, work/school absenteeism due to breathing problems, self-reported use of bronchodilators, and overnight hospitalization for breathing problems. The ROC analyses for COPD-PS and CAO diagnosis were completed to examine concurrent validity of the COPD-PS screener.

17Of the 697 patients completing both the development survey and spirometry, 445 (64%) and 252 (36%) were from general practice and specialist sites, respectively. A total of 295 patients who had spirometry results meeting strict ATS standards were included in the analytic sample (105 [36%] from general practice; 190 [64%] from specialist sites). Table 2 presents patient characteristics by provider type. The average patient age for the total sample was 62.1 ± 13.0 years (range 35–91 years). Approximately 40% of the sample was male. Patients enrolled from specialist sites were older and more likely to be male, white, and less likely to be Hispanic.

18Approximately 38% of patients had AO based on spirometry results, severity being mostly moderate-to-very severe (85%). Prevalence of AO differed by site type—17% and 50% of patients enrolled from general practice sites and specialist sites had AO, respectively (p < 0.0001). The mean pre- and postbronchodilator FEV1% predicted for patients with AO and without AO is shown in Table 3. In the AO group, the postbronchodilator mean FEV1% predicted was only slightly higher than the prebronchodilator mean FEV1% predicted, suggesting these patients had poorly reversible airway obstruction.

19Responses to potential items for the COPD-PS questionnaire met basic criteria for Likert-type questions. Item responses were well distributed, and most items had a standard deviation (SD) of at least 1.0. Item-level missing data were less than 3% for all items, except shortness of breath with activity (8% missing). Correlations of the survey items with AO diagnosis ranged from 0.00 to 0.38. In the first stage of logistic models, 23 items were significant (p < 0.10) in predicting AO status.

20A series of stepwise logistic regression analyses with the 23 items revealed that a combination of two symptom- and one functioning-based items best discriminated between patients with and without AO: shortness of breath frequency, production of sputum/phlegm frequency, and functional limitations due to breathing problems. Also the interaction of dyspnea and cough items was significantly predictive of AO. The same three variables performed the best in regression models with the total sample as well. Two items were significantly associated with AO,but in the negative direction. Awake at night with breathing problems and need to clear chest in the morning predicted a lack of AO but added little to the overall model fit. As such, only items with a positive association with AO were retained for the COPD-PS questionnaire.

21Patient characteristics were added to the logistic model; age and smoking history improved AO prediction. Three versions of smoking variables were assessed: pack-years, categorical smoking status (i.e., current, former, never-smoker) and a single item regarding lifetime consumption of at least 100 cigarettes. The simple, single-item measure presented the best trade-off between explanatory power and ease of administration. Secondhand smoke exposure was evaluated regardless of individual smoking status and was nearly a significant predictor (p < 0.08), but did not improve overall model fit. The final logistic model with 5 variables produced significant overall fit (likelihood ratio χ2 = 122.13, p < 0.0001; pseudo R2 = 0.47).

22Table 4 presents results for the final model. The results in the final analytic sample, confirmed to have met ATS standards for spirometry performance were compared with the full study sample results (likelihood ratio χ2 = 113.50, p < 0.0001) and were found to be consistent. Interactions of provider type with items and item-response weights were not statistically significant, indicating that items were not differentially predictive across provider type, and scoring weights functioned equally for general practice and specialist sites. The mean, median, and 5th and 95th percentiles for the 1000 bootstrap sample regression model results revealed little variation in model coefficients from the analytic sample results.

23Of the 3 techniques used to evaluate scores, the weighted-sum technique produced the best results for discriminating between patients with and without AO. Each item response was assigned a value of 0, 1, or 2, depending on the relative contribution of the response to identifying AO, and response values were summed across the items to produce a scale score ranging from 0 (unlikely to have AO) to 10 (likely to have AO). Table 5 presents COPDPS mean scores by selected patient characteristics. Figure 1 presents the ROC curve associated with the final COPD-PS score. The AUC corresponding to a model with 5 individual screener items was 0.86, and for the total score it was 0.81. Based on sensitivity (59.6%) and specificity (83.2%) for the total score, the positive likelihood ratio (LR+) for the survey was 3.56.

24Table 6 summarizes the performance of the COPD-PS score in identifying patients with AO across multiple score levels. In these analyses, screener scores were dichotomized to allow an analysis of cut-point performance relative to AO diagnosis. Lower cut-point scores were associated with higher sensitivity and lower specificity, while higher scores produced lower sensitivity and higher specificity. A cut-point in the range of 5 to 6 provided a good trade-off between sensitivity and specificity, as well as high correct classification rates for AO diagnosis.

25Table 7 presents mean scores for the analytic sample and by AO diagnosis for a random subsample of 57 patients completing the study surveys at entry and 2-week follow-up. For the analytic sample, the test-retest Pearson's product-moment correlation was 0.91 and the intraclass estimate was 0.91. Test-retest reliability estimates were slightly lower within each diagnostic group (AO and no AO), but the sample sizes may have been too small for accurate estimation.

26Convergent and discriminant validity results are presented in Table 8, where scales are arranged in order from those designed to assess more physical aspects of health to those measuring mental health (higher SF-12v2 scores represent better health while higher COPD-PS scores represent higher likelihood of having AO). COPD-PS scores had higher correlations with scales designed to measure physical functioning, role limitations due to physical health, and general health status. Lower correlations were found between COPD-PS scores and scales designed to assess well-being domains, such as mental health.

27Table 9 presents results of known-groups validity analyses. COPD-PS mean scores were significantly higher for patients with AO than without (6.8 vs 4.0, p < 0.0001). Scores for patients with mild AO were significantly higher than for patients with no AO (p 7lt; 0.01), but the difference between AO severity groups was not statistically significant. Patient- and physicianreported COPD also produced significantly higher mean scores. Bronchodilator use and overnight hospitalization were associated with significantly higher COPD-PS scores. Results from concurrent validity analyses demonstrated that the COPD-PS score predicted physician- reportedCOPDas well as spirometrybased diagnosis (model likelihood ratio χ2 = 148.57, p < 0.0001). AUC for the analytic sample was 0.88, for the continuous score sensitivity was 66% and specificity was 89%. Eighty percent of patients were correctly classified.

28Simple tools are needed to help identify persons who have COPD. A brief, easy-to-complete questionnaire based on patient-reported information can serve as a first-level screen. A screening assessment may help clinicians identify patients at risk for COPD, prompting clinical review and spirometric assessment, which are necessary to confirm a diagnosis. Furthermore, dissemination of a COPD screener in the general population might encourage individuals with pulmonary symptoms to visit their physician. This study represents the development and initial validation of a simple, reliable, self-scored COPD screening questionnaire. Three COPD-related items (breathlessness, productive cough, and activity limitation), a smoking history item (100 or more cigarettes smoked in lifetime), and an age item were used to construct the COPD-PS questionnaire. A weighted-sum score of these items resulted in a questionnaire that discriminated between patients with and without AO, and could be easily adapted to an electronic format for simple scoring.

29The hallmark symptoms of COPD are dyspnea, chronic cough, chronic sputum production, and a history of exposure to risk factors, such as smoking. The COPD-PS identifies these symptoms and risks, as well as considers age as a screening factor for COPD. The COPD-PS is unique in that it is a self-administered, self-scored questionnaire that utilizes a scoring system to predict spirometry-based diagnosis of the disease. Individual COPD-PS items were not all significant predictors of COPD in a multivariate logistic model, but their combination produced a well-fitting model and a final score that performed well in tests of sensitivity and specificity. Scores met generally accepted standards for individual-level reliability, suggesting that they are not only appropriate for group-level inferences but also can be applied reliably at the individual level.

30The clinician working group tested several different versions of items that assessed concepts typically associated with COPD. This allowed identification of questions that could be easily understood and completed by patients across a range of literacy skills. Some particular item wordings had a stronger relationship to COPD than others. Surprisingly, several concepts identified by clinicians as related to COPD—cough, frequent colds or bronchitis, and chest congestion or wheezing—did not serve to distinguish between patients with AO and those without AO. Although these are likely important aspects of the disease, self-reporting of cough, colds, and chest congestion did not improve the differentiation between AO and no AO when breathlessness, phlegm, functional limitations, smoking status, and age were considered. This may relate to the similarity shared by several pulmonary conditions for these nondistinguishing symptoms.

31The group studied the contribution of smoking status, measured in multiple ways, to prediction of COPD. Each smoking variable performed similarly in discriminating between patients with and without AO when tested in combination with other screening items. In our attempt to develop an easy-to-complete questionnaire that did not require complicated calculations, the combination of explanatory power and ease of administration resulted in the selection of a single-item smoking measure for the COPD-PS. Other state and national surveys have used a similar “smoked at least 100 cigarettes” item. Its inclusion in the COPD-PS allows for comparison of smoking rates with nationally published figures.

32We demonstrated validity of the COPD-PS through multiple concurrent and construct approaches. Higher COPD-PS scores can generally be interpreted as indicating an increased likelihood of AO. Lower cut-point levels were associated with greater detection of patients with AO (higher sensitivity) but also included some patients not having AO (low specificity). The opposite was true at higher scores. Thus, selection of a particular cut-point value can be adapted for a particular application. For example, if the goal is to identify as many individuals with potential COPD as possible, with less concern for including those who may not have the disease, a score corresponding to high sensitivity and lower specificity can be considered. This could result in significant health care resource expenditure. On the other hand, if an application requires a higher level of certainty that all identified individuals do have COPD, a score associated with higher specificity is desirable. As such, the optimal scoring utilized should be adapted to the situation and with regards to the health care utilization implication of this decision.

33In contrast to this self-report questionnaire, some earlier COPD screening tools require clinical information from the medical record or previous report of a physician diagnosis of COPD for scoring. Others require interviewer administration, include complicated calculations or skip patterns that make it difficult to self-administer, or are intended only for a particular population, such as current or former smokers. The value of self-report surveys relative to spirometry has been demonstrated for COPD case finding. van Schayck et al. identified an optimal combination of items from NHANES-III for distinguishing between those with and without COPD, but because that study did not include reversibility testing, it is unclear how well results will generalize to COPD diagnoses based on GOLD definitions of the disease.

34Price et al. generated an 8-item COPD questionnaire developed in patients with a positive smoking history and no previous respiratory diagnosis identified through 2 primary care physician office sites (United Kingdom and United States). This questionnaire included items related to age group, body mass index, pack-year history, and symptoms and featured a scoring system suitable for use in primary care settings. Scores on this case-finding tool were separated into 3 “zones,” indicating increased, unchanged, and decreased likelihood of COPD compared with the total study population (which as a whole has an increased risk of COPD due to inclusion criteria); patients with a higher likelihood should be referred for spirometry. Although some of the items are similar to those of the COPD-PS, its derivation suggests that this instrument is better suited for use in a physician's office in higher risk populations.

35Similarly, Freeman et al. identified 4 features (age, cough, dyspnea, and wheezing) that identified patients with COPD among a primary care population with a positive smoking history, history of use of respiratory medications, or a history of asthma. Calverley et al. developed a population-based screening questionnaire for COPD retrospectively, using NHANES III data. The COPD-PS differs from these other instruments because it can be used in a broader group of individuals, regardless of smoking history or known presence of respiratory problems. Although other instruments include age and symptom-based items, the COPD-PS also contains a disease-impact item, which allows the patient to describe activity limitations due to breathing problems.

36A limitation of this study includes the definition for COPD. Although the screener identified patients with AO and not necessarily COPD, it does identify patients for whom spirometry would be indicated and in whom additional clinical evaluation would be required to confirm a COPD diagnosis. Furthermore, the important contribution of the smoking item, the limited bronchoreversibility of patients with AO, and the high correlation with physician-reported COPD strongly support that cases with AO identified had a COPD diagnosis. In fact there was only fair concordance between subject reported or clinician diagnosed COPD and confirmed COPD; one in five spirometry-identified COPD patients did not report having a previous COPD history by either self-report or clinician diagnosis.

37An additional limitation reflects the source of the analytic sample studied. Patients were recruited from a clinical setting to allow postbronchodilator spirometric assessment. In addition to patients from specialist sites, patients from general practice sites were enrolled, with the intention of accessing individuals who were not already seeing a pulmonary specialist for respiratory symptoms. Similarly, recruited patients were being seen for routine, previously scheduled, office visits to maximize generalizability. It is important to realize that the prevalence of COPD was higher in the cohort studied than would be anticipated in general population samples, which may have increased the positive predictive value of the COPD-PS. Validation in an independent general population sample is required to define the operating characteristics in this setting.

38Patients were visiting their physician for other medical problems, which might confound the diagnosis of COPD. An additional limitation was the low rate of adherence to ATS standards for spirometry testing. More than half of the tests carried out in general practice sites did not meet either strict reproducibility or acceptability standards for 3 tracings, according to external review. However, even in 2 of the 4 specialist sites, there were many tracings that were excluded.

39Despite this, results of analysis in the analytic sample were similar to the results for the entire study population. Importantly, the lack of adequate spirometry performance in a majority of subjects supports the need for aggressive training in spirometric testing if a screening instrument such as the COPDPS is to be widely employed. Previous investigators have confirmed that a majority of maneuvers performed in the primary care setting without access to training failed to meet reproducibility criteria. This group and others have confirmed improvement in spirometry quality with limited training. Using primary care physicians to identify early COPD with spirometry will require ensuring that spirometry is properly performed.

40Validation studies are needed to confirm the performance of the COPD-PS questionnaire among individuals in community settings. Future studies may choose to evaluate whether additional item content can improve the performance of a brief COPD screener. Cognitive interviews or focus groups with patients or individuals from the general population may provide additional information about the acceptability and readability of the COPD-PSquestions. Although a secondhand smoke item approached statistical significance in our analytic sample, future studies evaluating this item in nonsmokers may be of interest. The developmental survey did not include self-report items for calculating body mass index. Items that separate asthma and COPD could also be evaluated. The COPD-PS can be evaluated for its utility as a first-stage screener in various settings, such as population screening, clinical practice, disease management, or population-based research.