Diagnostic Accuracy of Point-of-Care Lung Ultrasonography and Chest Radiography in Adults With Symptoms Suggestive of Acute Decompensated Heart Failure

1Acute decompensated heart failure (ADHF) is the primary cause in up to 40% of older adults presenting with dyspnea, one of the leading reasons for emergency department visits in the United States. The diagnostic workup for ADHF can be challenging and often requires several tests. The insensitivity of guideline-recommended tools for diagnosing ADHF, such as chest radiography (CXR), physical examination, and brain-type natriuretic peptide (BNP),, is known to delay treatment, which is associated with an increase in mortality., In particular, the sensitivity of CXR in detecting pulmonary edema is limited, with 20% being a false-negative., Given the limitations of current tools to diagnose ADHF, the National Heart, Lung, and Blood Institute Working Group on Emergency Department Management of Acute Heart Failure has prioritized the development of new techniques for the diagnosis and monitoring of ADHF.

2Point-of-care lung ultrasonography (LUS), which is performed and interpreted at the bedside by the treating clinician, has emerged as a practical diagnostic tool for several lung pathologies. Growing evidence indicates that LUS has comparable or superior accuracy over CXR for many of the most common causes of dyspnea.,, Sonographic B-lines are hyperechoic reverberation artifacts that extend vertically from the pleural surface to the bottom of the screen and move synchronously with lung sliding. The number of B-lines seen on LUS has been shown to offer a semiquantitative measure of extravascular lung water content.,, However, data on the diagnostic accuracy of LUS for cardiogenic pulmonary edema are conflicting, with reported sensitivity ranging from 57% to higher than 95%., Given its potential advantages over CXR, including its ease of acquisition, immediate availability of results, and evidence of comparable accuracy, LUS could have important implications for standard of care in the evaluation of patients with dyspnea at risk for ADHF.

3Although previous systematic reviews have focused on the accuracy of various tools to diagnose ADHF in patients presenting with dyspnea,, none of them has directly compared the accuracy of LUS with CXR accuracy. The objective of this study was to perform a systematic review with meta-analysis to determine the comparative accuracy of LUS and CXR for the diagnosis of cardiogenic pulmonary edema in patients presenting with dyspnea.

4This systematic review was conducted in compliance with the recommendations from the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. This study followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline. The study protocol was registered on PROSPERO prior to study selection.

5A comprehensive literature search of MEDLINE, Embase, and Cochrane Library databases was performed in November 2017 and was updated in May 2018. A search of the gray literature was also performed through May 2018 and included conference proceedings from 2014 to 2018 of the American College of Chest Physicians (Annual Meeting Abstract supplements), American Heart Association (Scientific Sessions abstracts), and American College of Cardiology (Annual Scientific Sessions abstracts) as well as from ClinicalTrials.gov and ProQuest Dissertations and Theses databases.

6The search strategy (eTable 1 in the Supplement) was developed by our principal investigator (A.M.M.) in collaboration with a medical librarian (K.D.). No language or year limits were applied. Retrieved titles and abstracts were independently reviewed by 2 of us (G.S. and A.S.). Full-text versions of relevant studies were retrieved for further evaluation, and the inclusion criteria were applied independently by 2 of us (A.M.M. and A.H.). The inclusion criteria required studies to be prospective cohorts of adult patients presenting with acute dyspnea to any clinical setting in which both LUS and CXR were performed on initial assessment of all patients. In addition, a reference standard of an ADHF diagnosis had to be made by an independent expert after a medical record audit or a combination of echocardiographic findings and BNP criteria. Sufficient data to calculate both sensitivity and specificity were also required. Study participants could not be a subset of patients from another included paper (ie, no overlapping samples or duplicate patients).

7Two of us (A. M. and A. H.) independently extracted data by using standardized data-extraction sheets. When pertinent data were not included in a manuscript, the study’s corresponding author was contacted. If no reply was received or the authors did not have the requested information, the data were labeled as not specified.

8Data were extracted into 3 tables. Table 1 and Table 2 show relevant study characteristics. The numeric data necessary to create 2-by-2 contingency tables were extracted into a third table, from which sensitivity and specificity were then calculated. In the case of multiple individuals interpreting the LUS within the primary study, we determined the mean LUS results (ie, the number of B-lines) of all interpreters. In the case of an individual study reporting more than 1 threshold, our a priori plan was to use the threshold closest to those of all other included studies.

9A customized QUADAS-2 tool was applied to assess the risk of bias and applicability to the research question. Two of us (A.M.M. and A.H.) applied the tool independently to all studies. Disagreements between us were resolved by discussion. Interrater agreement of QUADAS-2 ratings were assessed using Cohen κ statistic and Gwet AC1.,

10Based on the QUADAS-2 tool, each article was evaluated for risk of bias in 4 domains. High risk of bias in each domain was defined as follows: (1) patient selection, the study enrollment was not consecutive or the study excluded patients that could introduce spectrum bias, which is the phenomenon that the performance of a diagnostic test will vary with the spectrum of severity of disease present in the cohort; (2) index test, the LUS or CXR results were interpreted without blinding to the reference standard or the other index test; (3) reference standard, the reference standard for ADHF diagnosis was interpreted without blinding to the results of the index tests (LUS or CXR); and (4) flow and timing, the interval between LUS and CXR exceeded 2 hours. eFigure 1 in the Supplement provides the QUADAS-2 tool used in this systematic review.

11With regard to assessment of applicability, each article was evaluated for low and high concern for applicability to the research question. Using the patient selection, index test, and reference standard domains, we defined low applicability concern as follows: (1) patient selection, the patient presented to the health care setting with acute dyspnea; (2) index test, the CXR was performed according to the hospital’s standard procedure; (3) index test, the LUS was acquired according to the international evidence-based recommendations for point-of-care LUS (Volpicelli criteria: 2 lung fields with 3 or more B-lines present bilaterally) or a modification of it; and (4) reference standard, the diagnosis was based on medical record audit by 1 or more attending physicians or by structural heart disease on echocardiography and on BNP greater than 100 ng/l (to convert to picogram/milliliter, multiply by 1.0) or N-terminal pro-BNP greater than 900 pg/mL.

12The a priori intention was to attempt a meta-analysis using the hierarchical summary receiver operating characteristic (HSROC) curve model., The HSROC model is a statistically rigorous approach for the meta-analysis of diagnostic test accuracy studies. It assumes that an underlying receiver operating characteristic curve exists within each study. Pooled sensitivity and specificity were calculated from the parameter estimates of the HSROC model, as were positive and negative likelihood ratios. Positive and negative predictive values were calculated from pooled estimates of sensitivity and specificity as well as from pooled prevalence of the included studies. Using test type (LUS vs CXR) as a covariate, we performed an overall likelihood ratio test to evaluate the overall differences in sensitivity and specificity between index tests. Individual tests were also performed for differences in sensitivity and specificity between index tests. Statistical tests were performed as 2-sided tests with a P =.05 level of significance. The χ2 and Wald tests were used to determine P values. Meta-analysis was performed using the SAS metadas macro, version 9.4 (SAS Institute Inc).

13Individual study results for sensitivity and specificity were plotted on a forest plot to visually assess and explore study variability. With regard to the explanation for the variability seen between studies, we identified a priori 2 possible sources of variability: (1) spectrum of disease, which is the range of ADHF disease severity as measured by the need for positive pressure ventilation, intensive care unit admission, or outpatient care, and (2) threshold effect, which is the criteria used to define a positive test result. We assessed for threshold effect using a visual inspection of summary receiver operating characteristic (SROC) curves and Spearman rank correlation for sensitivity and 1-specificity. Distribution of the study results closely along the estimated SROC curve suggests that differences in the threshold for a positive result among studies explain some of the variability observed. Publication bias was not assessed as no accepted method exists for its evaluation in a meta-analysis of diagnostic test accuracy studies.

14Figure 1 is a study flow diagram detailing search results and study inclusion. The search identified 1377 nonduplicate titles that underwent screening, of which 43 articles (3.1%) underwent full-text review. After application of the exclusion criteria, we identified 6 studies eligible for data extraction, representing a total of 1827 patients.,,,,, Figure 1 provides reasons for exclusion and the number of studies excluded under each reason.

15Table 1 summarizes the basic characteristics of the included studies. Four studies (67%) were conducted in emergency department cohorts and 2 studies (33%) in internal medicine ward patients who initially presented to the emergency department with dyspnea. Table 2 summarizes relevant index test and reference standard characteristics for included studies.

16The number of LUS operators per study ranged from 1 to at least 12 (some studies reported multiple sonographers but did not specify the exact number), with each patient assessed by a single sonographer in 5 (83%) of 6 studies. In Vitturi et al, the exception study, LUS was performed twice by 2 different sonographers on each patient to assess interoperator variability. Three studies (50%) reported the LUS interrater agreement as κ, which ranged from 0.70 to 1.00. In 4 studies (67%), the sonographers acquiring the images were blinded to all clinical information except that which was seen at the bedside (ie, physical appearance of the patient) and interpreted the ultrasonography images. In 2 studies (33%), an LUS expert blinded to all clinical information interpreted previously recorded images. The length of the LUS image clip ranged from a still image to 10 seconds, and the number of lung zones examined per patient ranged from 4 to 12. All studies used the Volpicelli criteria as the threshold for positive LUS or a modification of it.

17Chest radiographs were typically obtained using postero-anterior view and interpreted by a radiologist in all studies. Blinding of the radiologist to clinical information was unclear in all included studies. The interval time between LUS and CXR ranged from fewer than 1 to 12 hours.

18Five (83%) of 6 studies used expert medical record review as the reference standard. The remaining study, Öhman et al, used echocardiography, BNP, and CXR criteria as the reference standard. Figure 2 summarizes the estimate of sensitivity and specificity obtained from each study.,,,,, Vitturi et al did not offer a global estimate of CXR sensitivity and specificity but instead described test characteristics of particular CXR findings. In this case, we used the highest estimates for sensitivity and specificity of CXR that were reported in the study. The study authors reported no conflicts of interests.

19For LUS, estimates of sensitivity ranged from 58% to 97% and specificity ranged from 69% to 94% among included studies. For CXR, estimates ranged from 70% to 90% for sensitivity and 61% to 98% for specificity (Figure 2).

20Pooled estimates for LUS, calculated from the parameter estimates of the HSROC model, were 0.88 (95% Cl, 0.75-0.95) for sensitivity and 0.90 (95% Cl, 0.88-0.92) for specificity. In contrast, for CXR the pooled estimate for sensitivity was 0.73 (95% Cl, 0.70-0.76) and for specificity was 0.90 (95% Cl, 0.75-0.97).

21The overall and the individual tests performed on the HSROC model found the relative sensitivity ratio of LUS, compared with CXR, to be 1.2 (95% CI, 1.08-1.34; P <.001) but found no difference in specificity between tests (relative specificity ratio, 1.0; 95% CI, 0.90-1.11; P =.96). Table 3 shows additional test characteristics for CXR and LUS calculated from the HSROC model and from pooled ADHF prevalence in all included studies.

22The results of the risk of bias and applicability concern assessment of individual studies using the QUADAS-2 tool are shown in eTable 2 in the Supplement. Cohen κ was 0.36 (95% CI, 0.08-0.64) and Gwet AC1 was 0.61 (95% CI, 0.40-0.83), indicating fair to substantial interrater agreement beyond that of chance of QUADAS-2 assessments. These results are within the range of findings in other studies that used QUADAS-2 as the quality assessment tool. With regard to the patient selection domain, enrollment in 3 of the 6 trials (50%) was not consecutive but rather a convenience sample based on the availability of an LUS sonographer. Although it is possible that convenience sampling introduced an element of spectrum bias (ie, sicker patients may present at night rather than the day), it is unlikely to have greatly affected the relative performance of LUS to CXR. Given that all CXRs were interpreted by a radiologist, available expertise in CXR interpretation was unlikely to have contributed to a relative difference in accuracy between imaging modalities for the included studies with convenience samples.

23The reference standard domain was found to be at high risk of bias across all studies. Study adjudicators for 5 of the 6 included studies (83%) had access to CXR results during medical record audit, suggesting review bias. The 1 remaining study, Öhman et al, had CXR incorporated into the reference standard criteria, demonstrating incorporation bias. Thus, reference standard results were at high risk of bias, likely leading to overestimates in CXR accuracy across all studies.

24Both Perrone et al and Vitturi et al were found to be at high risk of bias in the flow and timing domain, with CXR performed several hours before LUS. Given that symptomatic patients may have received diuretics in the interval between index tests, we expected this intervention to underestimate LUS sensitivity compared with CXR.

25With regard to the applicability assessment, only 1 study met the criteria for high concern. Sartini et al included patients who did not match the review question for the patient selection domain, as 50 study participants (37%) were treated for ADHF prior to undergoing CXR and LUS. Therefore, the imaging study was not used in the initial decision to diagnose or treat ADHF. No other concerns were found regarding applicability to the research question in any other domain.

26Although not part of the QUADAS- 2 assessment, unclear or incomplete blinding of LUS and CXR interpreters to clinical data occurred. In 4 of the 6 included studies (67%), LUS was interpreted by the sonographer who obtained the images; although the sonographers were blinded to all other clinical information, they may have seen information at the bedside that could have affected their interpretation. Similarly, to what extent the radiologists were blinded to clinical data was unclear in all but 2 studies: Baker et al reported that radiologists routinely reviewed previous chest imaging, and Öhman et al reported blinding of radiologists to laboratory data and final diagnosis only.

27Visual inspection of the forest plot for LUS sensitivity revealed 2 potential outliers, Baker et al and Sartini et al, both of which reported LUS sensitivities lower than almost all of the other studies on this topic. Visual inspection of the SROC curves (eFigure 2 in the Supplement) demonstrated that the distribution of accuracy estimates found in each study followed the best-fit SROC curve, supporting the idea that differences in the threshold used to define a positive result between studies explain some of the variability observed between LUS results. Spearman rank correlation coefficient (ρ = 0.6) provided further evidence of the threshold effect contributing to the variability seen between studies for LUS but not for CXR. We were unable to evaluate variability based on spectrum of disease as no studies of outpatient or intensive care unit cohorts met the inclusion criteria.

28A possible outlier for CXR sensitivity was the high estimate in Öhman et al, which may be associated with incorporation bias: CXR was part of the reference standard criteria in that study.

29A possible outlier for CXR specificity was the low estimate in Perrone et al, which may be explained by 10% of patients having both pulmonary and cardiac processes associated with their abnormal imaging findings. Consistent with this premise, the LUS specificity estimate was lower in Perrone et al, compared with other included studies.

30Among adults presenting to a hospital setting for acute dyspnea, a 15% absolute difference in sensitivity was found between LUS and CXR (0.88 vs 0.73), favoring LUS, but no statistically significant difference in specificity was found for the detection of pulmonary edema from ADHF. Specifically, for every 100 patients presenting with dyspnea owing to cardiogenic pulmonary edema, LUS can diagnose 15 more cases than CXR without an increase in the number of false-positives. In addition, we identified threshold effect to be a likely contributor to the variability seen in LUS accuracy results. The estimates for sensitivity and specificity are in agreement with other systematic reviews that evaluated the accuracy of LUS in the diagnosis of cardiogenic pulmonary edema.,, To our knowledge, this is the first study that compared LUS accuracy with that of CXR by including only studies that performed both tests in all study participants, thereby minimizing the risk of bias and confounding owing to the differences in reference standards and study design. We chose this methodologic approach to prioritize internal validity over statistical power, knowing that it would result in fewer included studies. In spite of this tradeoff, the analysis of these data demonstrates that LUS, compared with CXR, has better sensitivity in the detection of pulmonary edema from ADHF and has comparable specificity.

31The high variability present between LUS sensitivity results appears to be driven by the lower estimates of LUS sensitivity found in Sartini et al and Baker et al. In these studies, LUS was acquired using a still image or a 3-second video clip. It has been shown that shorter clips can underestimate the number of B-lines seen., The short clip length used in both Sartini et al and Baker et al studies may explain their lower estimates of LUS sensitivity. In addition, in Sartini et al, 50 patients (37%) received diuretic therapy prior to presentation to the emergency department. In a subgroup analysis that excluded these patients, LUS sensitivity was found to be 83% and specificity was found to be 86%, whereas CXR was found to be 64% sensitive and 94% specific. Signs of pulmonary edema seen on CXR are known to lag in resolution. In contrast, LUS findings may be more responsive to dynamic changes in volume status.,, Therefore, the low sensitivity of LUS in the main analysis may have been associated with the resolution of LUS abnormalities from diuretics administered prior to LUS testing. The test characteristics in the post hoc subgroup analysis that excluded those with prehospital diuretics are more consistent with the available evidence on the accuracy of LUS,,

32In addition to improved accuracy, LUS may offer other benefits compared with CXR, including avoidance of ionizing radiation and immediate availability of results. Lung ultrasonography has also be shown to be easy for clinicians to learn, perform, and interpret.,,,, As presented in the clinical practice guidelines, the role of CXR in the evaluation of ADHF is, in part, to assess other causes of dyspnea. Because LUS has been shown to offer comparable or superior accuracy over many of the other most common causes of dyspnea,,,, it has the potential to become an initial imaging modality in the evaluation of patients with dyspnea. Future prospective studies are needed to determine if the use of LUS in the initial evaluation of adults presenting with dyspnea improves diagnosis, treatment, and outcomes of patients with ADHF.

33The main limitation of this systematic review with meta-analysis is the small number of included studies. Including only studies that evaluated the accuracy of both LUS and CXR to minimize bias contributed to the small number of included studies. This limited inclusion, in turn, decreased the precision of our estimates as well as our ability to formally evaluate for possible causes of variability, including clinical setting, using subgroup analysis, and meta-regression. However, on visual inspection of all forest plots (Figure 2), no clear evidence of a difference in test characteristic estimates was found between emergency department and inpatient cohorts for either imaging modality.

34Another important limitation is the risk of bias introduced by the lack of blinding of outcome adjudicators to CXR in 5 of the studies and by the incorporation of CXR as part of the reference standard in the remaining study. Thus, the direction of bias across all included studies likely favors CXR. In addition, these results are generalizable only to patients presenting to the hospital with acute dyspnea. The study is representative of a large proportion of the studies investigating LUS accuracy for cardiogenic pulmonary edema.

35The findings suggest that LUS is as specific and more sensitive than CXR in the identification of cardiogenic pulmonary edema. Given the potential advantages of its use, LUS should be considered as an adjunct imaging modality in the evaluation of patients with dyspnea at risk of ADHF.