Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning

Abstract

Remarkable progress has been made in image recognition, primarily due to the availability of large-scale annotated datasets and deep convolutional neural networks (CNNs). CNNs enable learning data-driven, highly representative, hierarchical image features from sufficient training data. However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pre-trained CNN features, and conducting unsupervised CNN pre-training with supervised fine-tuning. Another effective method is transfer learning, i.e., fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. In this paper, we exploit three important, but previously understudied factors of employing deep convolutional neural networks to computer-aided detection problems. We first explore and evaluate different CNN architectures. The studied models contain 5 thousand to 160 million parameters, and vary in numbers of layers. We then evaluate the influence of dataset scale and spatial image context on performance. Finally, we examine when and why transfer learning from pre-trained ImageNet (via fine-tuning) can be useful. We study two specific computer-aided detection (CADe) problems, namely thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. We achieve the state-of-the-art performance on the mediastinal LN detection, and report the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories. Our extensive empirical evaluation, CNN model analysis and valuable insights can be extended to the design of high performance CAD systems for other medical imaging tasks.

Shown for reading. Its publisher allows quoting it, not republishing it.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 73

1. 1.

   Deng J., “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 248–255.

   Not in our library

2. 2.

   Russakovsky O., “ImageNet large scale visual recognition challenge,” ArXiv:1409.0575, 2014.

   Not in our library

3. 3.

   LeCun Y., Bottou L., Bengio Y., and Haffner P., “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998.

   Not in our library

4. 4.

   Krizhevsky A., Sutskever I., and Hinton G. E., “ImageNet classification with deep convolutional neural networks,” Proc. NIPS, pp. 1097–1105, 2012.

   Not in our library

5. 5.

   Krizhevsky A., Learning multiple layers of features from tiny images, M.S. thesisDept. Comp. Sci. Univ. Toronto, Toronto, Canada: 2009.

   Not in our library

6. 6.

   Girshick R., Donahue J., Darrell T., and Malik J., “Region-based convolutional networks for accurate object detection and semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 1, pp. 142–158, Jan. 2016.

   Not in our library10.1109/tpami.2015.2437384

7. 7.

   He K., Zhang X., Ren S., and Sun J., “Spatial pyramid pooling in deep convolutional networks for visual recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1904–1916, Sep. 2015.

   Not in our library10.1109/tpami.2015.2389824

8. 8.

   Everingham M., “The pascal visual object classes challenge: A retrospective,” Int. J. Comput. Vis., vol. 111, no. 1, pp. 98–136, 2015.

   Not in our library

9. 9.

   van Ginneken B., Setio A., Jacobs C., and Ciompi F., “Off-the-shelf convolutional neural network features for pulmonary nodule detection in computed tomography scans,” Proc. IEEE 12th Int. Symp. Biomed. Imag., pp. 286–289, Apr. 2015.

   Not in our library

10. 10.

    Bar Y., Diamant I., Greenspan H., and Wolf L., “Chest pathology detection using deep learning with non-medical training,” Proc. IEEE 12th Int. Symp. Biomed. Imag., pp. 294–297, 2015.

    Not in our library

11. 11.

    Shin H., “Interleaved text/image deep mining on a large-scale radiology image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1090–1099.

    Not in our library

12. 12.

    Ciompi F., “Automatic classification of pulmonary peri-fissural nodules in computed tomography using an ensemble of 2d views and a convolutional neural network out-of-the-box,” Med. Image Anal., 2015.

    Not in our library10.1016/j.media.2015.08.001

13. 13.

    Menze B., Reyes M., and Van Leemput K., “The multimodal brain tumor image segmentation benchmark (BRATS),” IEEE Trans. Med. Imag., vol. 34, no. 10, pp. 1993–2024, Oct. 2015.

    Not in our library10.1109/tmi.2014.2377694

14. 14.

    Pan Y., “Brain tumor grading based on neural networks and convolutional neural networks,” Proc. 37th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., pp. 699–702, 2015.

    Not in our library10.1109/embc.2015.7318458

15. 15.

    Shen W., Zhou M., Yang F., Yang C., and Tian J., “Multi-scale convolutional neural networks for lung nodule classification,” in IPMI, 2015, pp. 588–599.

    Not in our library10.1007/978-3-319-19992-4_46

16. 16.

    Carneiro G., Nascimento J., and Bradley A. P., “Unregistered multiview mammogram analysis with pre-trained deep learning models,” Proc. MICCAI, pp. 652–660, 2015.

    Not in our library

17. 17.

    Wolterink J. M., Leiner T., Viergever M. A., and Išgum I., “Automatic coronary calcium scoring in cardiac CT angiography using convolutional neural networks,” Proc. MICCAI, pp. 589–596, 2015.

    Not in our library10.1016/j.media.2016.04.004

18. 18.

    Schlegl T., Ofner J., and Langs G. Unsupervised pre-training across image domains improves lung tissue classification Medical Comput. Vision: Algorithms for Big Data, New York: Springer, 2014, pp. 82–93.

    Not in our library

19. 19.

    Hofmanninger J. and Langs G., “Mapping visual features to semantic profiles for retrieval in medical imaging,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 457–465.

    Not in our library

20. 20.

    Carneiro G. and Nascimento J., “Combining multiple dynamic models and deep learning architectures for tracking the left ventricle endocardium in ultrasound data,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 11, pp. 2592–2607, Nov. 2013.

    Not in our library10.1109/tpami.2013.96

21. 21.

    Li R., “Deep learning based imaging data completion for improved brain disease diagnosis,” in Proc. MICCAI, 2014.

    Not in our library10.1007/978-3-319-10443-0_39

22. 22.

    Roth H., “Improving computer-aided detection using convolutional neural networks and random view aggregation,” IEEE Trans. Med. Imag..

    Not in our library10.1109/tmi.2015.2482920

23. 23.

    Barbu A., “Automatic detection and segmentation of lymph nodes from CT data,” IEEE Trans. Med. Imag., vol. 31, no. 2, pp. 240–250, Feb. 2012.

    Not in our library10.1109/tmi.2011.2168234

24. 24.

    Feulner J., Zhou S. K., Hammon M., Hornegger J., and Comaniciu D., “Lymph node detection and segmentation in chest CT data using discriminative learning and a spatial prior,” Med. Image Anal., vol. 17, no. 2, pp. 254–270, 2013.

    Not in our library10.1016/j.media.2012.11.001

25. 25.

    Feuerstein M., “Mediastinal atlas creation from 3-d chest computed tomography images: Application to automated detection and station mapping of lymph nodes,” Med. Image Anal., vol. 16, no. 1, pp. 63–74, 2012.

    Not in our library10.1016/j.media.2011.05.005

26. 26.

    Lu L., Devarakota P., Vikal S., Wu D., Zheng Y., and Wolf M. Computer aided diagnosis using multilevel image features on large-scale evaluation Medical Comput. Vision. Large Data in Medical Imaging, New York: Springer, 2014, pp. 161–174.

    Not in our library

27. 27.

    Lu L., Bi J., Wolf M., and Salganicoff M., “Effective 3d object detection and regression using probabilistic segmentation features in CT images,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 1049–1056.

    Not in our library

28. 28.

    Lu L., “Accurate polyp segmentation for 3d CT colonography using multi-staged probabilistic binary learning and compositional model,” in Proc. IEEE Comput. Vis. Pattern Recognit. Conf., 2008, pp. 1–8.

    Not in our library

29. 29.

    Tajbakhsh N., Gotway M. B., and Liang J., “Computer-aided pulmonary embolism detection using a novel vessel-aligned multi-planar image representation and convolutional neural networks,” in Proc. MICCAI, 2015.

    Not in our library

30. 30.

    Lowe D. G., “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.

    Not in our library

31. 31.

    Dalal N. and Triggs B., “Histograms of oriented gradients for human detection,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 1, pp. 886–893, 2005.

    Not in our library

32. 32.

    Torralba A., Fergus R., and Weiss Y., “Small codes and large image databases for recognition,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 1–8, 2008.

    Not in our library

33. 33.

    Szegedy C., “Going deeper with convolutions,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1–9.

    Not in our library

34. 34.

    Chatfield K., Simonyan K., Vedaldi A., and Zisserman A., “Return of the devil in the details: Delving deep into convolutional nets,” Br. Mach. Vis. Conf., 2014.

    Not in our library

35. 35.

    Chatfield K., Lempitsky V. S., Vedaldi A., and Zisserman A., “The devil is in the details: An evaluation of recent feature encoding methods,” Br. Mach. Vis. Conf., 2011.

    Not in our library

36. 36.

    Seff A., Lu L., Barbu A., Roth H., Shin H.-C., and Summers R. M., “Leveraging mid-level semantic boundary cues for computer-aided lymph node detection,” Proc. MICCAI, 2015.

    Not in our library

37. 37.

    Depeursinge A., “Building a reference multimedia database for interstitial lung diseases,” Comput. Med. Imaging and. Grap., vol. 36, no. 3, pp. 227–238, 2012.

    Not in our library10.1016/j.compmedimag.2011.07.003

38. 38.

    Song Y., Cai W., Zhou Y., and Feng D. D., “Feature-based image patch approximation for lung tissue classification,” IEEE Trans. Med. Imag., vol. 32, no. 4, pp. 797–808, Apr. 2013.

    Not in our library10.1109/tmi.2013.2241448

39. 39.

    Song Y., “Large margin local estimate with applications to medical image classification,” IEEE Trans. Med. Imag., vol. 34, no. 6, pp. 1362–1377, Jun. 2015.

    Not in our library10.1109/tmi.2015.2393954

40. 40.

    Gao M., “Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks,” in Proc. MICCAI First Workshop Deep Learn. Med. Image Anal., 2015.

    Not in our library10.1080/21681163.2015.1124249

41. 41.

    Seff A., “2d view aggregation for lymph node detection using a shallow hierarchy of linear classifiers,” Proc. MICCAI, pp. 544–552, 2014.

    Not in our library10.1007/978-3-319-10404-1_68

42. 42.

    Lu L., Liu M., Ye X., Yu S., and Huang H., “Coarse-to-fine classification via parametric and nonparametric models for computer-aided diagnosis,” in ACM Conf. CIKM, 2011, pp. 2509–2512.

    Not in our library

43. 43.

    Farabet C., Couprie C., Najman L., and LeCun Y., “Learning hierarchical features for scene labeling,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1915–1929, Aug. 2013.

    Not in our library10.1109/tpami.2012.231

44. 44.

    Mostajabi M., Yadollahpour P., and Shakhnarovich G., “Feedforward semantic segmentation with zoom-out features,” ArXiv Preprint ArXiv:1412.0774, 2014.

    Not in our library

45. 45.

    Roth H., “DeepOrgan: Multi-level deep convolutional networks for automated pancreas segmentation,” Proc. MICCAI, 2015.

    Not in our library

46. 46.

    Liang X., “Human parsing with contextualized convolutional neural network,” Proc. IEEE Int. Conf. Comput. Vis., pp. 1386–1394, 2015.

    Not in our library

47. 47.

    Gao M., Lu L., Nogues I., Summers M. R., and Mollura D., “Segmentation label propagation using deep convolutional neural networks and dense conditional random field,” Proc. IEEE Int. Conf. Comput. Vis., pp. 1377–1385, 2015.

    Not in our library

48. 48.

    Wang H., “Multi-atlas segmentation with joint label fusion,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 3, pp. 611–623, Mar. 2013.

    Not in our library10.1109/tpami.2012.143

49. 49.

    Oquab M., Bottou L., Laptev I., and Sivic J., “Is object localization for free?—Weakly-supervised learning with convolutional neural networks,” Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 685–694, 2015.

    Not in our library

50. 50.

    Oquab M., Bottou L., Laptev I., and Josef S., “Learning and transferring mid-level image representations using convolutional neural networks,” Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1717–1724, 2015.

    Not in our library

51. 51.

    Zhu X., Vondrick C., Ramanan D., and Fowlkes C., “Do we need more training data or better models for object detection?,” Br. Mach. Vis. Conf., 2012.

    Not in our library

52. 52.

    Ciresan D., Giusti A., Gambardella L., and Schmidhuber J., “Mitosis detection in breast cancer histology images with deep neural networks,” Proc. MICCAI, 2013.

    Not in our library10.1007/978-3-642-40763-5_51

53. 53.

    Zhang W., “Deep convolutional neural networks for multi-modality isointense infant brain image segmentation,” NeuroImage, vol. 108, pp. 214–224, 2015.

    Not in our library10.1016/j.neuroimage.2014.12.061

54. 54.

    Li Q., “Medical image classification with convolutional neural network,” Proc. 13th Int. Conf. Control Automat. Robot. Vis., pp. 844–848, 2014.

    Not in our library

55. 55.

    Miller G. A., “Wordnet: A lexical database for English,” Commun. ACM, vol. 38, no. 11, pp. 39–41, 1995.

    Not in our library

56. 56.

    Jia Y., “Caffe: Convolutional architecture for fast feature embedding,” ACM Multimedia, vol. 2, p. 4, 2014.

    Not in our library

57. 57.

    Razavian A. S., Azizpour H., Sullivan J., and Carlsson S., “CNN features off-the-shelf: An astounding baseline for recognition,” Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, pp. 512–519, 2014.

    Not in our library

58. 58.

    Zhou B., Lapedriza A., Xiao J., Torralba A., and Oliva A., “Learning deep features for scene recognition using places database,” Proc. NIPS, pp. 487–495, 2014.

    Not in our library

59. 59.

    Gupta S., Girshick R., Arbeláez P., and Malik J., “Learning rich features from RGB-D images for object detection and segmentation,” Proc. Eur. Conf. Comput. Vis., pp. 345–360, 2014.

    Not in our library

60. 60.

    Gupta S., Arbeláez P., Girshick R., and Malik J., “Indoor scene understanding with RGB-D images: Bottom-up segmentation, object detection and semantic segmentation,” Int. J. Comput. Vis., vol. 112, no. 2, pp. 133–149, 2015.

    Not in our library

61. 61.

    Gupta A., Ayhan M., and Maida A., “Natural image bases to represent neuroimaging data,” Proc. ICML, pp. 987–994, 2013.

    Not in our library

62. 62.

    Chen H., “Automatic fetal ultrasound standard plane detection using knowledge transferred recurrent neural networks,” Proc. MICCAI, pp. 507–514, 2015.

    Not in our library

63. 63.

    Kim L., “Performance assessment of retroperitoneal lymph node computer-assisted detection using random forest and deep convolutional neural network learning algorithms in tandem,” in 102nd Annu. Meet. Radiol. Soc. N. Am., 2014.

    Not in our library

64. 64.

    Holmes III D., Bartholmai B., Karwoski R., Zavaletta V., and Robb R., “The lung tissue research consortium: An extensive open database containing histological, clinical, and radiological data to study chronic lung disease,” in MICCAI Open Sci. Workshop, 2006.

    Not in our library

65. 65.

    Long J., Shelhamer E., and Darrell T., “Fully convolutional networks for semantic segmentation,” Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 3431–3440, 2015.

    Not in our library10.1109/tpami.2016.2572683

66. 66.

    Chen L.-C., Papandreou G., Kokkinos I., Murphy K., and Yuille A. L., “Semantic image segmentation with deep convolutional nets and fully connected CRFs,” Proc. ICLR, 2015.

    Not in our library10.1109/tpami.2017.2699184

67. 67.

    Sermanet P., “Overfeat: Integrated recognition, localization and detection using convolutional networks,” Proc. ICLR, 2014.

    Not in our library

68. 68.

    Simonyan K. and Zisserman A., “Very deep convolutional networks for large-scale image recognition,” Proc. ICLR, 2014.

    Not in our library

69. 69.

    Hochreiter S., “The vanishing gradient problem during learning recurrent neural nets and problem solutions,” Int. J. Uncertainty, Fuzziness Knowl.-Based Syst., vol. 6, no. 02, pp. 107–116, 1998.

    Not in our library

70. 70.

    Hinton G. E., Osindero S., and Teh Y.-W., “A fast learning algorithm for deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554, 2006.

    Not in our library10.1162/neco.2006.18.7.1527

71. 71.

    Bengio Y., Simard P., and Frasconi P., “Learning long-term dependencies with gradient descent is difficult,” IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 157–166, Mar. 1994.

    Not in our library10.1109/72.279181

72. 72.

    Zeiler M. D. and Fergus R., “Visualizing and understanding con volutional networks,” Proc. Eur. Conf. Comput. Vis., pp. 818–833, 2014.

    Not in our library

73. 73.

    Agrawal P., Girshick R., and Malik J., “Analyzing the performance of multilayer neural networks for object recognition,” Proc. Eur. Conf. Comput. Vis., 2014.

    Not in our library

Cited by papers we hold 46

• Deep-Learning Integration of CNN–Transformer and U-Net for Bi-Temporal SAR Flash-Flood Detection · 2025
  Not checked
• Hybrid Attention-Enhanced Xception and Dynamic Chaotic Whale Optimization for Brain Tumor Diagnosis · 2025
  Not checked
• Flow Cytometry: The Next Revolution · 2023
  No notice
• Deep learning in veterinary medicine, an approach based on CNN to detect pulmonary abnormalities from lateral thoracic radiographs in cats · 2022
  Not checked
• Siamese Network-Based All-Purpose-Tracker, a Model-Free Deep Learning Tool for Animal Behavioral Tracking · 2022
  Not checked
• Investigating the power of eyes open resting state EEG for assisting in dementia diagnosis · 2022
  Not checked
• Deep Learning-Based Detection and Identification Method for Sports Health Video Dissemination · 2022
  Not checked
• Emerging Applications of Deep Learning in Bone Tumors: Current Advances and Challenges · 2022
  Not checked
• Data-Driven Framework for Understanding and Predicting Air Quality in Urban Areas · 2022
  Not checked
• Transfer learning for medical image classification: a literature review · 2022
  No notice
• A Review on Deep Learning Techniques for the Diagnosis of Novel Coronavirus (COVID-19) · 2021
  No notice
• Adaptive Local Ternary Pattern on Parameter Optimized-Faster Region Convolutional Neural Network for Pulmonary Emphysema Diagnosis · 2021
  Not checked
• Predicting Tyrosine Kinase Inhibitor Treatment Response in Stage IV Lung Adenocarcinoma Patients With EGFR Mutation Using Model-Based Deep Transfer Learning · 2021
  Not checked
• Generative adversarial network image synthesis method for skin lesion generation and classification · 2021
  Not checked
• Deep learning-based classification of blue light cystoscopy imaging during transurethral resection of bladder tumors · 2021
  Not checked
• Artificial Intelligence in Colorectal Cancer Screening, Diagnosis and Treatment. A New Era · 2021
  No notice
• Mapping single-cell data to reference atlases by transfer learning · 2021
  No notice
• Multimodal deep learning models for early detection of Alzheimer’s disease stage · 2021
  No notice
• A disease category feature database construction method of brain image based on deep convolutional neural network. · 2020
  Not checked
• LC-DNN: Local Connection Based Deep Neural Network for Indoor Localization With CSI · 2020
  Not checked

and 26 more

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.