Deep learning-enabled virtual histological staining of biological samples

Abstract

Histological staining is the gold standard for tissue examination in clinical pathology and life-science research, which visualizes the tissue and cellular structures using chromatic dyes or fluorescence labels to aid the microscopic assessment of tissue. However, the current histological staining workflow requires tedious sample preparation steps, specialized laboratory infrastructure, and trained histotechnologists, making it expensive, time-consuming, and not accessible in resource-limited settings. Deep learning techniques created new opportunities to revolutionize staining methods by digitally generating histological stains using trained neural networks, providing rapid, cost-effective, and accurate alternatives to standard chemical staining methods. These techniques, broadly referred to as virtual staining , were extensively explored by multiple research groups and demonstrated to be successful in generating various types of histological stains from label-free microscopic images of unstained samples; similar approaches were also used for transforming images of an already stained tissue sample into another type of stain, performing virtual stain-to-stain transformations. In this Review, we provide a comprehensive overview of the recent research advances in deep learning-enabled virtual histological staining techniques. The basic concepts and the typical workflow of virtual staining are introduced, followed by a discussion of representative works and their technical innovations. We also share our perspectives on the future of this emerging field, aiming to inspire readers from diverse scientific fields to further expand the scope of deep learning-enabled virtual histological staining techniques and their applications.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 106

1. 1.

   Bancroft, J. D. & Gamble, M. Theory and Practice of Histological Techniques. 6th edn. (Churchill Livingstone, Edinburgh, 2008).

   Not in our library

2. 2.

   Musumeci G. Past, present and future: overview on histology and histopathology. J. Histol. Histopathol. 2014;1:5.

   Not in our library

3. 3.

   Titford M. Progress in the development of microscopical techniques for diagnostic pathology. J. Histotechnol. 2009;32:9-19.

   Not in our library

4. 4.
   Histological Stains: A Literature Review and Case Study · 2015

   No notice
5. 5.

   Gurcan MN, et al. Histopathological image analysis: a review. IEEE Rev. Biomed. Eng. 2009;2:147-171.

   Not in our library10.1109/rbme.2009.2034865

6. 6.

   Titford M. The long history of hematoxylin. Biotech. Histochem. 2005;80:73-78.

   Not in our library10.1080/10520290500138372

7. 7.

   Foot NC. The Masson trichrome staining methods in routine laboratory use. Stain Technol. 1933;8:101-110.

   Not in our library

8. 8.

   Pollitt CC. Basement membrane pathology: a feature of acute equine laminitis. Equine Vet. J. 1996;28:38-46.

   Not in our library10.1111/j.2042-3306.1996.tb01588.x

9. 9.

   Coons AH, Creech HJ, Jones RN. Immunological properties of an antibody containing a fluorescent group. Exp. Biol. Med. 1941;47:200-202.

   Not in our library

10. 10.

    Ramos-Vara JA, Miller MA. When tissue antigens and antibodies get along: revisiting the technical aspects of immunohistochemistry—the red, brown, and blue technique. Vet. Pathol. 2014;51:42-87.

    Not in our library10.1177/0300985813505879

11. 11.

    Al-Janabi S, Huisman A, Van Diest PJ. Digital pathology: current status and future perspectives. Histopathology. 2012;61:1-9.

    Not in our library10.1111/j.1365-2559.2011.03814.x

12. 12.

    Madabhushi A, Lee G. Image analysis and machine learning in digital pathology: challenges and opportunities. Med. Image Anal. 2016;33:170-175.

    Not in our library10.1016/j.media.2016.06.037

13. 13.

    Wang HD, et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods. 2019;16:103-110.

    Not in our library10.1038/s41592-018-0239-0

14. 14.

    Wu YC, et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram. Light Sci. Appl. 2019;8:25.

    Not in our library10.1038/s41377-019-0139-9

15. 15.

    Jo Y, et al. Label-free multiplexed microtomography of endogenous subcellular dynamics using generalizable deep learning. Nat. Cell Biol. 2021;23:1329-1337.

    Not in our library10.1038/s41556-021-00802-x

16. 16.

    Cheng SY, et al. Single-cell cytometry via multiplexed fluorescence prediction by label-free reflectance microscopy. Sci. Adv. 2021;7:eabe0431.

    Not in our library10.1126/sciadv.abe0431

17. 17.

    Ounkomol C, et al. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods. 2018;15:917-920.

    Not in our library10.1038/s41592-018-0111-2

18. 18.

    Rivenson Y, et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 2019;3:466-477.

    Not in our library10.1038/s41551-019-0362-y

19. 19.

    Rivenson Y, et al. Emerging advances to transform histopathology using virtual staining. BME Front. 2020;2020:9647163.

    Not in our library10.34133/2020/9647163

20. 20.
    PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning · 2019

    No notice
21. 21.
    Deep learning-based transformation of H&E stained tissues into special stains · 2021

    No notice
22. 22.

    Zhang YJ, et al. Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light Sci. Appl. 2020;9:78.

    Not in our library10.1038/s41377-020-0315-y

23. 23.

    Li JX, et al. Biopsy-free in vivo virtual histology of skin using deep learning. Light Sci. Appl. 2021;10:233.

    Not in our library10.1038/s41377-021-00674-8

24. 24.

    Zhu, J. Y. et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). 2242–2251, 10.1109/ICCV.2017.244 (IEEE, Venice, Italy, 2017).

    Not in our library

25. 25.

    Rivenson, Y. et al. Deep learning-based virtual histology staining using auto-fluorescence of label-free tissue. Print at https://arxiv.org/abs/1803.11293 (2018).

    Not in our library

26. 26.

    Li XY, et al. Unsupervised content-preserving transformation for optical microscopy. Light Sci. Appl. 2021;10:44.

    Not in our library10.1038/s41377-021-00484-y

27. 27.

    Picon A, et al. Autofluorescence image reconstruction and virtual staining for in-vivo optical biopsying. IEEE Access. 2021;9:32081-32093.

    Not in our library

28. 28.

    Meng XY, Li X, Wang X. A computationally virtual histological staining method to ovarian cancer tissue by deep generative adversarial networks. Comput. Math. Methods Med. 2021;2021:4244157.

    Not in our library10.1155/2021/4244157

29. 29.

    Monici M. Cell and tissue autofluorescence research and diagnostic applications. Biotechnol. Annu. Rev. 2005;11:227-256.

    Not in our library10.1016/s1387-2656(05)11007-2

30. 30.

    Croce AC, Bottiroli G. Autofluorescence spectroscopy and imaging: a tool for biomedical research and diagnosis. Eur. J. Histochem. 2014;58:2461.

    Not in our library10.4081/ejh.2014.2461

31. 31.

    Bai BJ, et al. Label-free virtual HER2 immunohistochemical staining of breast tissue using deep learning. BME Front. 2022;2022:9786242.

    Not in our library10.34133/2022/9786242

32. 32.

    Nygate YN, et al. Holographic virtual staining of individual biological cells. Proc. Natl Acad. Sci. USA. 2020;117:9223-9231.

    Not in our library10.1073/pnas.1919569117

33. 33.

    Abraham, T. M. et al. Mode-mapping qOBM microscopy to virtual hematoxylin and eosin (H&E) histology via deep learning. Proceedings of SPIE 12136, Unconventional Optical Imaging III. 121360Q (SPIE, Strasbourg, France, 2022).

    Not in our library

34. 34.

    Borhani N, et al. Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy. Biomed. Opt. Express. 2019;10:1339-1350.

    Not in our library10.1364/boe.10.001339

35. 35.

    Pradhan P, et al. Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning. Biomed. Opt. Express. 2021;12:2280-2298.

    Not in our library10.1364/boe.415962

36. 36.

    Li D, et al. Deep Learning for virtual histological staining of bright-field microscopic images of unlabeled carotid artery tissue. Mol. Imaging Biol. 2020;22:1301-1309.

    Not in our library10.1007/s11307-020-01508-6

37. 37.

    Zhang GH, et al. Image-to-images translation for multiple virtual histological staining of unlabeled human carotid atherosclerotic tissue. Mol. Imaging Biol. 2022;24:31-41.

    Not in our library10.1007/s11307-021-01641-w

38. 38.

    Boktor M, et al. Virtual histological staining of label-free total absorption photoacoustic remote sensing (TA-PARS). Sci. Rep. 2022;12:10296.

    Not in our library10.1038/s41598-022-14042-y

39. 39.

    Kaza N, Ojaghi A, Robles FE. Virtual staining, segmentation, and classification of blood smears for label-free hematology analysis. BME Front. 2022;2022:9853606.

    Not in our library10.34133/2022/9853606

40. 40.

    Soltani S, et al. Prostate cancer histopathology using label-free multispectral deep-UV microscopy quantifies phenotypes of tumor aggressiveness and enables multiple diagnostic virtual stains. Sci. Rep. 2022;12:9329.

    Not in our library10.1038/s41598-022-13332-9

41. 41.

    Soltani S, et al. Deep UV microscopy identifies prostatic basal cells: an important biomarker for prostate cancer diagnostics. BME Front. 2022;2022:9847962.

    Not in our library10.34133/2022/9847962

42. 42.
    Deep learning enables ultraviolet photoacoustic microscopy based histological imaging with near real-time virtual staining · 2022

    Not checked
43. 43.

    Cao, R. et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nature Biomedical Engineering 1-11, https://www.nature.com/articles/s41551-022-00940-z (2022).

    Not in our library10.1038/s41551-022-00940-z

44. 44.

    Mayerich D, et al. Stain-less staining for computed histopathology. Technology. 2015;3:27-31.

    Not in our library10.1142/s2339547815200010

45. 45.

    Bayramoglu, N. et al. Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks. Proceedings of 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). 64–71 (IEEE, Venice, Italy, 2017).

    Not in our library

46. 46.

    Gadermayr, M. et al. Which way round? A study on the performance of stain-translation for segmenting arbitrarily dyed histological images. Proceedings of the 21st International Conference on Medical Image Computing and Computer-Assisted Intervention. 165–173 (Springer, Granada, Spain, 2018).

    Not in our library

47. 47.

    Levy JJ, et al. A large-scale internal validation study of unsupervised virtual trichrome staining technologies on nonalcoholic steatohepatitis liver biopsies. Mod. Pathol. 2021;34:808-822.

    Not in our library10.1038/s41379-020-00718-1

48. 48.

    Lin YY, et al. Unpaired multi-domain stain transfer for kidney histopathological images. Proc. AAAI Conf. Artif. Intell. 2022;36:1630-1637.

    Not in our library

49. 49.

    Liu ST, et al. Unpaired stain transfer using pathology-consistent constrained generative adversarial networks. IEEE Trans. Med. Imaging. 2021;40:1977-1989.

    Not in our library10.1109/tmi.2021.3069874

50. 50.

    Xie WS, et al. Prostate cancer risk stratification via nondestructive 3D pathology with deep learning-assisted gland analysis. Cancer Res. 2022;82:334-345.

    Not in our library10.1158/0008-5472.can-21-2843

51. 51.

    Liu, S. J. et al. BCI: breast cancer immunohistochemical image generation through pyramid Pix2pix. Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1814–1823 (IEEE, New Orleans, 2022).

    Not in our library

52. 52.

    Zhang RR, et al. MVFStain: multiple virtual functional stain histopathology images generation based on specific domain mapping. Med. Image Anal. 2022;80:102520.

    Not in our library10.1016/j.media.2022.102520

53. 53.

    Mercan, C. et al. Virtual staining for mitosis detection in breast histopathology. Proceedings of 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). 1770–1774 (IEEE, Iowa City, IA, USA, 2020).

    Not in our library

54. 54.

    Levy, J. et al. Preliminary evaluation of the utility of deep generative histopathology image translation at a mid-sized NCI cancer center. Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies. 302–311 (SCITEPRESS, Valletta, Malta, 2020).

    Not in our library

55. 55.

    Lahiani A, et al. Seamless virtual whole slide image synthesis and validation using perceptual embedding consistency. IEEE J. Biomed. Health Inform. 2021;25:403-411.

    Not in our library10.1109/jbhi.2020.2975151

56. 56.
    Deep learning-based virtual cytokeratin staining of gastric carcinomas to measure tumor–stroma ratio · 2021

    Not checked
57. 57.

    Im, K. et al. An introduction to performing immunofluorescence staining. in Biobanking: Methods and Protocols (ed Yong, W. H.), 299–311, 10.1007/978-1-4939-8935-5_26 (Humana Press, New York, 2019).

    Not in our library10.1007/978-1-4939-8935-5_26

58. 58.

    Ghahremani P, et al. Deep learning-inferred multiplex immunofluorescence for immunohistochemical image quantification. Nat. Mach. Intell. 2022;4:401-412.

    Not in our library10.1038/s42256-022-00471-x

59. 59.

    Burlingame EA, et al. SHIFT: speedy histological-to-immunofluorescent translation of a tumor signature enabled by deep learning. Sci. Rep. 2020;10:17507.

    Not in our library10.1038/s41598-020-74500-3

60. 60.

    Chen ZH, et al. Deep-learning-assisted microscopy with ultraviolet surface excitation for rapid slide-free histological imaging. Biomed. Opt. Express. 2021;12:5920-5938.

    Not in our library10.1364/boe.433597

61. 61.

    Stacke K, et al. Measuring domain shift for deep learning in histopathology. IEEE J. Biomed. Health Inform. 2021;25:325-336.

    Not in our library10.1109/jbhi.2020.3032060

62. 62.

    Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. 249–256 (JMLR.org, Sardinia, Italy, 2010).

    Not in our library

63. 63.

    Salehi, P. & Chalechale, A. Pix2Pix-based stain-to-stain translation: a solution for robust stain normalization in histopathology images analysis. Proceedings of 2020 International Conference on Machine Vision and Image Processing (MVIP), 1–7 (IEEE, Iran, 2020).

    Not in our library

64. 64.

    Rivenson Y, et al. Deep learning enhanced mobile-phone microscopy. ACS Photonics. 2018;5:2354-2364.

    Not in our library

65. 65.

    Lowe, D. G. Object recognition from local scale-invariant features. Proceedings of the Seventh IEEE International Conference on Computer Vision. 1150-1157 (IEEE, Kerkyra, Greece, 1999).

    Not in our library

66. 66.

    Fischler MA, Bolles RC. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM. 1981;24:381-395.

    Not in our library

67. 67.

    Bay H, et al. Speeded-up robust features (SURF). Comput. Vis. Image Underst. 2008;110:346-359.

    Not in our library

68. 68.

    Otsu N. A threshold selection method from gray-level histograms. IEEE Trans. Syst., Man, Cybern. 1979;9:62-66.

    Not in our library

69. 69.

    Zhang YJ, et al. Virtual staining of defocused autofluorescence images of unlabeled tissue using deep neural networks. Intell. Comput. 2022;2022:9818965.

    Not in our library

70. 70.

    Ashraf M, et al. A loss-based patch label denoising method for improving whole-slide image analysis using a convolutional neural network. Sci. Rep. 2022;12:1392.

    Not in our library10.1038/s41598-022-05001-8

71. 71.
    GANscan: continuous scanning microscopy using deep learning deblurring · 2022

    Not checked
72. 72.

    Reinhard E, et al. Color transfer between images. IEEE Comput. Graph. Appl. 2001;21:34-41.

    Not in our library

73. 73.

    Macenko, M. et al. A method for normalizing histology slides for quantitative analysis. Proceedings of 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 1107–1110 (IEEE, Boston, MA, USA, 2009).

    Not in our library

74. 74.

    Vahadane A, et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans. Med. Imaging. 2016;35:1962-1971.

    Not in our library10.1109/tmi.2016.2529665

75. 75.

    Shrivastava, A. et al. Self-attentive adversarial stain normalization. Proceedings of the International Conference on Pattern Recognition. 120–140, 10.1007/978-3-030-68763-2_10 (Springer, 2021).

    Not in our library10.1007/978-3-030-68763-2_10

76. 76.

    Janowczyk A, Basavanhally A, Madabhushi A. Stain normalization using sparse AutoEncoders (StaNoSA): application to digital pathology. Comput. Med. Imaging Graph. 2017;57:50-61.

    Not in our library10.1016/j.compmedimag.2016.05.003

77. 77.

    Goodfellow, I. et al. Generative adversarial nets. Proceedings of the 27th International Conference on Neural Information Processing Systems. 2672–2680 (MIT Press, Montreal, Canada, 2014).

    Not in our library

78. 78.

    Liu TR, et al. Deep learning-based holographic polarization microscopy. ACS Photonics. 2020;7:3023-3034.

    Not in our library10.1021/acsphotonics.0c01051

79. 79.

    Yang, X. L. et al. Virtual stain transfer in histology via cascaded deep neural networks. ACS Photonics9, 3134–3143, 10.1021/acsphotonics.2c00932 (2022).

    Not in our library

80. 80.

    Isola, P., et al. Image-to-image translation with conditional adversarial networks. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5967–5976, 10.1109/CVPR.2017.632 (IEEE, Honolulu, HI, USA, 2017).

    Not in our library

81. 81.

    Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015- 18th International Conference.: (Springer, Munich, Germany, 2015).

    Not in our library

82. 82.

    Wang Z, et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 2004;13:600-612.

    Not in our library10.1109/tip.2003.819861

83. 83.

    Chambolle A. An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 2004;20:89-97.

    Not in our library

84. 84.

    Huynh-Thu Q, Ghanbari M. Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 2008;44:800-801.

    Not in our library

85. 85.

    Wang, Z., Simoncelli, E. P. & Bovik, A. C. Multiscale structural similarity for image quality assessment. Proceedings of the Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. 1398–1402 (IEEE, Pacific Grove, CA, USA, 2003).

    Not in our library

86. 86.

    Heusel, M. et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Proceedings of the 31st International Conference on Neural Information Processing Systems. 6629–6640 (Curran Associates Inc., Long Beach, USA, 2017).

    Not in our library

87. 87.

    Salimans, T. et al. Improved techniques for training GANs. Proceedings of the 30th International Conference on Neural Information Processing Systems. 2234–2242 (Curran Associates Inc., Barcelona, Spain, 2016).

    Not in our library

88. 88.

    Landini G, Martinelli G, Piccinini F. Colour deconvolution: stain unmixing in histological imaging. Bioinformatics. 2021;37:1485-1487.

    Not in our library10.1093/bioinformatics/btaa847

89. 89.
    Deep learning in cancer diagnosis, prognosis and treatment selection · 2021

    No notice
90. 90.

    Turnaround Time (TAT) - Biopsies. https://documents.cap.org/documents/cap-22-turnaround-time-biopsies-2021.pdf (2021).

    Not in our library

91. 91.

    Gu Y, et al. Transfer recurrent feature learning for endomicroscopy image recognition. IEEE Trans. Med. Imaging. 2019;38:791-801.

    Not in our library10.1109/tmi.2018.2872473

92. 92.

    Yagi Y. Color standardization and optimization in Whole Slide Imaging. Diagn. Pathol. 2011;6:S15.

    Not in our library10.1186/1746-1596-6-s1-s15

93. 93.

    Tomczak K, Czerwińska P, Wiznerowicz M. Review The Cancer Genome Atlas (TCGA): an immeasurable source of knowledge. Contemp. Oncol. 2015;19:A68-A77.

    Not in our library10.5114/wo.2014.47136

94. 94.

    Bándi P, et al. From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYON17 challenge. IEEE Trans. Med. Imaging. 2019;38:550-560.

    Not in our library10.1109/tmi.2018.2867350

95. 95.
    Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge · 2022

    No notice
96. 96.

    Aubreville M, et al. Mitosis domain generalization in histopathology images - The MIDOG challenge. Med. Image Anal. 2022;84:102699.

    Not in our library10.1016/j.media.2022.102699

97. 97.

    Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. 9992–10002. (IEEE, Montreal, QC, Canada, 2021).

    Not in our library

98. 98.

    Dosovitskiy, A. et al. An image is worth 16x16 words: transformers for image recognition at scale. Print at https://arxiv.org/abs/2010.11929 (2021).

    Not in our library

99. 99.

    Chen, R. J. et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16123–16134, 10.1109/CVPR52688.2022.01567 (IEEE, New Orleans, LA, USA, 2022).

    Not in our library

100. 100.

     Gómez-de-Mariscal E, et al. DeepImageJ: a user-friendly environment to run deep learning models in ImageJ. Nat. Methods. 2021;18:1192-1195.

     Not in our library10.1038/s41592-021-01262-9

101. 101.
     CellProfiler 4: improvements in speed, utility and usability · 2021

     No notice
102. 102.
     QuPath: Open source software for digital pathology image analysis · 2017

     No notice
103. 103.
     PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning · 2019

     No notice
104. 104.

     Bai, B. J. et al. Label-free virtual HER2 immunohistochemical staining of breast tissue using deep learning. Print at https://arxiv.org/abs/2112.05240 (2021).

     Not in our library10.34133/2022/9786242

105. 105.

     de Haan K. et al. Deep learning-based transformation of the H&E stain into special stains. Print at https://arxiv.org/abs/2008.08871 (2020).

     Not in our library

106. 106.

     Bouteldja N, et al. Improving unsupervised stain-to-stain translation using self-supervision and meta-learning. J. Pathol. Inform. 2022;13:100107.

     Not in our library10.1016/j.jpi.2022.100107

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.