Deep Learning for Medical Image-Based Cancer Diagnosis

1In recent years, the global incidence of cancer has remained high. Tens of millions of people are newly diagnosed with various types of cancer every year. At the same time, millions to nearly tens of millions of people around the world are killed by various types of cancer. According to the 2020 global cancer burden data released by the International Agency for Research on Cancer (IARC) of the World Health Organization, the latest incidence and mortality trends of 36 cancer types in 185 countries worldwide are still grim. Based on the latest incidence rates, the world’s top ten cancers are female breast, lung, skin, prostate, colon, stomach, liver, rectum, esophageal, and cervix uteri.

2At present, the diagnosis of cancer mainly depends on imaging diagnosis and pathological diagnosis. In this case, early detection is the key to improving the survival rate of cancer patients; non-invasive and efficient early screening has become an essential research topic. Imaging techniques include B-ultrasound, X-ray, computed tomography (CT), magnetic resonance imaging (MRI), etc.. Through these imaging techniques, some cancerous symptoms of the body can be seen. A shadow in the lungs can be detected by CT, which can determine whether it is a symptom of lung cancer. MRI is not only used to assist in the diagnosis and differentiation of nasopharyngeal carcinoma but also can be used to evaluate the extent of the cancer lesion: whether it involves the surrounding soft tissue and bone and whether there is metastasis to nearby lymph nodes. Nodules or masses of different sizes in the thyroid can be found through a B-ultrasound examination and can also directly observe the size, shape, location, and boundary of the tumor in the thyroid through B-ultrasound. Faced with a large amount of complex medical imaging information and growing demand for medical imaging diagnosis, artificial imaging diagnosis has many shortcomings, such as a heavy workload, susceptibility to subjective cognition, low efficiency, and high misdiagnosis rate.

3Deep learning (DL), a branch of machine learning, is an algorithm based on an artificial neural network to learn features from data. Deep learning proposes a method that enables computers to learn pattern features automatically and integrates feature learning into the process of model building, thus reducing the incompleteness caused by artificial design features and realizing the development of end-to-end prediction models.

4Algorithms based on deep learning have advantages over humans in processing large data and complex non-deterministic data as well as in-depth mining of potential information in data. Using deep learning to interpret medical images can help doctors locate lesions, assist in diagnosis, reduce the burden on doctors, reduce medical misjudgments, and improve the accuracy and reliability of diagnosis and prediction results. Deep learning techniques were successfully applied in various fields through medical images and physiological signals. Deep models have demonstrated excellent performance in many fields, such as medical image classification, segmentation, lesion detection, and registration. Various types of medical images, such as X-ray, MRI, CT, etc., were used to develop accurate and reliable DL models to help clinicians diagnose lung cancer, rectal cancer, pancreatic cancer, gastric cancer, prostate cancer, brain tumors, breast cancer, etc..

5Due to the benefits of deep learning in cancer diagnosis, a large number of researchers are attracted. As the technology continues to improve, there is still spacious room for deep learning to be used in medical image analysis. In view of this, we review and summarize the development of deep learning in medical image-based cancer diagnosis through surveys, aiming to provide a more comprehensive introduction for beginners and to provide the latest deep learning techniques for researchers in this field.

6We conducted a systematic review according to the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA). Through Google Scholar, Elsevier, and Springer Link, we searched for papers on the application of deep learning in medical image-based cancer diagnosis with the keywords “deep learning”, “cancer”, and “medical image”. Based on the title and content, we eliminated irrelevant and duplicate papers and finally included 389 papers.

7In summary, the novelty and contributions of this work are as follows:The principle and application of radiological and histopathological images in cancer diagnosis are introduced in detail;This paper introduces 9 basic architectures of deep learning, 12 classical pretrained models, and 5 typical methods to overcome overfitting. In addition, advanced deep neural networks are introduced, such as vision transformers, transfer learning, ensemble learning, graph neural network, and explainable deep neural networks;The application of deep learning technology in medical imaging cancer diagnosis is deeply analyzed, including image classification, image reconstruction, image detection, image segmentation, image registration, and image fusion;The current challenges and future research hotspots are discussed and analyzed around data, labels, and models.

8The principle and application of radiological and histopathological images in cancer diagnosis are introduced in detail;

9This paper introduces 9 basic architectures of deep learning, 12 classical pretrained models, and 5 typical methods to overcome overfitting. In addition, advanced deep neural networks are introduced, such as vision transformers, transfer learning, ensemble learning, graph neural network, and explainable deep neural networks;

10The application of deep learning technology in medical imaging cancer diagnosis is deeply analyzed, including image classification, image reconstruction, image detection, image segmentation, image registration, and image fusion;

11The current challenges and future research hotspots are discussed and analyzed around data, labels, and models.

12The rest of this paper is organized as follows: In Section 2, we will introduce the common imaging techniques. In Section 3, we will show the basic architecture of deep learning, classical pretrained models, advanced deep neural networks, and overfitting prevention techniques. In Section 4, we will present a comprehensive review of the application of deep learning in medical image-based cancer diagnosis. In Section 5, we will describe the experimental details and a discussion of the results. The conclusion of this research will be described in Section 6.

13Currently, the most widely used medical images include radiology, pathology, endoscopy, etc. Radiological imaging techniques, such as X-ray, CT, MRI, B-ultrasound, etc., play an important role in the diagnosis of deep tissue lesions in the human body. The common radiological imaging techniques in cancer diagnosis are shown in Table 1. Pathological images refer to images of disease tissues and cells obtained by microscopy and other imaging techniques, including cytological and histological images.

14Computed tomography (CT) is a technology that obtains three-dimensional reconstruction images of the inside of the human body by rotating and irradiating the human body with X-rays. The X-ray beam is used to scan the layer of a certain thickness of the human body. After receiving the X-ray through the layer, the detector converts it into visible light, which is converted into an electrical signal by the photoelectric converter, and then converted into a digital signal by an analog-to-digital converter and input to the computer for processing, and finally, the image is formed.

15The basic principle of CT is shown in Figure 1 and the basic concepts and features of CT are shown in Table 2. According to the rays used, it can be divided into: X-ray CT (X-CT) and gamma-ray CT (γ-CT); the term “CT” usually refers to X-ray CT.

16The four areas of most interest in CT are CT colonoscopy (virtual colonoscopy), CT lung screening of current and former smokers, CT cardiac screening, and CT whole-body screening. Lung cancer screening with low-dose computed tomography has been shown to reduce mortality by 20–43%. Tian et al. developed a deep learning model to predict high PD-L1 expression in non-small cell lung cancer using computed tomography (CT) images and to infer the clinical outcome of immunotherapy. Body composition on preoperative chest computed tomography is an independent predictor of hospital length of stay (LOS) and postoperative complications after lobectomy for lung cancer. Measuring body composition using diagnostic computed tomography (CT) has emerged as a method for assessing sarcopenia (low muscle mass) in cancer patients.

17Magnetic resonance imaging (MRI) is the interaction of spinning nuclei with electromagnetic waves in a magnetic field to map the spatial position and related properties of specific nuclei or protons in an object. The MRI system consists of four main parts: the main magnet, gradient system, radio frequency system, and computer system. The general imaging process is shown in Figure 2 and the basic concepts and features of MRI are shown in Table 3.

18MRI has obvious advantages in diagnosing liver cancer and indeed the performance of liver cancer magnetic resonance is relatively specific. Liver cancer cases have typical imaging performance characteristics on magnetic resonance. Magnetic resonance has a variety of sequences, each of which has a certain value in the diagnosis of liver cancer. Very small liver cancer lesions can be detected through magnetic resonance. In addition, MRI has a high value in the diagnosis of other abdominal cancer, such as pancreatic cancer and kidney cell carcinoma. MRI has the potential as a biomarker for the diagnosis, treatment, and prognosis of renal cell carcinoma (RCC).

19MRI has been shown to be equal to or superior to other imaging modalities in the diagnosis of pancreatic cancer and its versatility can accurately delineate the pancreatic parenchyma as well as ductal structures, such as enhanced MRI, which can help distinguish pancreatic cancer from focal pancreas inflammation. MRI has its unique superiority in examining blood vessels, which can be examined in different parts according to different coils. MRI is also the most commonly used examination for the diagnosis of brain tumors, providing detailed information on brain tumor anatomy, cellularity, and vascular supply. MRI, especially dynamic contrast-enhanced MRI, is currently considered the most sensitive method for detecting breast cancer without ionizing radiation and an effective screening alternative for high-risk populations.

20MRI has a high diagnostic value for hypertrophy tumors and lung cancer staging. MRI is significantly better than CT in the diagnosis of the staging of bladder cancer, prostate cancer, and cervical cancer. Wu et al. developed a deep-learning model to diagnose cervical cancer lymph node metastasis using magnetic resonance imaging before surgery. Multiparametric magnetic resonance imaging (mpMRI) can accurately perform local staging, predict tumor aggressiveness, and monitor response to therapy for bladder cancer (BCa) patients. Contrast-enhanced MR and diffusion-weighted MR (DW-MRI) can distinguish non-muscle-invasive bladder cancer from muscle tumors with 80% accuracy. mpMRI is a routine imaging method for the diagnosis of prostate cancer but 10–20% of prostate tumors are missed. MRI in combination with prostate-specific membrane antigen positron emission tomography can reduce false negatives in prostate cancer (csPCa).

21The good tissue resolution of MRI can better demonstrate the invasive changes in the rectal wall and surrounding infiltration, especially the high-resolution magnetic resonance imaging (HRMRI), intravoxel incoherent motion diffusion-weighted imaging (IVIMDWI), dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI), diffusional kurtosis imaging (DKI), and magnetic resonance spectroscopy (MRS); other imaging techniques have made MRI a comprehensive assessment of effective imaging methods of rectal cancer clinical diagnosis and treatment of rectal cancer.

22The sound waves with more than 200,000 vibrations per second are not felt by ears and are called ultrasonic. Ultrasound (US) can spread in a certain direction and penetrate objects. If an obstacle is encountered, echo will occur. Taking real-time two-dimensional color flow ultrasounds as an example, the imaging system block diagram is shown in Figure 3 and the basic concepts and features of ultrasound are shown in Table 4.

23Through B-ultrasound, a variety of clear sectional graphs of internal organs of the human body can be obtained. Most tumors can also be detected, such as breast cancer, thyroid cancer, liver cancer, pancreatic cancer, bladder cancer, kidney cancer, ovarian cancer, etc.. Still, B-ultrasound cannot detect some cancers, such as bone tumors, gastrointestinal tumors, and lung tumors. As a result of the fact that the lungs and gastrointestinal tract contain gas, ultrasonic waves cannot penetrate the lungs and gastrointestinal tract due to reflection against the gas during the examination, so the detection of lung cancer and gastrointestinal tumors is limited. Bone tumors cannot be penetrated by ultrasound when passing through bone, so it also greatly influences the examination of bone tumors.

24Safe and effective delivery of anticancer drugs to lesions is one of the grand challenges in cancer therapy. An ultrasound-triggered drug delivery system (UTDDS) was developed into a new type of efficient and non-invasive drug delivery technology, such as using ultrasound to break through the blood–brain barrier to deliver drugs to brain tumors. It can position ultrasound radiation in the tumor area and promote the regular and quantitative release of the drug through ultrasound stimulation to achieve a high concentration of local chemotherapy drugs and improve the anti-cancer effect. The thermal effect produced by high-intensity ultrasound is helpful for cancer treatment.

25X-ray is a kind of electromagnetic wave with a very short wavelength. The wavelength is about 10 nm to ~0.1 nm. A typical X-ray imaging process is shown in Figure 4 and the basic concepts and features of X-ray are shown in Table 5.

26X-ray is mainly used to detect bone lesions. Generally, X-ray images of bone in cancer areas differ from those of surrounding healthy bone and muscle areas, providing a low-cost screening tool for diagnosing and visualizing bone cancer. X-rays are also useful for detecting soft tissue lesions. For example, chest X-rays (CXR) diagnose lung diseases such as pneumonia, lung cancer, or emphysema. Abdominal X-rays detect intestinal infarcts, free gas, and free fluid. The sensitivity of chest X-rays for symptomatic lung cancer is only 77% to 80%. Chest X-rays with artificial intelligence can capture more lung cancers.

27Positron emission tomography (PET) imaging utilizes the β+ decay of radioactive isotopes, allowing noninvasive imaging, and is well suited for monitoring early cellular/molecular events during disease processes, as well as during drug or radiation therapy. The process of PET imaging mainly includes detection data generation, data acquisition and storage, data processing and reorganization, image reconstruction, and image display and processing, as shown in Figure 5. The basic concepts and features of PET are shown in Table 6.

28Comprehensive PET–CT can improve the accuracy of the staging diagnosis of non-small cell lung cancer (NSCLC). Despite the risk of false-positive bone lesions, PET–CT outperforms all other imaging modalities in detecting primary distant metastases from high-risk prostate cancer and its role in lung cancer treatment has expanded exponentially. MRI in combination with prostate-specific membrane antigen positron emission tomography can reduce false negatives for clinically significant prostate cancer (csPCa) compared with MRI, potentially reducing the number of prostate biopsies required to diagnose csPCa.

29Histopathology refers to observing the morphological changes of histopathology cells under a microscope after biopsy or when surgical specimens are made into tissue slides to diagnose disease. The generation of histopathological images mainly includes histopathological image acquisition, tissue slide, chemical staining, and histopathological image analysis. The basic concepts and features of histopathological images are shown in Table 7.

30The concept of deep learning originates from the research of artificial neural networks. A multi-layer perceptron with multiple hidden layers is a deep learning structure. The commonly used models for deep learning mainly include convolutional neural network (CNN), deep belief network (DBN), deep autoencoder (DAE), restricted Boltzmann machine (RBM), etc. The basic deep learning architectures are shown in Table 8.

31Using a convolutional neural network (CNN) is the most prominent technique of deep learning in the field of image processing. CNN is mostly used for medical image processing tasks among the different neural network architectures. It is a kind of feed-forward neural network with convolution calculation and a deep structure. It is specially designed to process neural networks with similar grid structure data, such as time series and image data. CNNs can be trained using the backpropagation algorithm. The main structure of a convolutional neural network includes the input layer, convolutional layer, pooling layer, fully connected layer, and output layer, as shown in Figure 6.

32The purpose of the convolution operation is to extract different input features. Some convolutional layers may only extract low-level features such as edges, lines, and corners. More layers of networks can iteratively extract more complex features from low-level features. The convolutional layer plays the role of feature extraction but it does not reduce the number of features of the picture.

33The final fully connected layer still faces many parameters, so the pooling layer is required to reduce the number of features. The pooling layer mainly performs subsampling processing on the feature map learned by the convolutional layer, mainly consisting of two types: max pooling and average pooling. The former takes the maximum value within the window as the output, while the latter takes the mean of all the values within the window as the output. The pooling layer reduces the input dimension of subsequent network layers, reduces the model’s size, improves the calculation speed, and improves the feature map’s robustness to prevent overfitting. Finally, the fully connected layer is used to realize recognition and judgment.

34CNN combines local perception, weight sharing, and down sampling to fully use the locality and other characteristics of the data, optimize the network structure, and ensure a certain degree of invariance in displacement and deformation.

35A fully convolutional network (FCN) is a framework for image semantic segmentation proposed by Shelhamer et al. in 2015. It replaces the fully-connected layer behind the traditional CNN with a convolutional layer. It is a fully-convolutional network without a fully-connected layer which realizes pixel-level classification of images, thereby solving the semantic-level image segmentation problem.

36The FCN network structure is mainly divided into two parts: the full convolution part and the deconvolution part. The full convolution part entails some classic CNN networks (such as VGG, ResNet, etc.) used to extract features. Deconvolution is introduced into the up sampling process to up sample the feature map of the last convolutional layer to restore it to the same size as the input image so that a prediction can be generated for each pixel while retaining the spatial information in the original input image. Finally, pixel classification is performed on the up sampled feature map and the output is a picture that has been labeled. Therefore, the input of FCN can be a color image of any size.

37FCN uses the corresponding relationship between the output result and the input image, directly gives the classification of the corresponding region of the input image and cancels the sliding window selection candidate box in the traditional target detection. FCN itself still has many limitations. For example, it does not consider global information, cannot solve the problem of instance segmentation, is not sensitive to the details in the image, and the speed is far from real-time.

38The autoencoder (AE) concept was first proposed by Rumelhart et al.. Using an autoencoder entails an unsupervised data compression and data feature expression method. It is based on the backpropagation algorithm and optimization methods (such as the gradient descent method), using the input data as supervision to guide the neural network to try to learn a mapping relationship to obtain a reconstructed output of the original input.

39The autoencoder can be seen as a three-layer network (input layer, hidden layer, and output layer), including an encoder and a decoder. The encoder maps the input data of the high-dimensional space to the encoding of the low-dimensional space to achieve data compression and the decoder decompresses the input data to achieve recurrence. The decoder is usually eliminated and the encoder model is retained after completing the training phase to extract the input data features. The structure of the autoencoder is shown in Figure 7.

40In the encoding stage, the input is encoded by the encoding function, as shown in Equation (1), and the encoding is obtained to realize data compression. where is the activation function of the encoder, which is usually taken as sigmoid function, as shown in Equation (2). where and represent the weight matrix and bias vector of the encoder (i.e., between the input layer and the hidden layer), respectively.

41In the decoding stage, the decoding function is used to map the code to the original high-dimensional space to achieve the reproduction of the input, as shown in Equation (3): where and are the weight matrix between the hidden layer and the output layer (for simplicity, the weight matrix here is taken as the transpose of W), is the activation function of the decoder, usually taken as the sigmoid function or the identity function, as shown in Equations (4) and (5), respectively.

42Single-layer autoencoders are not enough to obtain a good data representation due to their simple shallow structural features. We can use deeper neural networks to better capture the semantic information of the data and improve its representation ability. A stacked autoencoder (SAE) generally adopts layer-wise unsupervised pre-training to learn network parameters. The essence of SAE is to increase the number of hidden layers on the basis of AE to obtain a better feature extraction capability. Due to the hierarchical structure, one of the most important features of SAE is that highly nonlinear and complex patterns can be learned or discovered.

43When the sample has a sparse representation, the performance of the classification task will be improved. The sparse autoencoder is obtained by adding some sparsity constraints on the basis of the traditional autoencoder. This sparsity is for the hidden layer neurons of the autoencoder. By suppressing most of the output of the hidden layer neurons, the network can achieve a sparse effect so that few useful feature items can be obtained. A neuron is considered active when the output of the neuron is close to 1 and it is considered inhibitory when the output is close to 0. The neuron is limited by our desire to be inactive most of the time.

44In addition to properties such as the minimum reconstruction error or sparsity, effective data representation may require other properties, such as robustness to partial destruction of data. A denoising autoencoder (DAE) is an autoencoder that accepts damaged data as an input and trains them to predict the original undamaged data as an input. The core idea of DAE is to increase the robustness of data encoding by introducing noise and improving the model’s generalization ability.

45There are many other variants of autoencoders, such as a variational autoencoder (VAE), contraction autoencoder (CAE), etc. The common variants of autoencoder are shown in Table 9. Yousuff et al. proposed a mixed-dimensional reduction method based on a deep automatic encoder for a machine-learning model of a melanoma cancer diagnosis. In this method, the Neighborhood Component Analysis (NCA) technique is applied to transform the 2090-dimensional spectral features into a 2090-dimensional vector and the deep autoencoder model is used to process non-linearity in the data and generate potential space. Xu et al. applied a stacking sparse autoencoder (SSAE) for nuclei detection on high-resolution histopathological images of breast cancer. Stacked sparse autoencoder models can capture high-level feature representations of pixel intensities. A single-layer sparse autoencoder was successfully used to extract advanced features in the diagnosis of clinically significant prostate cancer (PCa) using multi-parameter magnetic resonance imaging (mpMRI) biomarkers in highly unbalanced datasets.

46For cancer survival prediction, Huang et al. used a typical autoencoder structure to reconstruct the original input, adopted sparse coding technology to optimize the network structure, and minimized the number of network weights to identify information features accordingly so that the optimal features were selected and the generalization ability was enhanced. Munir et al. designed a sparse deep convolutional autoencoder model to assist breast cancer diagnoses. The input image in this model can be the low-rank, multi-channel, and approximate thermal base.

47The extreme learning machine (ELM) is a fast learning algorithm proposed by Huang et al. which can recognize images with a single hidden layer neural network. During the training, there is no need to adjust or update parameters, only the hidden layer nodes are adjusted to find the optimal solution. Compared with traditional classification methods such as CNN and SVM, ELM has a faster learning speed and stronger generalization performance. However, feature learning using ELM methods may not be effective for some image classification applications due to their shallow architecture.

48Taking advantage of the corresponding network, the feature extraction performance of the convolutional neural network and the fast training of the extreme learning machine are combined in a deep convolutional extreme learning machine (DC-ELM). As shown in Figure 8, the structure of DC-ELM consists of an input layer, an output layer, and multiple hidden layers. The hidden layers are alternately arranged as convolutional and pooling layers which can effectively extract high-level features from the input image. The convolutional layer is composed of several feature maps. The same feature map shares the same input weight but the input weight of different feature maps differs. The pooling layer uses the square root pooling layer to enable the network to have frequency selective and translation invariance. It has the same number and size of feature maps as the previous convolution layer. The last layer is fully connected with the output layer and adopts the random pooling strategy to reduce the size of its feature graph and save significant computing resources.

49A recurrent neural network (RNN) is a class of recursive neural networks that takes sequence data as the input, makes recursion in a sequence’s progression direction, and links cyclic units in a chain. RNN is suitable for dealing with timing-related issues such as video, voice, and text. In the field of medical images, an RNN is used to assist in cancer classification and assessment, prediction, and clinical data modeling. In order to achieve better accuracy, RNNs are often combined with other networks (such as convolutional cyclic neural network, convolutional grid neural network, hybrid method of the recurrent neural network and graph neural network (RGNN), and wavelet recurrent neural network), or the internal structure of RNNs can be improved (such as bidirectional recurrent neural network, long short-term memory network (LSTM), and fuzzy recurrent neural network (FR–Net)).

50In RNNs, the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to calculate the current output. That is, the nodes between the hidden layers are connected and the input of the hidden layer includes the output of the input layer and the output of the previously hidden layer. In theory, RNNs can process sequence data of any length. However, it is often assumed that the current state is only related to the previous states to reduce complexity in practice. The model structure is shown in Figure 9.

51,, and are all vectors representing the values of the input layer, hidden layer, and output layer, respectively. is the weight matrix from the input layer to the hidden layer and is the weight matrix from the hidden layer to the output layer. The weight matrix is the last value of the hidden layer as the input weight of this time.

52After the network receives the input at time, the value of the hidden layer is and the output value is. The value of depends not only on, but also on. We can use the following formula to express the calculation method of the cyclic neural network:

53It should be noted that in the same hiding layer,, and at different times are all equal, which is the parameter sharing of RNN. These parameters are updated when backpropagation is performed.

54The parameter learning of the cyclic neural network can be learned through the backpropagation algorithm over time. That is, the error is passed forward step by step in the reverse order of time. Unfortunately, when the input sequence is relatively long, the gradient explosion or gradient disappearance will occur, which is also called the long-term dependence problem. Gating mechanisms are introduced to improve recurrent neural networks to solve this problem.

55Long and short-term memory (LSTM) is a kind of RNN proposed by Hochreiter et al. in 1997, which is suitable for processing and predicting important events with relatively long intervals and delays in time series. It is used to solve the problem of explosion or disappearance that usually occurs due to the long-term dependence of RNN. Gers et al. added “Forget Gate” to LSTM in 2000, allowing the network to reset its state in order to strengthen the proper memory reset function. Forgetting may occur rhythmically or in an input-dependent manner. An ordinary LSTM cell consists of a cell, an input gate, an output gate, and a forget gate.

56The ability of the LSTM to retain long-term information lies in the design of the “gate” structure. In LSTM, the first stage is the forget gate which determines what information needs to be forgotten from the cell state. The next stage is the input gate, which determines what new information can be stored in the cell state. And the last stage is the output gate, which determines what values to output.

57Medical image data acquisition time points will appear uneven and irregular. In order to model both regular and irregular longitudinal samples, LSTM networks are widely used in serial observation modeling in fields such as medical imaging. Researchers have shown that LSTM is superior to other models in processing irregular medical data. Koo et al. developed an online decision support system based on the LSTM model for survival prediction of prostate cancer, providing individualized survival outcomes based on initial treatment plans. Time-series tumor marker (TM) data were used to further improve the predictive performance of LSTM models, even with widely varying intervals between tests, so that occult tumors could be detected earlier.

58A generative adversarial network (GAN) was systematically proposed by Goodfellow et al.. The GAN network structure contains two models: one is the generator and the other is the discriminator. During the training process, both the real data extracted from the dataset and the fake data that the generator keeps creating throughout the training process are included. The generator is used to capture the distribution of the entire data. The generator fits and approximates the real data distribution; the discriminator estimates whether a sample comes from real data or is generated. The GAN model structure diagram is shown in Figure 10.

59We input random noise into the generator to generate samples, the function is just a function represented by a neural network, which converts random and unstructured vectors into structured data with the aim of being statistically indistinct from the training data.

60The data satisfying the real distribution is recorded as. and are sent to the discriminator at the same time for training. The discriminator is trained in much the same way as any other binary classifier, except that the pseudo-classes data come from a constantly changing distribution as the generator learns rather than from a fixed distribution. The generator and the discriminator form a mutual confrontation relationship. During repeated game training, the recognition ability of the network becomes stronger and stronger. Finally, the network reaches the optimal state when the discriminator cannot distinguish whether the generated result is true (discrimination probability is 0.5).

61In the application of medical images to assist in cancer diagnosis, GAN can be used to generate medical images that are identical to real images to solve the problem of insufficient training data.

62The deep belief network (DBN) was proposed by Geoffrey Hinton in 2006. It is an effective method to solve the problems of slow learning speed and the overfitting phenomenon of deep neural networks. It is a probabilistic generation model that gives the entire network a better initial weight through layer-by-layer training to reach the optimal solution after fine-tuning.

63The restricted Boltzmann machine (RBM) plays the most important role in layer-by-layer training, which is a two-layer neural network. The first layer is a visible layer representing data used to input training data. The second layer is a hidden layer of hidden units representing features that capture high-order correlations in the data, used as feature detectors. Adjacent layers of RBM are connected. Each neuron in the visible layer is connected to all neurons in the hidden layer but there is no connection between neurons in the same layer and the output state of all neurons is only two kinds.

64DBN is a neural network composed of multi-layer RBM. It can be regarded as either a generation model or a discriminant model. Its training process is to pre-train to obtain weights by using an unsupervised greedy layer-by-layer method. The classic DBN network structure is a deep neural network composed of several layers of RBM and a layer of BP, as shown in Figure 11.

65DBN is mainly divided into two steps in the process of training the model.

66Step 1 pre-training: train each layer of the RBM network separately and unsupervised to ensure that feature information is retained as much as possible when feature vectors are mapped to different feature spaces.

67Step 2 fine-tuning: set the BP network in the last layer of DBN, receive the output feature vector of RBM as its input feature vector, and train the entity relationship classifier under supervision. And each layer of the RBM network can only ensure that the weight value is optimal for the feature vector mapping of this layer, not for the feature vector mapping of the entire DBN, so the backpropagation network also propagates error information from top to bottom to each layer of RBM, thus fine-tuning the entire DBN network.

68The process of the RBM network training model can be regarded as the initialization of weight parameters of a deep BP network so that DBN overcomes the shortcomings of the BP network that are easy to fall into local optimum and long training times due to the random initialization of weight parameters. The top layer of the above training model with supervised learning can be replaced with any classifier model according to the specific application field instead of the BP network. By training the weights between its neurons, the entire neural network can generate training data according to the maximum probability. We can not only use DBN to identify features and classify data but also use it to generate data to solve the problem of insufficient sample size.

69The application of deep belief networks involves two main challenges: the method of fine-tuning the network weights and biases and the number of hidden layers and neurons. Ronoud et al. applied extreme learning machine (ELM) and backpropagation (BP) algorithms to DBN to optimize the fine-tuning method of network weights and used the genetic algorithm to optimize the DBN structure in the proposed model to correctly give the number of hidden layers in DBN and the number of neurons in each layer.

70The deep Boltzmann machine (DBM) is a deep learning model based on the restricted Boltzmann machine, proposed by Salakhutdinov et al.. It is essentially composed of stacked multi-layer restricted Boltzmann machines. Uniquely from the deep belief network, the middle layer of the DBM is bidirectional and connected to the adjacent layer. Compared with RBM and DBN, DBM has a powerful shape representation ability which can capture more global and local constraints of object shape so as to construct a strong probabilistic object shape model that meets the requirements of realism and generalization.

71Wu et al. built a heart shape model by training a three-layer DBM to characterize local and global heart shape variations for better heart motion tracking. Syafiandini et al. used the multimodal deep Boltzmann machine to learn important genes (biomarkers) on gene expression data of human colorectal cancer, including several patient phenotypes, such as the occurrence of lymph nodes and distant metastases. Hess et al. used deep Boltzmann machines to model immune cell gene expression patterns in lung adenocarcinoma and partitioned DBMs to simulate the expression of immune cell-related genes in lung adenocarcinoma. Their approach solves the problem that DBMs are limited to the amount of individual training data far greater than the amount of feature data.

72This section will introduce typical deep learning architectures based on neural networks, the basics of which are shown in Table 10.

73The LeNet-5 was proposed by Lecun et al. in 1998 to solve the visual task of handwritten character recognition. The convolutional network architecture is based on the following three ideas: local receptive field, shared weights, and sub-sampling in time or space, which is the latter pooling layer, to ensure a certain degree of invariance to shift, scale, and distortion.

74LeNet-5 has a total of seven layers. In addition to the input layer, each layer contains trainable parameters. Each network layer generates multiple feature maps and each feature map can extract a type of feature of the input data through a convolution filter. The LeNet network includes the basic modules of deep learning: convolution, pooling, and fully connected layers, as shown in Figure 12. Feature extraction is performed through operations such as convolution and pooling, and finally, classification and recognition are realized using full connections.

75The weight-sharing feature of the convolutional layer saves a considerable amount of calculation and memory space compared with the fully connected layer. In the convolution operation, neurons are locally connected in spatial dimension but fully connected in depth. For the two-dimensional image, the local pixel correlation is strong and the local connection feature ensures that the learned filter can respond strongly to the local input features.

76AlexNet deepens the network structure on the basis of LeNet and learns richer and higher-dimensional image features. Compared to LeNet, AlexNet has a deeper network structure with five convolutional layers and three fully connected layers. The output of the last fully connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels.

77In AlexNet, the rectified linear unit (ReLU) is used as the activation function of CNN which successfully solves the problem of gradient dispersion of the sigmoid function when the network is deep. In addition, the ReLU function is simpler than the sigmoid function and requires less computation, which speeds up the training speed. During the training process of the neural network, the powerful parallel computing capability of the GPU is used to handle a large number of matrix operations to speed up the training. AlexNet distributes parallel training on two GPUs, stores half of the neuron parameters in the video memory of each GPU, and the GPU only communicates in specific layers.

78The local response normalization (LRN) layer was proposed to create a competition mechanism for the activity of local neurons so that the value with a larger response becomes relatively larger and suppresses neurons with smaller feedback. The generalization ability of the model is enhanced. Overlapping pooling is used successfully and experiments show that overlapping pooling layers are less prone to overfitting. Data augmentation and dropout are used to suppress overfitting, where data augmentation includes two methods: generating image translations and horizontal reflections and changing the intensity of the RGB channel in the training image. The dropout technique reduces the complex co-adaptations of neurons.

79ZF-Net is a network designed by Zeiler et al. in 2013 with minor improvements based on AlexNet. The functions of the intermediate feature layer and the operation of classification can be understood through novel visualization techniques. Since the size and the stride of the first-layer convolution kernel of AlexNet are too large, the extracted features are mixed with a lot of high-frequency and low-frequency information and lack intermediate-frequency information, so the size of the first-layer filter of ZF-Net is adjusted from 11 × 11 to 7 × 7 and the stride was changed from 4 to 2. This new architecture not only preserves more features in layer 1 and layer 2 but it also improves classification performance.

80The deconvnets in ZF-Net are used to map the intermediate layer features to the input pixel space, thereby showing the activation patterns contained in the feature map. In ZF-Net, the deconvnets are not used in any learning capability but just as extensions of already trained convolutional networks.

81ZF-Net proves that the shallow network learns the image edge, color and texture features, while the deep network learns abstract features of the image and points out the effective reasons and performance improvement methods of the network. At the same time, ZF-Net demonstrates that the deeper network model has better classification robustness under conditions such as image translation and rotation and the higher the level of feature maps, the stronger the feature invariance. Occlusion experiments showed that the ZF-Net model is highly sensitive to the local structure in images rather than just using the broad scene context. During network training, the low-level parameters converge quickly and the higher the level, the longer the training time is required to converge.

82Simonyan et al. designed the VGGNet and proved that increasing the depth of the network can affect the final performance of the network to a certain extent. By fixing other parameters of the architecture, a network with a very small (3 × 3) convolutional filter at all layers is used for a comprehensive evaluation of the depth-increasing network. Continuously deepening the network structure can improve performance and pushing the depth to 19 weight layers can achieve significant improvements over the existing technical configuration.

83The structure of the VGGNet is very consistent and concise. The convolutional kernel size (3 × 3) and max-pooling size (2 × 2) of the same size are used in the whole network. Layers are separated by max-pooling. All the activation units of hidden layers use the ReLU function. An improvement of VGG16 compared to AlexNet is to use several consecutive 3 × 3 convolutional kernels instead of larger convolutional kernels (5 × 5, 7 × 7, 11 × 11) in AlexNet. Specifically, in VGG, a stack of two 3 × 3 convolutional layers replaces one 5 × 5 convolutional layer and a stack of three 3 × 3 convolutional layers replaces one 7 × 7 convolutional layer.

84Since VGGNet uses a fixed 3 × 3 convolutional kernel, the increase in the number of layers brings stronger nonlinearity which makes the model’s ability better. Although the number of layers increases, the smaller convolutional kernel reduces the number of parameters in the convolutional layer, equivalent to increasing regularization compared with the large convolutional kernel. In this way, under the condition of ensuring the same perceptual field, the depth of the network is improved, and the effect of the neural network is improved to a certain extent.

85The VGGNet has three fully connected layers. According to the difference in the total number of convolutional and fully connected layers, six network structures (A, A-LRN, B, C, D, and E) are designed. The famous VGG16 (13 convolutional layers and 3 fully connected layers) and VGG19 (16 convolutional layers and 3 fully connected layers) correspond to D and E, respectively. There is no essential difference between them but the network depth is different.

86In addition to confirming that the classification error decreases as the depth of the ConvNet increases, VGGNet also confirms that deep nets with small filters are better than shallow nets with large filters and that using scale jittering to enhance the training set does help to obtain multi-scale images statistics.

87GoogLeNet is a deep learning structure proposed by Szegedy et al. in 2014. Previous structures such as AlexNet and VGG obtained better training effects by increasing the depth of the network, while GoogLeNet deepened the network (22 layers) and also innovated on the network structure by introducing the Inception structure instead of the traditional operation of simple convolution in combination with the activation function.

88Its core idea is to perform multi-scale processing and feature dimensionality reduction; a large number of convolution kernels of different sizes (1 × 1, 3 × 3, and 5 × 5) are used to perform the convolution operation so that features of different sizes are obtained and the feature extraction ability of the single-layer network is enhanced.

89GoogLeNet improves the original Inception structure by adding a suitable 1 × 1 convolutional layer before the 3 × 3 and 5 × 5 convolutional layers so that the model parameters can be reduced to a certain extent. The main architecture of GoogLeNet is a 22-layer convolutional neural network stacked with the improved Inception module. At the end of the network, the average pooling was used instead of the fully connected layer. Facts have proved that the top-1 accuracy can be increased by 0.6%. In fact, a fully connected layer was added at the end, mainly for the convenience of fine-tuning in the future. Although the full connections are removed, the network still uses dropout. In order to avoid gradient disappearance, the network additionally adds two auxiliary softmax for forward gradient transmission. During training, their loss is added to the overall loss of the network with a discounted weight. At inference time, these auxiliary networks are removed. All convolutions, including those inside the Inception modules, use rectified linear activations.

90ResNet was proposed by He et al. in 2015. It is a deep neural network formed by stacking many residual modules. Before ResNet was proposed, all neural networks comprised convolutional and pooling layers. With the superposition of the convolutional layer and the pooling layer, not only does the learning effect become better and better but there will be two problems: one is the vanishing/exploding gradients and the other is the degradation problem. As the number of layers increases, the prediction effect worsens. ResNet solves the problem of vanishing/exploding gradients through data preprocessing and the use of the batch normalization (BN) layer in the network. The residual structure is used to artificially allow some layers of the neural network to skip the connection of neurons in the next layer; the layers are connected to weaken the strong connection between each layer. This approach can solve the degradation problem in deep networks.

91The residual structure uses a shortcut connection method, skips one or more layers, and performs simple identity mapping, which neither adds additional parameters nor increases computational complexity, and the entire network can still be trained end-to-end using stochastic gradient descent (SGD) and backpropagation, as shown in Figure 13.

92In the residual network structure, assuming that the input is x, the learned feature is denoted as P(x), and the residual that can be learned is denoted as Z(x); hence, the original learned feature is P(x) = Z(x) + x. When the residual is Z(x) = 0, the stacked layer only does the identity mapping at this time and the goal of subsequent learning is to approach the residual to 0.

93The biggest difference between an ordinary direct-connected convolutional neural network and ResNet is that ResNet has many bypass branches that connect the input directly to the latter layer so that the latter layer can directly learn the residual. ResNet allows raw input information to be fed directly into the latter layers.

94DenseNet was proposed by Huang et al. in 2017, which directly connects all layers (with matching feature-maps sizes) to each other feed-forwardly. For each layer, it uses the feature maps of all previous layers as an input and its own feature maps as an input for all subsequent layers. For a network with K layers, the mth layer has m inputs, consisting of feature maps from all previous convolutional blocks. Its own feature maps are passed to all K−m subsequent layers. DenseNet contains a total of connections, which is a dense connection compared to ResNet. Moreover, DenseNet directly connects feature maps from different layers, increasing the input changes of subsequent layers, which can realize feature reuse and improve efficiency.

95The network structure of DenseNet is mainly composed of dense blocks and transition layers, as shown in Figure 14. In a dense block, the feature maps of each layer have the same size and can be connected in the channel dimension. Since the input of the latter layer will be very large, the bottleneck layer can be used inside a dense block to reduce the amount of calculation. The transition layer mainly connects two adjacent dense blocks and reduces the size of the feature map.

96The DenseNet layers are very narrow; the “collective knowledge” of the network is only added to a small part of feature maps, the rest of the feature maps are kept unchanged, and the final classifier makes a decision according to all the feature maps in the network. The DenseNet architecture clearly distinguishes between information added to the network and information retained. DenseNet requires fewer parameters than traditional convolutional networks because redundant feature maps do not need to be relearned.

97Each layer of DenseNet can directly access the gradient from the loss function and the original input signal, thereby achieving implicit deep supervision and alleviating the phenomenon of gradient disappearance, which helps to train deeper network architectures. Dense connections represent regularization effects, which reduce overfitting on tasks with small training set sizes. Notably, DenseNet can consume a lot of GPU memory if not implemented properly.

98MobileNet is an efficient small model based on a streamlined architecture and its core idea is to use depthwise separable convolutions to build lightweight deep neural networks for mobile and embedded vision applications. Two simple global hyperparameters (width and resolution multiplier) are introduced to effectively trade between latency and accuracy. These hyperparameters allow model builders to choose the right size model for their application based on the constraints of the problem.

99The basic unit of MobileNet is a depthwise separable convolution which can be decomposed into two parts: a depthwise convolution and pointwise convolution, both of which use batchnorm and ReLU nonlinearities. A depthwise convolution is different from a standard convolution. For a standard convolution, its convolution kernel is used on all input channels while a depthwise convolution uses a different convolution kernel for each input channel. In other words, one convolution check corresponds to one input channel.

100The network is uniformly thinned at each layer by a width multiplier. A resolution multiplier is applied to the input image so that the internal representation of each layer is subsequently reduced by the same multiplier. The width multiplier and resolution multiplier were introduced into MobileNet to reduce the amount of calculation and the number of parameters and to construct a model with a smaller and lower computational cost. The accuracy of MobileNetV1 is slightly lower than that of VGG16 but better than that of GoogLeNet. However, MobileNet has absolute advantages in terms of the calculation and parameter volume.

101MobileNetV2 is based on the network architecture of MobileNetV1 and introduces the inverted residual with a linear bottleneck, which significantly improves its accuracy and realizes the excellent effect of multiple image classification and detection tasks for mobile applications. MobileNetV3 combines a hardware network architecture search (NAS) and NetAdapt algorithm to tune to suit mobile phone CPUs and constructs two new MobileNet models: MobileNetV3-Large and MobileNetV3-Small, which are targeted high-resource and low-resource use cases, respectively. Among them, platform-aware NAS searches the global network structure by optimizing each network block and the NetAdapt algorithm is used to search the number of filters in each layer. Looking back at the MobileNet series, the accuracy rate is gradually improving and the delay is also decreasing.

102Zhang et al. is a neural network structure specially designed for mobile devices with very limited computing power. It mainly uses two technologies: pointwise group convolution and channel shuffle, which greatly reduce the computing overhead while retaining the model accuracy.

103In ShuffleNet, a pointwise group convolution and bottleneck structure are used to increase the number of channels without significantly increasing the floating point operations (FLOPs). However, the pointwise group convolution only uses the channel information within the group and does not use the channel correlation information between groups. In order to solve this problem, channel shuffle is introduced to realize the exchange of information between groups to improve accuracy.

104Given a computational budget, ShuffleNet can use a wider range of feature maps. In ShuffleNet, deep convolution is only performed on bottleneck feature maps to prevent overhead as much as possible. Additionally, ShuffleNet is difficult to implement efficiently on low-power mobile devices.

105Since FLOPs cannot accurately estimate the actual runtime, Ma et al. proposed four guidelines for lightweight network design: (i) the same number of input and output channels to minimize memory access costs; (ii) excessive group convolution increase memory access cost; (iii) network fragmentation reduces the degree of parallelism; and (iv) element-wise operations should be reduced. According to the four guidelines, Ma et al. introduced the “channel split” on the basis of ShuffleNet V1 and proposed ShuffleNet V2, which is effective and accurate.

106SqueezeNet maximizes the computing speed without reducing the accuracy of the model. It can achieve an effect similar to AlexNet on the ImageNet dataset but the number of parameters is 50 times less than that of AlexNet. At the same time, combined with model compression technology, the SqueezeNet model file is about 510 times smaller than AlexNet.

107The design of the SqueezeNet architecture mainly adopts three strategies: (i) replace the 3 × 3 filter with a 1 × 1 filter, and the parameters are reduced to 1/9 of the original; (ii) reduce the number of input channels to 3 × 3 filters; and (iii) own sampling is performed at the latter stage of the network so that the convolutional layers have large activation maps and the larger activation maps retain more information, which can provide higher classification accuracy. The first two strategies are about reducing the number of parameters in a CNN while trying to maintain accuracy. The third strategy is to maximize accuracy with a limited parameter budget.

108The fire module is the basic building block in SqueezeNet. A fire module consists of two parts: squeeze and expand. Among them, squeeze represents a squeeze layer in the structure of SqueezeNet, which uses a 1 × 1 convolution kernel to convolve the feature maps of the previous layer, and its main purpose is to reduce the dimension of the feature map. The inception structure used by expand consists of a set of continuous 1 × 1 convolutions and a set of continuous 3 × 3 convolutions and then spliced.

109The conventional convolution operation maps the cross-channel correlations and spatial correlations of the input feature mapping at the same time. The Inception series structure focuses on decomposing the above process and realizes the decoupling of cross-channel correlation and spatial correlation to a certain extent. The XceptionNet is improved on the basis of InceptionV3, using depthwise separate convolution instead of the traditional Inception block to achieve an entire decoupling of cross-channel correlation and spatial correlation. In addition, the XceptionNet also introduces residual connections.

110The XceptionNet architecture consists of 36 convolutional layers structured into 14 modules that constitute the feature extraction library of the network. There are linear residual connections around all modules except the first and last modules. The parameter counts for XceptionNet are similar to those for Inception V3. Compared to Inception V3, XceptionNet has slightly better classification performance on ImageNet datasets and significantly improved on JFT datasets.

111The output of many neural networks is the final classification category label, but on many occasions, such as medical image processing, medical personnel not only want to know the category of the image but also want to know the location distribution of various tissues in the image. U-net can realize the positioning of picture pixels. The network classifies each pixel in the image and the final output is a segmented image according to the category of pixels. U-net was proposed by Ronneberger et al. in 2015.

112It was originally a fully convolutional neural network designed specifically for medical image segmentation. It is now widely used as the main tool for medical image segmentation tasks and has achieved good results. U-net is small in scale compared with many other semantic segmentation networks, so it can also be used for some real-time tasks.

113The U-net architecture builds on and expands the network architecture of FCN and its architecture consists of two paths, including a contracting path for capturing context and a symmetric expanding path for precise localization. The contracting path, or the encoder or analysis path, consists of several convolutions, pooling, and down sampling of images, similar to regular convolutional networks and provides classification information. The expanding path, also known as the decoder or synthesis path, combines the high-resolution features from the contraction path with the up sampled output to restore the shape of the original image, giving a per-pixel prediction. The up sampling stage in the U-net architecture adds a lot of feature channels, allowing more original image texture information to be propagated in high-resolution layers.

114U-net has no fully connected layers and is valid for use for convolution throughout. That is, the segmentation map only contains pixels so that the segmentation results can be guaranteed to be based on no missing context features, so the input and output image sizes are not too large same. Using data augmentation with elastic deformation requires only a small number of annotated images.

115There have been many variants of U-Net, many of which still continue the core idea, adding new modules or incorporating other design concepts. Zhou et al. summarized the seven improvement mechanisms of U-Net, as shown in Table 11. The characterization and performance of common variation in U-Net are shown in Table 12 and Table 13, respectively.

116In medical imaging, datasets are difficult and expensive to build, especially for images related to cancer. Due to factors such as patient privacy and data security, sample data are usually scarce. Medical image annotation is costly and needs to be completed by experienced medical professionals. Transfer techniques have been proven to work well in the field of medical images, for example, using a pre-trained CNN architecture to extract features in an image, feeding them into a fully connected layer, and using average pooling classification to detect and classify breast cancer, Zhen et al. verified the feasibility of transfer learning of a neural network-based rectum dose-toxicity prediction model for cervical cancer radiotherapy.

117We first need to define two basic concepts to understand the definition of transfer learning. One is “domain”, which is represented by D. Domain D consists of two parts: feature space X and marginal distribution. Therefore, the domain is defined as shown in Equation (8).

118The other is “task”. We denote it by T; the task is domain-specific. That is, it makes no sense to talk about a task without the existence of a specific domain. In a specific domain, the task T consists of a label space Y and a function. is the prediction function, which can predict the label of the unknown sample X. From the perspective of probability, is approximately equal to, which is the conditional probability distribution of Y under the condition of X. Thus, we obtain the task definition of the task, as shown in Equation (9).

119Generally speaking, transfer learning includes multiple source domains to a target domain or a single source domain to a target domain. The most common case is the latter. We strictly define transfer learning, that is, given source domain and its matching task are represented by and respectively, and target domain and its relevant task are represented by and respectively, then transfer learning entails acquiring knowledge in and to help improve the learning of prediction function in.

120According to the transfer content, transfer learning can be divided into four categories:Instance-based transfer learning entails reusing part of the data in the source domain through the heavy weight method of the target domain learning;Feature-representation transfer learning aims to learn a good feature representation through the source domain, encode knowledge in the form of features, and transfer it from source domain to the target domain for improving the effect of target domain tasks. Feature-based transfer learning is based on the assumption that the target and source domains share some overlapping common features. In feature-based methods, a feature transformation strategy is usually adopted to transform each original feature into a new feature representation for knowledge transfer;Parameter-transfer learning means that the tasks of the target domain and source domain share the same model parameters or obey the same prior distribution. It is based on the assumption that individual models for related tasks should share a prior distribution of some parameters or hyperparameters. Generally, there are usually two specific ways to achieve this. One is to initialize a new model with the parameters of the source model and then fine-tune it. Secondly, the source model or some layers in the source model are solidified as feature extractors in the new model. Then an output layer is added for the target problem and learning on this basis can effectively utilize previous knowledge and reduce training costs;Relational-knowledge transfer learning involves knowledge transfer between related domains, which needs to assume that the source and target domains are similar and can share some logical relationship, and attempts to transfer the logical relationship among data from the source domain to the target domain.

121Instance-based transfer learning entails reusing part of the data in the source domain through the heavy weight method of the target domain learning;

122Feature-representation transfer learning aims to learn a good feature representation through the source domain, encode knowledge in the form of features, and transfer it from source domain to the target domain for improving the effect of target domain tasks. Feature-based transfer learning is based on the assumption that the target and source domains share some overlapping common features. In feature-based methods, a feature transformation strategy is usually adopted to transform each original feature into a new feature representation for knowledge transfer;

123Parameter-transfer learning means that the tasks of the target domain and source domain share the same model parameters or obey the same prior distribution. It is based on the assumption that individual models for related tasks should share a prior distribution of some parameters or hyperparameters. Generally, there are usually two specific ways to achieve this. One is to initialize a new model with the parameters of the source model and then fine-tune it. Secondly, the source model or some layers in the source model are solidified as feature extractors in the new model. Then an output layer is added for the target problem and learning on this basis can effectively utilize previous knowledge and reduce training costs;

124Relational-knowledge transfer learning involves knowledge transfer between related domains, which needs to assume that the source and target domains are similar and can share some logical relationship, and attempts to transfer the logical relationship among data from the source domain to the target domain.

125A brief summary and comparison of the above four types of transfer learning are shown in Table 14.

126Ensemble learning (EL) is a class of machine learning framework, not a single machine learning algorithm. It completes learning tasks by building multiple base-learners and combining them with certain strategies. The base learner is usually generated from the training data by a base learning algorithm, which can be a decision tree, a neural network, or another type of machine learning algorithm. Most ensemble methods use a single base learning algorithm to generate a homogeneous base learner. If multiple learning algorithms are used, such as both decision trees and neural networks, a heterogeneous learner is generated.

127Ensemble learning can alleviate the challenges arising from machine learning, such as class imbalance, concept drift, and the curse of dimensionality. It can be used for classification problem integration, regression problem integration, feature selection integration, outlier detection integration, and many other occasions. The overall prediction performance of the integration is better than that of a single basic learner.

128The construction methods of ensemble learning can be divided into two categories: parallel method and serial method. The parallel method refers to the construction of multiple independent learners and taking the average of their predicted results. There is no strong dependence between individual learners and a series of individual learners can be generated in parallel, usually homogeneous weak learners. The representative algorithms are bagging and random forest series algorithms. The serial approach means that multiple learners are built sequentially and there is a strong dependency between individual learners because a series of individual learners need to be generated sequentially, usually heterogeneous learners. Boosting series are representative algorithms, such as AdaBoost, gradient lifting, etc.

129Ensemble learning techniques have been proven to be an effective way to improve the robustness and performance of any medical image classification pipeline. They are also used in cancer diagnosis and prognosis. Brunese et al. proposed a method for identifying the components of an ensemble learner. The ensemble learner was used for the graded detection of brain cancer from magnetic resonance images, with 99% accuracy in detecting benign grade I and malignant grade II, III, and IV brain cancers. Abdar et al. constructed a model called CWV-BANNSVM based on the confidence weighted voting method and the boosting ensemble technique to achieve breast cancer detection with 100% accuracy. Using ensemble learning techniques not only improves the accuracy of cancer detection but also extracts interpretable diagnostic rules. In, an ensemble learning model was constructed for the detection of prostate cancer using the stacking technology of random forest classifiers, achieving both accuracy and interpretability.

130Graph neural network (GNN) is a framework for directly learning graph-structured data using deep learning. Each node representation is calculated by propagating and aggregating features of adjacent nodes and GNN can capture long-term dependencies between data on graphs. GNN can perform prediction or classification tasks at the level of the entire graph or for each node or edge.

131GNN can improve the effectiveness of survival prediction for cancer patients. Gao et al. designed a multimodal graph neural network (MGNN) framework to predict cancer survival where multimodal data include clinical data, gene expression, and copy number alteration. MGNN was verified as having strong robustness on four cancer datasets (metastatic colorectal cancer, breast invasive carcinoma, pan-lung cancer, and pediatric acute lymphoid leukemia—Phase II).

132In recent years, variants of GNN such as graph convolutional network (GCN), graph attention network (GAT), gated graph neural network (GGNN), graph autoencoder, and graph recurrent network (GRN) have demonstrated excellent performance in many cancer diagnoses.

133Interpretability is important for assessing the molecular origin of cancer-related genes, detecting potential artifacts, and increasing trust in modeling methods. GCN can classify unlabeled nodes in the network according to their associated feature vectors and the topology of the network, integrating graph-based data and feature vectors in a natural way. Schulte-Sasse et al. used GCN to develop a machine learning approach known as explainable multi-omics graph integration (EMOGI) to prioritize cancer genes from large datasets. Through their approach, multidimensional multiomics node features, as well as topological features of protein–protein interaction networks, can identify not only highly mutated cancer genes but also genes that contain other types of alterations or interact with other cancer genes. The study fed genomic data from 16 cancer types into the method, identifying 165 new candidate genes that could cause cancer. omicsGAT was introduced for cancer subtype analyses. This approach integrates the graph-based learning and attention mechanism into the mRNA sequencing (RNA-seq) data analysis. The multi-head attention mechanism in omicsGAT can protect the information of specific samples more effectively by assigning different attention coefficients to adjacent samples.

134Explainability refers to “the degree to which an observer can understand the cause of a decision”. Although applying deep learning to cancer detection has achieved remarkable results, neural networks are often regarded as a powerful “black box” with weak explainability. Neural networks are often found to have some unexpected errors in practical use, and explainability may help to discover potential errors and improve the model. In addition, explainability is required based on ethical and legal issues. Especially in the medical field, the explainability of deep learning models is crucial to creating safe and responsible systems suitable for clinical implementation. Explainable artificial intelligence can give an interpretable prediction result based on input data symptoms or medical images to assist doctors in diagnosis.

135Many interpretability classification methods exist, such as global interpretation vs. local interpretation, Ad-Hoc vs. Post-Hoc, etc. The classifications of interpretability and their characteristics are shown in Table 15.

136Explainable deep neural networks (xDNN) constitute a deep learning architecture that synergizes both an inference and reasoning. It is non-iterative and non-parametric which explains its efficiency in terms of time and computational resources. xDNN is based on prototypes, which are actual training data samples (images) with local peaks of the empirical data distribution called typicality as well as of the data density. This generative model is identified in a closed form equivalent to the probability distribution function but derived from the training data fully automatically, without user- or problem-specific thresholds, parameters, or intervention. The numerical Grad-CAM (numGrad-CAM) used in a CNN model can provide an explainability interface for the diagnosis of brain tumors. Windisch et al. trained a neural network to distinguish MRI slices containing vestibular schwannomas, glioblastoma, or no tumor, making the decision of the network easier to interpret, helping to identify potential biases, and selecting appropriate training data.

137Transformers can develop rapidly in the field of computer vision, mainly due to the following reasons: (i) a strong ability to learn long-distance dependencies. Transformers come with a long dependency feature and use the attention mechanism to capture global context information and extract more powerful features. (ii) Strong multi-modal fusion ability. The input of the transformer does not need to maintain a two-dimensional image. Usually, the initial embedding vector can be obtained by directly operating on the pixels and the information of other modalities can be directly fused at the input end after being converted into a vector. (iii) The model is more explainable. In multi-head attention structure of the transformer, each head applies an independent self-attention mechanism which enables the model to learn relevant information in different representation subspaces for different tasks.

138A vision transformer (ViT) migrates the transformer model to computer vision tasks, constituting an image classification method based entirely on the self-attention mechanism. It follows the original design of the transformer as closely as possible. Due to the sequential nature of the video, transformers perform comparably to traditional CNNs and RNNs on video tasks.

139Since the input of the transformer structure is a two-dimensional matrix, the shape of the matrix can be expressed as (N, D), where N is the length of the sequence and D is the dimension of each vector in the sequence. Therefore, in the ViT algorithm, it is first necessary to try to convert the image of H × W × C into the input (N, D), where (H, W) is the resolution of the original image and C is the number of channels. The specific implementation method is:The image of H × W × C is changed into a sequence of N × (P2 × C), where P is the size of the image block. This sequence can be viewed as a series of flattened image patches. That is, the image is cut into small patches and then flattened. The sequence contains a total of N = H × W/P2 image patches and the dimension of each image patch is (P2 × C). After the above transformation, N can be regarded as the length of the sequence;Since the dimension of each image patch is (P2 × C) and the vector dimension we actually need is D, we also need to Embed the image patch. That is, each image patch will be linearly transformed and the dimension will be compressed to D.

140The image of H × W × C is changed into a sequence of N × (P2 × C), where P is the size of the image block. This sequence can be viewed as a series of flattened image patches. That is, the image is cut into small patches and then flattened. The sequence contains a total of N = H × W/P2 image patches and the dimension of each image patch is (P2 × C). After the above transformation, N can be regarded as the length of the sequence;

141Since the dimension of each image patch is (P2 × C) and the vector dimension we actually need is D, we also need to Embed the image patch. That is, each image patch will be linearly transformed and the dimension will be compressed to D.

142In recent years, the application of a vision transformer in the medical field has attracted the attention of many researchers and it has also been successfully applied to cancer classification based on medical images, including skin cancer, prostate cancer, colorectal cancer, and breast cancer, etc.

143Based on the traditional vision transformer, a series of ViT improvement methods were proposed for the performance of vision tasks. For example, aiming at the problem that the traditional ViT model does not pay enough attention to the local details of medical images, Zhou et al. designed an adaptive sparse interaction ResNet-ViT dual-branch network (ASI-DBNet) to improve the classification accuracy of brain cancer. The ResNet-ViT parallel structure is designed to simultaneously capture and preserve the local and global information of pathological images and an adaptive sparse interaction block (ASIB) is used to realize the interaction between the ResNet branch and the ViT branch. In, a multi-scale visual transformer model for gastric cancer histopathological image detection was constructed which realizes the global automatic detection of gastric cancer images. In this model, the position-encoded transformer model and the convolutional neural network with local convolution are two key modules used to extract global and local information, respectively. In the rapid on-site evaluation (ROSE) technique for the diagnosis of pancreatic cancer, the shuffle instance-based visual transformer (SI-ViT) method was designed to reduce the perturbation between instances and enhance the modeling ability between instances.

144In the process of neural network training, because the network parameters are constantly changing according to the gradient descent, the distribution of data in each network layer will change differently. That is, the phenomenon of internal covariate shift appears. Internal covariate shift will cause the input data distribution to shift in a certain direction, resulting in slow network data training. At this time, batch normalization (BN) was introduced to solve the above problems. It is a technique to normalize the activation in the middle layer of the deep neural network, which can greatly improve the speed and accuracy of model training as well as improve network generalization performance.

145Consider a mini-batch of size k,. The batch normalization process is shown in Equations (10)–(13).

146Here, denotes the mini-batch mean, denotes mini-batch variance, k denotes the mini-batch size, and denotes the normalization process. is a small constant for numerical stability.

147In order to maintain the nonlinearity, the parameters and are added into the batch normalization, which is called the SCALE factor and SHIFT factor, respectively. The two parameters are used to scale and shift the input, which is conducive to the coordination of distribution and weight and enhances the expression ability of the network.

148The highlight of BN is the normalized activation, which can be applied to any set of activations in the network and integration of this normalization into the network architecture itself. By normalizing activations across the network, it prevents small changes in layer parameters from being amplified as data propagate through the deep network. Normalizing the data can concentrate the data distribution in a range with a large gradient, speed up the convergence of the model, and allow a larger learning rate to be used.

149Dropout refers to the random dropping of units in a neural network, including both hidden and visible units. The core idea of dropout is randomly dropping units and their connections during deep learning network training which thins the network to prevent complex co-adaptation to the training data, thereby reducing overfitting.

150Figure 15a is a standard fully connected neural network. Figure 15b results from applying dropout to (a). It randomly drops some neurons at a certain probability.

151Notably, deleting a unit only temporarily removes it from the network along with all its incoming and outgoing connections. In general, each unit usually retains a fixed probability, p, independently of others, where p can be chosen using a validation set or simply set to 0.5. For input units, the optimal retention probability is usually closer to 1.

152With the need for practical applications, dropout variants have emerged one after another. DropConnect introduced by Wan et al. does not apply dropout directly on neurons but on the weights and biases connecting these neurons. Instead of setting the neuron’s output to 0, the weight or bias of each neuron is set to 0 with a certain probability. Standout, introduced by Ba et al., is a dropout method that tries to improve the standard dropout by adaptively selecting neurons to be dropped according to the value of the weight. Methods to explain dropout from a Bayesian perspective, such as fast dropout and variational dropout.

153The essence of deep learning is to optimize the value of all weights to achieve an optimal solution state. Among them, the layers that need to update the weights include convolutional layers, BN layers, and FC layers. In optimization, weight initialization is important to obtain the optimal solution.

154After weight initialization, the neural network can iteratively update the weight parameter to achieve better performance. In deep learning, the weight initialization method of neural networks has a crucial impact on the convergence speed and performance of the model. If the weight initialization is not appropriate, the model may fall into the local-optimal solution, resulting in the model prediction effect being unideal and even making the loss function oscillate and the model not converge. The most common methods of weight initialization are shown in Table 16.

155The difficulty in obtaining medical images has always limited it as well as the small amount of data and the high cost of image labeling, which lead to the medical image data being quickly overfitted during the model training process and unable to achieve the best diagnostic results. Data augmentation is a technique that expands the training data by using algorithms to augment the size and quality of the training dataset.

156Data augmentation algorithms in computational vision can be roughly divided into two categories: data augmentation based on basic image processing techniques and data augmentation based on deep learning. Data augmentation methods based on basic image processing techniques mainly include geometric transformations, color space, noise injection, kernel filters, mixing images, random erasing, combination, etc., as shown in Table 17. Data augmentation algorithms based on deep learning mainly include feature space augmentation, adversarial training, GAN-based data augmentation, neural style transfer, meta-learning data augmentations, etc., as shown in Table 18. In Table 18, the data augmentation method based on basic image processing technology is denoted as M and the data augmentation algorithm based on deep learning is denoted as D.

157A cancer diagnosis is a classification problem whose nature requires very high classification accuracy. In recent years, deep learning theory has broken through the bottleneck of manual feature selection and has made breakthrough progress in image classification, improving the accuracy of medical image classification and recognition and improving the generalization and promotion capabilities of its applications. CNN is the most successful architecture among deep learning methods. In 1995, Lo et al. applied CNN to medical image analysis.

158Fu’adah et al. designed a model that can automatically identify skin cancer and benign tumor lesions using CNN. The model used several optimizers, including RMSprop, Adam, SGD, and Nadam, with a learning rate of 0.001. Among them, the Adam optimizer can identify skin lesions in the ISIC dataset into four categories with an accuracy rate of 99%. DBN was also introduced into the diagnosis of cancer. The detection performance of the DBN-based CAD system for breast cancer is significantly better than that of the traditional CAD system. Anand et al. developed a convolutional extreme learning machine (DC-ELM) algorithm for the assessment of cancer of the knee bone based on histopathological images. The test results showed an accuracy rate of 97.27%. The multi-classifier system based on a deep belief network designed by Beevi et al. can accurately detect mitotic cells, which provides better performance for important diagnosis of breast cancer grading and prognosis.

159Shahweli applied an enhancer deep belief network constructed with two restricted Boltzmann machines to identify lung cancer with an accuracy of 96%. For brain tumor detection based on MRI, Kumar et al. used group search-based multi-verse optimization (GS-MVO) to reduce the feature length and handed over the optimally selected features to the DBN to achieve higher classification accuracy. Abdel-Zaher et al. proposed an automatic diagnostic system for detecting breast cancer. The system is based on the DBN unsupervised pre-training stage and then a back-propagation neural network stage with supervision. A pretrained back-propagation neural network with an unsupervised stage DBN achieves higher classification accuracy than a classifier with only one supervised stage. In breast cancer cases, the overall neural network accuracy was increased to 99.68%, the sensitivity was 100%, and the specificity was 99.47%.

160Jeyaraj et al. developed an unsupervised generative model based on the DBM for the classification of patterns of regions of interest in complex hyperspectral medical images; the classification accuracy and the success rate are superior to the traditional convolution network. Nawaz et al. designed a multi-class breast cancer classification method based on the CNN model, which can not only classify breast tumors into benign or malignant but can also predict the subclasses of tumors, such as fibroadenoma, lobular carcinoma, etc. Jabeen et al. proposed a breast cancer classification framework from ultrasound images using deep learning and the best-selected features fusion. The experiments were performed on the augmented Breast Ultrasound Image (BUSI) dataset with the best accuracy of 99.1%.

161El-Ghany et al. proposed a fine-tuned learning model based on a pretrained ResNet101 network for the diagnosis of multi-types of cancer lesions. A benchmark cancer lesion dataset of over 25,000 histopathology images trained with transfer learning can classify photos of colon and lung cancer into five categories: adenocarcinomas lung, squamous cell carcinomas, benign lung, benign colon, and adenocarcinomas colon. The application results of deep learning in medical image classification are shown in Table 19.

162The goal of image detection is to determine where objects are located in a given image (usually with a bounding box) and which class each object belongs to. That is, it includes the two tasks of correct localization and accurate classification. In recent years, object detection algorithms based on deep learning have mainly formed two categories: candidate region-based algorithms and regression-based algorithms. The object detection algorithm based on the candidate region is also called the two-stage method, which divides object detection into two stages: one is to generate a candidate region and the other is to put the candidate region into the classifier for classification and detection.

163The object detection algorithm based on candidate regions can obtain richer features and a higher accuracy but the detection speed is relatively slow. Typical algorithms based on candidate regions include R-CNN series, such as R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, etc. In 2014, R-CNN was proposed by Girshick et al.. Although R-CNN has achieved great success in the field of object detection, there are problems, such as the need to train multiple models and a large number of repeated calculations, which leads to the low detection efficiency of R-CNN. In 2015, the team further improved the R-CNN method and proposed the Fast R-CNN model. Fast R-CNN only extracts features from the input image once during the detection process, thereby avoiding the waste of computing resources in feature extraction. However, it still uses selective search to generate proposals, which will also limit the detection speed. In 2015, Ren et al. proposed the Region Proposal Network (RPN) and combined it with Fast R-CNN, which is called the Faster R-CNN network.

164In the research on automatic detection and classification of oral cancer, Welikala et al. used ResNet-101 for object classification and Faster R-CNN for object detection. Image classification achieved an F1 score of 87.07% in identifying images containing lesions and 78.30% in identifying images requiring referral. Mask R-CNN is one of the most popular object detection algorithms at present. It abandons the process of pre-determining the region of interest in the traditional two-stage detection algorithm and can directly process the image and then perform network detection on the target area. Therefore, the algorithm can greatly reduce the detection time, and at the same time, the accuracy rate is also greatly improved.

165In, Mask R-CNN was used to detect gastric cancer pathological sections, segment cancer fossa, and optimize by adjusting parameters. This method finally made it obtained the test result with an AP value of 61.2 when detecting medical images. In, a Mask R-CNN is used to search the entire set of images and detect suspicious lesions, which allows the entire image without prior breast segmentation search and achieves an accuracy of 0.86 on a per-slice basis analysis in the training dataset.

166The regression-based object detection algorithm has only one stage and directly regresses the category and coordinates of the target on the input image, and the calculation speed is relatively fast. Typical regression-based algorithms mainly include YOLO series, SSD series, etc.

167Due to a large number of medical image data modalities and large changes in lesion scales, scholars often improve object detection algorithms in the field of natural images to make them suitable for lesion detection in the field of medical images. Gao et al. used the LSTM model and long-distance LSTM (DLSTM) to collect the time changes of longitudinal data in CT detection of lung cancer. The temporal emphasis model (TEM) in DLSTM supports learning at regular and irregular sampling intervals. Three-dimensional CNN was used to diagnose breast cancer in a weakly supervised manner and locate lesions in dynamic contrast enhancement (DCE) MRI data, showing high accuracy with an overall Dice distance of 0.501 ± 0.274. Studies have shown that the weakly supervised learning method can locate lesions in volumetric radiological images with image-level labels only.

168Asuntha et al. proposed a CNN method based on the Fuzzy Particle Swarm Optimization (FPSO) algorithm (FPSOCNN) to detect and classify lung cancer. This method greatly reduces the computational complexity of CNN. Shen et al. proposed a globally-aware multiple instance classifier (GMIC) to localize malignant lesions in a weakly supervised manner. The model is capable of processing medical images at the original resolution and is able to generate pixel-level saliency maps, which provide additional interpretability. Applying the model to screening mammography classification and localization requires less memory and is faster to train than ResNet-34 and faster than RCNN.

169Ranjbarzadeh et al. proposed C-ConvNet/C-CNN to simultaneously mine local features and global features through two different paths and introduced a new distance attention (DWA) mechanism to consider the influence of tumor center location and brain within the model. In, a center-point matching detection network (SCPM-Net) based on 3D sphere representation was proposed, which consisted of two parts: sphere representation and center points matching. The model automatically predicts nodule radius, position, and offsets of nodules without manually designing nodule/anchor parameters. In, a two-stage model for breast cancer detection using thermographic images was proposed. In the first stage, VGG16 is used to extract features from the image. In the second stage, the Dragonfly Algorithm (DA) based on the Grunwald-Letnikov (GL) method is used to select the optimal feature subset. The evaluation results show that the model reduces 82% of the features compared with the VGG16 model.

170In, a dense dual-task network (DDTNet) was used to simultaneously realize the automatic detection and segmentation of tumor-infiltrating lymphocytes (TILs) in histopathological images. A semi-automatic method (TILAnno) was used to generate high-quality boundary annotations for TILs in H- and E-stained histopathology images. Maqsood et al. proposed the transferable texture convolutional neural network (TTCNN), which only includes three layers of convolution and one energy layer. The network integrates the energy layer to extract texture features from the convolutional layer. The convolutional sparse image decomposition method is used to fuse all the extracted feature vectors, and finally, the entropy-controlled firefly method is used to select the best features. The application results of deep learning in medical image detection are shown in Table 20.

171Image segmentation refers to dividing an entire image into a series of regions which can be considered a problem of pixel-level classification. Accurate medical image segmentation can assist doctors in judging the condition, quantitatively analyzing the lesion area, and providing a reliable basis for correct disease diagnosis. Image segmentation tasks can be divided into two categories: semantic segmentation and instance segmentation. Semantic segmentation assigns a class to each pixel in an image but objects within the same class are not differentiated. Instance segmentation only classifies specific objects. This looks similar to target detection. The difference is that target detection outputs the boundary box and category of the target and instance segmentation outputs the mask and category of the target.

172The early traditional medical image segmentation methods mainly include boundary ex-traction, region-based segmentation, threshold-based segmentation, etc.. Among them, NormalizedCut, GraphCut, and GrabCut are the most commonly used segmentation techniques. With the progress of deep learning networks, a new generation of image segmentation models, such as FCN, U-Net, and their variants, were produced and their segmentation performance was significantly improved.

173The image segmentation method based on deep learning establishes the mapping of pixels and the pixels in a certain range to instances or categories through the known sample data. This kind of method uses the powerful nonlinear fitting ability of deep learning and uses a large number of sample data to participate in training. The mapping model established by this method has high accuracy.

174FCN is applied to the cancer region segmentation system to fully extract the feature information of different scales in the input image and has a better segmentation effect than the network that does not introduce information of different scales. Recurrent Fully Convolutional Networks (RFCNs) are used to directly solve the segmentation problem in multi-slice MR images.

175Wang et al. designed a fully convolutional network which applied multi-parametric magnetic resonance images to detect prostate cancer, including a stage of prostate segmentation and a stage of tumor detection. Dong et al. applied a hybrid fully convolutional neural network (HFCNN) to segment the liver and detect liver metastases. In the system, the loss function is evaluated on the whole image segmentation object. The network processes full images rather than patches, merging different scales by adding links that blend the last detection with lower layers with finer metrics, producing lesion heat maps. Shukla et al. proposed a classification and detection method for liver cancer, which used cascaded fully convolutional neural networks (CFCNs) to train using the input of segmented tumor regions. On 3DIRCAD datasets of different volumes, the total accuracy of the training and testing process is 93.85%, which can minimize the error rate.

176The U-Net model is an improvement and extension of FCN. It follows the idea of FCN for image semantic segmentation, that is, using a convolutional layer and pooling layer for feature extraction and then using the deconvolution layer to restore image size. Experiments have proved that the U-Net model can obtain more accurate classification results with fewer training samples. U-Net was developed specifically for medical image data and did not require many annotated images. In addition, due to the existence of high-performance GPU computing, networks with more layers can be trained, and the emergence of U-Net in the field of medical image segmentation has been increasing in recent years.

177Ayalew et al. designed a method based on the U-Net architecture to segment the liver and tumors from abdominal CT scan images. The number of filters and network layers of the original U-Net model were modified to reduce the network complexity and improve segmentation performance. Dice scores using this algorithm were 0.96 for liver segmentation, 0.74 for the segmentation of tumors from the liver, and 0.63 for the segmentation of tumors from abdominal CT scan images. However, it still faces great challenges in segmenting small and irregular tumors. The application results of deep learning in medical image segmentation are shown in Table 21.

178Medical image registration refers to seeking a kind of (or a series of) spatial transformations for a medical image to make it consistent with the corresponding points on another medical image. This consistency means that the same anatomical point on the human body has the same spatial position in the two matching images. Using the correct image registration method can accurately fuse a variety of information into the same image, making it easier and more accurate for doctors to observe lesions and structures from various angles. At the same time, through the registration of dynamic images collected at different times, the changes in lesions and organs can be quantitatively analyzed, making a medical diagnosis, surgical planning, and radiotherapy planning more accurate and reliable.

179To solve the problem of missing data caused by tumor resection, Wodzinski et al. proposed a nonrigid image registration method based on an improved U-Net architecture for breast tumor bed localization. The algorithm works simultaneously at several image resolutions to handle large deformations and a specialized volume penalty is proposed to incorporate medical knowledge of tumor resection into the registration process.

180In order to realize the prediction of organ-at-risk (OAR) segmentation on cone-beam CT (CBCT) from the segmentation on planning CT, Han et al. proposed a CT-to-CBCT deformable registration model, which can enable accurate deformation registration CT and CBCT images of pancreatic cancer patients treated with high biological radiation doses. The model uses regularity loss, image similarity loss, and OAR segmentation similarity loss to penalize the mismatch between warped CT segmentation and manually drawn CBCT segmentation. Compared with intensity-based algorithms, this registration model not only improves segmentation accuracy but also reduces the processing time by an order of magnitude. In, a novel pipeline was proposed to achieve accurate registration from 2D US to 3D CT/MR. This registration pipeline starts with a classification network for coarse orientation estimation, followed by a segmentation network for predicting ground-truth planes in 3D volumes, enabling fully automated slice-to-volume registration in one shot. In, a method for the deformation simulation of inter-fraction in high-dose rate brachytherapy was proposed, which is applied to the deformable image registration (DIR) algorithm based on deep learning, which can directly realize the inter-fraction image alignment of HDR sessions for inter-fraction high dose rate brachytherapy in cervical cancer.

181In, a CBCT–CBCT deformable image registration was proposed for radiotherapy of abdominal cancer patients. It is based on unsupervised deep learning and its registration workflow includes training and reasoning stages, which share the same feedforward path through a spatial transformation-based network (STN). STNS consists of global generative adversarial networks (GlobalGAN) and local GAN (LocalGAN), which predict coarse and fine-scale motions, respectively. Xie et al. introduced point metric and masking techniques and proposed an improved B-splines-based DIR method to address the large deformation and non-correspondence due to tumor resection and clip insertion to improve registration accuracy. The point metric minimizes the distance between two point sets with known correspondences for regularization of intensity-based B-spline registration. Masking techniques reduce the impact of non-corresponding regions in breast computed tomography (CT) images. This method can be applied to the determination of the target body in the radiotherapy treatment planning after breast cancer surgery. In, LungRegNet was proposed for unsupervised deep learning-based deformable image registration (DIR) of 4D-CT lung images. It consists of two subnetworks, CoarseNet and FineNet, which predict lung motion on coarse-scale images and local lung motion on fine-scale images, respectively. The method showed excellent robustness, registration accuracy, and computational speed, with a mean and standard deviation of Target Registration Error (TRE) of 1.00±0.53 mm and 1.59±1.58 mm, respectively. In order to maintain the original topology during deformation to enhance image registration performance, a cycle-consistent deformable image registration called CycleMorph was proposed by. The model can be applied to both 2D and 3D registration and can be easily extended to a multi-scale to solve the memory problem in large-volume registration. The application results of deep learning in medical image registration are shown in Table 22.

182Image reconstruction is a method to reconstruct an image based on the data obtained from object detection. Due to the limitations of the physical imaging system, it is difficult for some medical equipment to obtain real medical images and doctors cannot clearly see the specific conditions of lesions in the images, resulting in misdiagnosis, which is not conducive to accurate diagnosis and treatment. Therefore, under the existing hardware conditions, image reconstruction technology can break through the inherent limitations of hardware, improve the quality of medical images, and reduce operating costs but also provide medical personnel with clear images and further improve the accurate diagnosis of diseases. Technology based on deep learning improves the speed, accuracy, and robustness of medical image reconstruction.

183Kim et al. used conventional methods and deep learning-based imaging reconstruction (DLR) with two different noise reduction factors (MRIDLR30 and MRIDLR50) to reconstruct axial T2WI in patients who underwent long-term rectal cancer chemoradiotherapy (CRT) and high-resolution rectal MRI and measured the tumor signal-to-noise ratio (SNR). The results showed that the MR images produced by DLR had a higher resolution and signal-to-noise ratio and that the specificity of identifying pathological complete responses (pCR) was significantly higher than that of conventional MRI. In, their results confirmed that the DLIR algorithm for pancreatic protocol dual-energy computed tomography (DECT) significantly improved image quality and reduced the variability of iodine concentration (IC) values compared with hybrid infrared. Kuanar et al. proposed a GAN-based autoencoder network capable of denoising low-dose CT images, which first maps CT images to a low-dimensional manifold and then images are recovered from their corresponding manifold representations.

184A major disadvantage of MRI is the long examination time. The deep learning image reconstruction (DLR) method was introduced and achieved good results in scanning acceleration. In the diagnosis of prostate cancer using multiparametric magnetic resonance imaging (mpMRI), Gassenmaier et al. proposed an accelerated deep learning image reconstruction T2-weighted turbo spin echo (TSE) sequence, which reduced the acquisition time by more than 60%. In order to solve the problem that traditional optical image reconstruction methods based on the finite element method (FEM) are time-consuming and cannot fully restore the lesion contrast, Deng et al. proposed FDU-Net, which consists of a fully connected subnetwork, a convolutional encoder–decoder subnetwork, and a U-Net. Among them, the U- Net is used for fast and end-to-end reconstruction of 3D diffuse optical tomography (DOT) images. Training is performed on digital phantoms consisting of randomly located singular spherical inclusions of various sizes and contrasts. The results show that after training, the ability of the FDU-Net to recover the real inclusion contrast and location without using any inclusion information in the reconstruction process is greatly improved and the FDU-Net trained on simulated data can be successfully measured from real patients. Breast tumors were reconstructed from the data.

185Feng et al. developed a deep learning-based algorithm (Z-Net) for MRI-guided non-invasive near-infrared spectral tomography (NIRST) reconstruction, which was the first algorithm to use DL for combined multimodality image reconstruction and contributed to better detection of breast cancer. The method avoids MRI image segmentation and light propagation modeling and can simultaneously recover chromophor concentrations of oxy-hemoglobin (HbO), deoxy-hemoglobin (Hb), and water through end-to-end training. Trained neural networks with only simulated datasets can be directly used to distinguish between malignant and benign breast tumors. Wei et al. proposed a real-time 3D MRI reconstruction method from cine-MRI based on unsupervised network for radiation therapy for thoracic and abdominal tumors in MRI-guided radiotherapy for liver cancer. In this method, reference 3D-MRI and cinema-mri are used to generate training data. The application results of deep learning in medical image reconstruction are shown in Table 23.

186Medical image fusion can be divided into intra-modality and inter-modality image synthesis. Inter inter-modality image synthesis refers to the synthesis of images between two different imaging modalities, such as MR-to-CT, CT-to-MR, PET-to-CT, etc. On the other hand, intra-modality synthesis refers to converting images between two different protocols of the same imaging mode, such as between MRI sequences or restoring images from low-quality protocols to high-quality ones. Deep learning-based methods show higher performance in medical synthetic image accuracy than traditional methods. Deep learning models can more effectively map the correlation between input and output in the design of image transformation and synthesis rules and construct high-level features (such as shapes and objects) by learning low-level features (such as edges and textures) for medical image synthesis with supervised or unsupervised network architectures. Among these architectures, the representative methods include GAN and U-net.

187In, a generative model was used to generate new images to reduce dataset imbalances to improve the performance of the automatic gastric cancer detection system. The synthetic network can also generate realistic images even when the lesion image dataset is small. This method allows lesion patches to be attached to various parts of normal images. The experimental results show that the dataset bias is reduced but when the number of synthetic images input to the training dataset is changed, the performance of the model changes. When 20,000 synthetic images were added, the model achieved the highest AP score and when more images were added, the performance of the model decreased. Yu et al. proposed a three-dimensional conditional generation adversarial network (cGAN) and local adaptive synthesis scheme to synthesize fluid-attenuated inversion recovery (FLAIR) images from T1 so as to effectively deal with the single modal brain tumor segmentation based on T1.

188Saha et al. proposed TilGAN, an efficient generative adversarial network, to generate high-quality synthetic pathological images of tumor-infiltrating lymphocytes (TILs) and then classify TIL and non-TIL regions. The TilGAN-generated image obtained higher Inception scores and lower initial kernel distances as well as Fréchet Inception distances than the real image. In the classification and diagnosis of skin cancer, Abhishek et al. proposed a conditional GAN-based Mask2Lesion model, which is trained with segmentation masks available in the training dataset and is used to generate new lesion images with any arbitrary mask, augmenting the original training dataset.

189Qin et al. proposed a style-based generation adversarial network (GAN) model to generate skin lesion images in order to solve the problem of a lack of labeled data and an imbalance in dataset categories. This model applies the data augmentation technique based on GANs to the classification of skin lesions and improves the classification performance. Baydoun et al. proposed the sU-cGAN model for MR–CT image translation for cervical cancer diagnosis and treatment. In this model, a shallow U-Net (sU-Net) with an encoder/decoder depth of 2 is used as the generator of a conditional GAN (cGAN) network. The trainable parameters of sU-cGAN are less than those of ordinary cGAN networks.

190Zhang et al. used a conditional generative adversarial network to generate synthetic computed tomography images of head and neck cancer patients from CBCT, while maintaining the same cone-beam computed tomography anatomy with accurate computed tomography numbers. The proposed method overcomes the limitations of cone beam computed tomography (CBCT) imaging artifacts and Hounsfield unit imprecision. Sun et al. proposed double U-Net CycleGAN for 3D MR to CT image synthesis. This method solves the problem that GAN and its variants often lead to spatial inconsistencies in contiguous slices when applied to medical image synthesis. Experimental results show that this method can realize the conversion of MRI images to CT images by using unordered paired data and synthesizing better 3D CT images with less computation and memory. Chen et al. used U-net to generate synthetic CT images from conventional MR images in seconds for intensity-modulated radiation therapy treatment planning for the prostate.

191Bahrami et al. designed an efficient convolutional neural network (eCNN) for generating accurate synthetic computed tomography (sCT) images from MRI. The network is based on the encoder–decoder network in the U-net model, without softmax layers and the residual network, and the eCNN model shows effective learning ability using only 12 training objects. Synthetic MRI (SyMRI) technology can generate various image contrasts (including fluid-attenuated inversion recovery images, T1WIs, T2-weighted images, and double inversion recovery sequences) and adjust subtle parameter values with information from a single scan to improve the visualization of lesions, which can also quantify T1, T2, and proton density (PD). In, studies have shown that synthetic MRI variables can be used to quantitatively assess bone lesions in the lower trunk of prostate cancer patients and that PD values can be used to determine the viability of prostate cancer bone metastases. Trans-rectal ultrasound (TRUS) images are one of the effective methods to visualize prostate structures. Pang et al. proposed a method to generate TRUS-style images from prostate MR images for prostate cancer brachytherapy. The application results of deep learning in medical image synthesis are shown in Table 24.

192Although deep learning has made many breakthroughs in cancer diagnosis based on medical images, there are still some problems or difficulties in applying deep learning in clinical precision diagnosis and personalized treatment. In this section, we will discuss this topic.

193Existing deep learning models have high requirements on the number and quality of images. However, compared with other imaging fields, medical image data are difficult to obtain, especially annotation data, which is reflected in the high cost, cumbersome, and time-consuming annotation; some medical image acquisition methods may even cause physical harm to patients and large amounts of data cannot be naturally obtained. Through literature research, we found that when the deep learning model is applied to the medical image-assisted diagnosis of cancer, the training dataset is relatively small, and the current clinical diagnosis methods entail supervised learning, so insufficient data will inevitably affect the accuracy and stability of diagnosis and prediction. In response to this problem, researchers have designed various solutions, including using data augmentation technology and image fusion to enrich the number of samples, improving the deep learning model, and using the ideas of weak supervision, transfer learning, and multi-task learning to improve the accuracy of classification prediction, and so on, which will be the focus of continuous research.

194It is also worth mentioning that GAN-based data augmentation techniques mainly generate medical images from noisy data. Data transformation, oversampling, and adversarial training can be used in combination or multiple data augmentation techniques can be used simultaneously to generate the desired data from the original images. However, overly extensive augmented data can lead to overfitting of the model, potentially worse than before augmentation. In, through experimental methods on three different datasets, it was shown that the effectiveness of GANs as a source of medical imaging data is not always reliable, even if the generated images are almost indistinguishable from real data. It is good practice to monitor overfitting during incremental augmentation and define a maximum level of data augmentation based on maximum training accuracy and minimum loss.

195In the clinic, some types of cancer are very common (e.g., adenocarcinoma and ductal carcinoma) while others are less common (e.g., squamous cell carcinoma), which will result in a dataset with an extreme degree of class imbalance and the performance of the classifier is affected. Models trained on such datasets will be biased towards the most prevalent majority class.

196Studies have shown that the use of generating new samples and class-specialized ensemble technique can balance datasets and improve the classification performance of rare cancer types. In, a regularized deep-learning ensemble framework is used to learn from imbalanced multiclass datasets for cancer detection. proposed a deep neural network architecture called VGGIN-Net that includes transfer learning methods, adding batch normalization, flatten, dropout, and dense layers to the network structure, which helps for transferring domain knowledge from a larger ImageNet object dataset to a smaller imbalanced breast cancer dataset. However, the class imbalance in the context of deep learning has not been fully studied and is still a direction for future research.

197Various imaging modes such as CT, PET, MRI, and US have their own advantages and disadvantages. In order to obtain better image effects and bring more valuable diagnostic results to the clinic, fusion imaging technology is used to reorganize and superimpose resources of different imaging modes at the same time, that is, exploit the strengths of all imaging modalities and eliminate or minimize the weaknesses of each mode. A single image generated by fused imaging techniques is more informative and accurate than any single source image and the purpose of image fusion is not only to reduce the amount of data but also to construct images that are more suitable for human and machine perception.

198Image fusion techniques are mainly classified into six categories: spatial domain fusion, transformation fusion, fuzzy logic, morphological methods, and sparse representation methods. Fusion imaging modes mainly include CT–MRI, MRI–PET, PET–CT, MRI–SPECT, MRI–MRA, Ultrasound–MRI, and X-ray–Ultrasound. Fusion imaging plays an important role in early detection, accurate diagnosis, and scientific treatment of cancer.

199Image fusion methods based on deep learning were rapidly developed and widely applied but the existing models lack generalization capabilities and can only effectively fuse specific types of images. Moreover, most current multimodal image fusion on medical images is based on two-modal fusion, which cannot improve the accuracy of tumor contours. The fusion of two or more modalities of medical images is beneficial to the quantification, qualitative, and localization of lesion tissues more clearly.

200The latest research introduces a global-local feature extraction strategy and a spatial-frequency fusion strategy to realize three-modal fusion and retain complete texture detail information and global contour information. Meanwhile, a multi-attention mechanism is adopted to extract more effective deep features and more accurate location information. Multi-modal medical image fusion methods need to be paid more attention in the future.

201Most deep learning networks, such as CNN, need to label medical images; yet manually labeling medical images is a difficult task. Therefore, minimizing the annotation cost, speeding up the annotation process, and reducing the burden on annotators have become critical challenges in deep learning-based medical image analysis. To address these issues, strategies based on semi-supervised, self-supervised, and multi-instance learning have attracted extensive attention in recent years. Specific solutions include generating pseudo-labels for unlabeled data, generating more labeled images for model optimization, and learning meaningful patterns from unlabeled image data.

202Many researchers work on generating pseudo-labels for unlabeled data to enrich training datasets, where the quality of pseudo-labels and the measure of confidence play crucial roles. Self-supervised learning (SSL) is used to extract and learn latent features of large-scale unlabeled datasets without human annotations, which is a promising solution to the challenges posed by the difficulty of large-scale annotation management. In, a high-quality classifier from a dataset with only a small fraction of labeled tumor areas is achieved by combining positive and unlabeled learning with domain adaptation techniques.

203Label noise is unavoidable in many medical image datasets due to a variety of reasons, such as the limited attention or expertise of annotators. Label noise often degrades the performance of deep learning models in medical image-based classification, detection, segmentation, registration, etc. However, few studies directly address the issue of label noise in deep learning for medical image analysis. In the future, learning based on instance-related label noise, multi-label data with label noise, etc., will be an extremely challenging task.

204Traditional deep learning-based training models require a large amount of manually labeled data and it is difficult to obtain enough labeled medical images for training, which hinders the further development of traditional supervised learning algorithms. In recent years, researchers have gradually liberated from the traditional paradigm of supervised learning. Paradigms such as weakly supervised learning, semi-supervised learning, and self-supervised learning were concerned and developed. In particular, self-supervised learning has achieved great success and is considered a bridge between supervised and unsupervised learning while facilitating the development of weakly supervised and semi-supervised learning. However, existing supervised learning strategies for medical images have a lot of room for improvement in terms of performance.

205The explainability of the model is very high in clinical applications. The black box characteristic of the deep learning method itself and the lack of tools to examine the behavior of the black box model are the main reasons hindering its application in the medical field. Images of different cancers vary widely and, while significant research advances have been made in explainable deep-learning cancer detection systems, most of the methods used in the literature are local and post-hoc approaches and usually explainable models or methods for specific cancer image analysis, which are still far from the clinical requirements for the formation of explainable imaging markers. Explainable methods of deep learning models will be a research focus in cancer diagnosis, in addition to the need to include clinicians in the design process of these algorithms and the need to build inherently explainable deep learning models for cancer detection.

206The goal of deep learning-based cancer diagnosis is to build models that are both robust and generalizable, that is, models that can predict results efficiently and accurately in a variety of situations. Due to the different acquisition, enhancement methods, and modalities of medical images as well as domain shifts on training and test data, there may be many differences in the image performance of the same disease, which significantly affects the performance of clinical cancer diagnosis.

207Radiomics is a quantitative medical imaging method and its analysis can be carried out on medical images from different modalities to obtain characteristic data of high throughput of medical images. Studies have shown that radiological features are generally not robust to medical image acquisition parameters, such as spatial resolution and image extraction settings. Genomics is a branch of science that studies the structure and function of the entire genome at the molecular level. Studies have shown that molecular/genomic profiling performed after surgery is the most accurate way to assess a patient’s risk and the need for further surgery and adjuvant therapy.

208By associating genomic data with radiomic features obtained from medical images in multiple dimensions, radiogenomics reveals potential carcinogenic mechanisms, improves cancer diagnosis, non-invasive prediction, and survival prediction at the molecular level and is currently becoming the forefront of precision medicine in oncology. Di Donato et al. showed that preoperative MRI radiomic analysis of patients with endometrial cancer could well predict tumor grade, myometrial invasion, lymphovascular space invasion, and lymph node metastasis. Models based on the radiological characteristics of CT and PET can detect lung nodules, distinguish malignant and benign lesions, and characterize their histology, stage, and genotype. In, a time series radiomics predictive model is designed for serial MRI evaluation of standardizing PCa active surveillance. In order to enhance the stability and reproducibility of radiomic features, it is necessary to evaluate and validate feature selection methods and predictive modeling in the future.

209This paper reviews and summarizes the research progress of deep learning applied to medical image-based cancer diagnosis. It mainly includes the following aspects: (i) introduces the basic working mechanism of commonly used medical image modalities (CT, MRI, US, X-ray, PET, and histopathology images); (ii) summarizes 9 basic architectures of deep learning and 12 classic Pretrained Models; (iii) introduced five advanced deep neural networks; (iv) sorted out commonly used over-fitting techniques; and (v) 5 applications of deep learning in cancer diagnosis based on medical images are introduced and discussed from three aspects: the data, label, and model.

210Although deep learning technologies and methods have made great progress in cancer diagnosis based on medical images in recent years, research on cancer diagnosis still faces many challenges due to the accuracy and rapidity requirements of cancer diagnosis tasks and the complexity of medical images, which mainly include the following aspects:Datasets problems.

211Datasets problems.

212At present, the lack of high-quality datasets in the medical image field has affected the development of deep learning for cancer diagnosis. Due to the lack of data on rare cancers, the current deep-learning research on cancer mainly focuses on common cancers and less attention has been paid to rare cancers.

213The model lacks explainability.

214As discussed above, explainability is very important in the medical field. A medical diagnosis system must be transparent, understandable, and explainable to make it easier for clinicians and patients to accept the clinical use of deep learning models. The lack of explainability of the training process and results of the deep learning model seriously hinders its application and promotion as well as hindering the development of intelligent medical diagnosis.

215Poor generalization ability.

216Given that the images of different cancers differ greatly, the methods proposed in the current research are mainly limited to the images of different cancers and different imaging types. Therefore, factors such as different equipment, different patient populations, and different imaging parameters cause deep learning to often face a significant performance drop when processing images in new data domains. In order to achieve clinical usability, it is urgent to improve the adaptive and generalization ability of deep learning model.

217Lack of high-performance models for multi-modal images.

218Multi-modal data and multi-scale data fusion can improve the accuracy of cancer diagnosis. However, many questions regarding the technical, analytical, and clinical aspects of multimodal fusion remain unresolved. The lack of generalization of the model in cross-center and multi-modal data scenarios also hinders its further development in the medical field.

219From the challenges discussed above, future research on applying deep learning to a medical image-based cancer diagnosis can start from the following four aspects. (i) The establishment of a growing database of public standards for cancer that is freely available to researchers. (ii) The deep neural network-based pretrained models have the potential to be improved, which can be researched in the aspects of multi-modal data fusion and unsupervised and self-supervised learning. (iii) Vision transformer, integrated learning, etc., are worthy of further study in medical image-based cancer diagnosis. (iv) Few-Shot learning can be further studied in overcoming the small amount of cancer data and the lack of standards.