Convolutional Neural Networks-Based MRI Image Analysis for the Alzheimer’s Disease Prediction From Mild Cognitive Impairment

1Alzheimer’s disease (AD) is the cause of over 60% of dementia cases, in which patients usually have a progressive loss of memory, language disorders and disorientation. The disease would ultimate lead to the death of patients. Until now, the cause of AD is still unknown, and no effective drugs or treatments have been reported to stop or reverse AD progression. Early diagnosis of AD is essential for making treatment plans to slow down the progress to AD. Mild cognitive impairment (MCI) is known as the transitional stage between normal cognition and dementia, about 10–15% individuals with MCI progress to AD per year. It was reported that MCI and AD were accompanied by losing gray matter in brain, thus neuropathology changes could be found several years before AD was diagnosed. Many previous studies used neuroimaging biomarkers to classify AD patients at different disease stages or to predict the MCI-to-AD conversion. In these studies, structural magnetic resonance imaging (MRI) is one of the most extensively utilized imaging modality due to non-invasion, high resolution and moderate cost.

2To predict MCI-to-AD conversion, we separate MCI patients into two groups by the criteria that whether they convert to AD within 3 years or not. These two groups are referred to as MCI converters and MCI non-converters. The converters generally have more severe deterioration of neuropathology than that of non-converters. The pathological changes between converters and non-converters are similar to those between AD and NC, but much milder. Therefore, it much more difficult to classify converters/non-converters than AD/NC. This prediction with MRI is challenging because the pathological changes related to AD progression between MCI non-converter and MCI converter are subtle and inter-subject variable. For example, ten MRI-based methods for predicting MCI-to-AD conversion and six of them perform no better than random classifier. To reduce the interference of inter-subject variability, MRI images are usually spatially registered to a common space. However, the registration might change the AD related pathology and loss some useful information. The accuracy of prediction is also influenced by the normal aging brain atrophy, with the removal of age-related effect, the performance of classification was improved.

3Machine learning algorithms perform well in computer-aided predictions of MCI-to-AD conversion. In recent years, deep learning, as a promising machine learning methodology, has made a big leap in identifying and classifying patterns of images. As the most widely used architecture of deep learning, convolutional neural networks (CNN) has attracted a lot of attention due to its great success in image classification and analysis. The strong ability of CNN motivates us to develop a CNN-based prediction method of AD conversion.

4In this work, we propose a CNN-based prediction approach of AD conversion using MRI images. A CNN-based architecture is built to extract high level features of registered and age-corrected hippocampus images for classification. To further improve the prediction, more morphological information is added by including FreeSurfer-based features (FreeSurfer, RRID:SCR_001847). Both CNN and FreeSurfer features are fed into an extreme learning machine as classifier, which finally makes the decision of MCI-to-AD. Our main contributions to boost the prediction performance include: (1) Multiple 2.5D patches are extracted for data augmentation in CNN; (2) both AD and NC are used to train the CNN, digging out important MCI features; (3) CNN-based features and FreeSurfer-based features are combined to provide complementary information to improve prediction. The performance of the proposed approach was validated on the standardized MRI datasets from the Alzheimer’s Disease Neuroimaging Initiative (ADNI – Alzheimer’s Disease Neuroimaging Initiative, RRID:SCR_003007) and compared with other state-of-the-art methods on the same datasets.

5The proposed framework is illustrated in Figure 1. The MRI data were processed through two paths, which extract the CNN-based and FreeSurfer-based image features, respectively. In the left path, CNN is trained on the AD/NC image patches and then is employed to extract CNN-based features on MCI images. In the right path, FreeSurfer-based features which were calculated with FreeSurfer software. These features, which were further mined with dimension reduction and sparse feature selection via PCA and Lasso, respectively, were concatenated as a features vector and fed to extreme learning machine as classifier. Finally, to evaluate the performance of the proposed approach, the leave-one-out cross validation is then used.

6Data used in this work were downloaded from the ADNI database. The ADNI is an ongoing, longitudinal study designed to develop clinical, imaging, genetic, and biochemical biomarkers for the early detection and tracking of AD. The ADNI study began in 2004 and its first 6-year study is called ADNI1. Standard analysis sets of MRI data from ADNI1 were used in this work, including 188 AD, 229 NC, and 401 MCI subjects. These MCI subjects were grouped as: (1) MCI converters who were diagnosed as MCI at first visit, but converted to AD during the longitudinal visits within 3 years (n = 169); (2) MCI non-converters who did not convert to AD within 3 years (n = 139). The subjects who were diagnosed as MCI at least twice, but reverse to NC at last, are also considered as MCI non-converters; (3) Unknown MCI subjects who missed some diagnosis which made the last state of these subjects was unknown (n = 93). The demographic information of the dataset are presented in Table 1. The age ranges of different groups are similar. The proportions of male and female are close in AD/NC groups while proportions of male are higher than female in MCI groups.

7MRI images were preprocessed following steps in. All images were first skull-stripped according to, and then aligned to the MNI151 template using a B-spline free-form deformation registration. In the implementation, we follow the Tong’s way to register images, showing that the effect of deformable registration with a control point spacing between 10 and 5 mm have the best performance in classifying AD/NC and converters/non-converters. After that, image intensities of the subjects were normalized by deform the histogram of each subject’s image to match the histogram of the MNI151 template. Finally, all MRI images were in the same template space and had the same intensity range.

8Normal aging has atrophy effects similar with AD. To reduce the confounding effect of age-related atrophy, age correction is necessary to remove age-related effects, which is estimated by fitting a pixel regression model to the subjects’ ages. We assume there are N healthy subjects and M voxels in each preprocessed MRI image, and denote ym∈R1 × N as the vector of the intensity values of N healthy subjects at mth voxel, and α∈R1 × N as the vector of the ages of N healthy subjects. The age-related effect is estimated by fitting linear regression model ym = ωmα + bm at mth voxel. For nth subject, the new intensity of mth voxel can be calculated as y′mn = ωm(C-αn) + ymn, where ymn is original intensity, αn is age of nth subject. In this study, C is 75, which is the mean age of all subjects.

9A CNN was adopted to extract features from MRI Images of NC and AD subjects. Then, the trained CNN was used to extract image features of MCI subjects. To explore the multiple plane images in MRI, a 2.5D patch was formed by extracting three 32 × 32 patches from transverse, coronal, and sagittal plane centered at a same point. Then, three patches were combined into a 2D RBG patch. Figure 2 shows an example of constructing 2.5D patch. For a given voxel point, three patches of MRI are extracted from three planes and then concatenated into a three channel cube, following the same way of composing a colorful patch with red/green/blue channels that are commonly used in computer vision. This process allows us to mine fruitful information form 3D views of MRI by feeding the 2.5D patch into the typical color image processing CNN network. Data augmentation was used to increase training samples, by extracting multiple patches at different locations from MRI images. The choice of locations has three constraints, (1) The patches must be originated in either left or right hippocampus region which have high correlation with AD; (2) There must be at least two voxels distance between each location; (3) All locations were random chosen. With these constraints, 151 patches were extracted from each image and the sampling positions were fixed during experiments. The number of samples was expanded by a factor of 151, which could reduce over-fitting.

10Typically extracted patches are presented in Figure 3. Figure 3A shows four 2.5D patches obtained from one subject. These patches are extracted from different positions and show different portions of hippocampus, which means these patches contain different information of morphology of hippocampus. When trained with these patches that spread in whole hippocampus, CNN learns the morphology of whole hippocampus. Figure 3B shows patches extracted in same position from four subjects of different groups, demonstrating that the AD subject has the most severe atrophy of hippocampus and expansion of ventricle. This implies that obvious differences are existed between AD and NC. However, the MCI subjects have the medium atrophy of hippocampus, and non-converter is more like NC rather than AD, and converter is more similar to AD. The difference between converter and non-converter is smaller than the difference between AD and NC.

11The architecture of the CNN is summarized in Figure 4. The network has an input of 32 × 32 RGB patch. There are three convolutional layers and three pooling layers. The kernel size of convolutional layer is 5 × 5 with 2 pixels padding, and the kernel size and stride of pooling layers is 3 × 3 and 2. The input patch has a size of 32 × 32 and 3 RBG channels. The first convolutional layer generates 32 feature maps with a size of 32 × 32. After max pooling, these 32 feature maps were down-sampled into 16 × 16. The next two convolutional layers and average pooling layers finally generate 64 features maps with a size of 4 × 4. These features are concatenated as a feature vector, and then fed to full connection layer and softmax layer for classification. There are also rectified linear units layers and local response normalization layers in CNN, but are not shown for simplicity.

12The CNN was trained with patches from NC and AD subjects, and there are 62967 (subject number 417 times 151) patches which are randomly split into 417 mini-batches. Mini-batch stochastic gradient descent was used to update the coefficients of CNN. In each step, a mini-batch was fed into CNN, and then error back propagation algorithm was carried out to computer gradient gj of jth coefficient θj, and update the coefficient as θ′j = θj + θn j, in which θn j = mθn-1 j- η(gj + λθj) is the increment of θj at nth step. The momentum m, learning rate η and weight decay λ are set as 0.9, 0.001, and 0.0001, respectively, in this work. It is called one epoch with all mini-batches used to train CNN once. The CNN was trained with 30 epochs. Once the network was trained, CNN will be used to extract high level features of MCI subjects’ images. The 1024 features output by the last pooling layer were taken as CNN-based features. Thus, CNN generates 154624 (1024 × 151) features for each image.

13The FreeSurfer (version 4.3) was used to mine more morphological information of MRI images, such as cortical volume, surface area, cortical thickness average, and standard deviation of thickness in each region of interest. These features can be downloaded directly from ADNI website, and 325 features are used to predict MCI-to-AD conversion after age correction. The age correction for FreeSurfer-based features is similar as described above, but on these 325 features instead of on intensity values of MRI images.

14Redundant features maybe exist among CNN-based features, thus we introduced the principle component analysis (PCA) and least absolute shrinkage and selection operator (LASSO) to reduce the final number of features.

15PCA is an unsupervised learning method that uses an orthogonal transformation to convert a set of samples consisting of possibly correlated features into samples consisting of linearly uncorrelated new features. It has been extensively used in data analysis. In this work, PCA is adopted to reduce the dimensions of features. Parameters of PCA are: (1) For CNN-based features, there are 1024 features for each patch. After PCA, PC features were left for each patch, since there are 151 patches for one subject, there are still PC × 151 features for each subject; (2) For FreeSurfer-based features, PF features were left for each MCI subject.

16LASSO is a supervised learning method that uses L1 norm in sparse regression as follows:

17Where y∈R1 × N is the vector consisting of N labels of training samples, D∈RN × M is the feature matrix of N training samples consisting of M features, λ is the penalty coefficient that was set to 0.1, and α∈R1 × M is the target sparse coefficients and can be used for selecting features with large coefficients. The LASSO was solved with least angle regression, and L features are selected after L iterations. Parameters of LASSO are: (1) For CNN-based features, LC features were selected from PC × 151 features for each MCI subject; (2) For FreeSurfer-based features, LF features were selected from PF features. After PCA and LASSO, there were LC + LF features.

18Figure 5 shows more details of CNN-based features. 151 patches are extracted from all MRI images, including AD, NC, and MCI. First, the CNN is trained with patches of all AD and NC subjects. After that, the trained CNN is used to output 1024 features from each MCI patch. The 1024 features of each patch are reduced to PC features by PCA, and then features of all 151 patches from one subject are concatenated, and Lasso is used to select LC most informative features from them.

19The extreme learning machine, a feed-forward neural network with a single layer of hidden nodes, learns much faster than common networks trained with back propagation algorithm. A special extreme learning machine, that adopts kernel to calculates the outputs as formula (2) and avoids the random generation of input weight matrix, is chosen to classify converters/non-converters with both CNN-based features and FreeSurfer-based features. In formula (2), the Ω is a matrix with elements Ωi,j = K(xi, xj), where K(a, b) is a radial basis function kernel in this study, [x1,…, xN] are N training samples, y is the label vector of training samples, and x is testing sample. C is a regularization coefficient and was set to 1 in this study.

20In our implementation, CNN was accomplished with Caffe1, LASSO was carried out with SPAMS2, and extreme learning machine was performed with shared online code3. The hippocampus segmentation was implemented with MALPEM4 for all MRI images. Then all hippocampus masks were registered as corresponding MRI images, and then overlapped to create a mask containing hippocampus regions. All image features were normalized to have zero mean and unit variance before training or selection. To evaluate the performance, Leave-one-out cross validation was used as.

21To validate the robustness of the CNN, several experiments have been performed with the CNN. In experiments, the binary decisions of CNN for 151 patches were united to make final diagnosis of the testing subject. We compared the performance in four different conditions: (1) The CNN was trained with AD/NC patches and used to classify AD/NC subjects; (2) The CNN was trained with converters/non-converters patches and used to classify converters/non-converters; (3) The CNN was trained with AD/NC patches and used to classify converters/non-converters; (4) The condition is similar with (3), but with different sampling patches in each validation run.

22The results are shown in Table 2. The CNN has a poor accuracy of 68.49% in classifying converters/non-converters when trained with converters/non-converters patches, but CNN has obtained a much higher accuracy of 73.04% when trained with AD/NC patches. This means that the CNN learned more useful information from AD/NC data than that from converters/non-converters data. And the prediction performance of CNN is close when different sampling patches are used.

23In this section, we present the performance of CNN-based features, FreeSurfer-based features, and their combinations. The PC, PF, LC, and LF parameters were set to 29, 150, 35, and 40, respectively, which were optimized in experiments. Finally, 75 features were selected and fed to the extreme learning machine.

24Performance was evaluated by calculating accuracy (the number of correctly classified subjects divided by the total number of subjects), sensitivity (the number of correctly classified MCI converters divided by the total number of MCI converters), specificity (the number of correctly classified MCI non-converters divided by the total number of MCI non-converters), and AUC (area under the receiver operating characteristic curve). The performances of the proposed method and the approach with only one type of features are summarized in Table 3. These results indicates that the approaches with only CNN-based features or FreeSurfer-based features have similar performances, and the proposed method combining both features achieved best accuracy, sensitivity, specificity and AUC. Thus, it is meaningful to combine two features in the prediction of MCI-to-AD conversion. The AUC of the proposed method reached 86.1%, indicating the promising performance of this method. The receiver operating characteristic (ROC) curves of these approaches are shown in Figure 6.

25We investigated the impact of age correction on the prediction of conversion here. The prediction accuracy in Table 3 and the ROC curves in Figure 6 implied that age correction can significantly improve the accuracy and AUC, Thus, age correction is an important step in the proposed method.

26In this section, we first compared the extreme learning machine with support vector machine and random forest. The performances of three classifiers are shown in Table 4, indicating that extreme learning machine achieves the best accuracy and AUC among three classifiers.

27Then we compared the proposed method with other state-of-the-art methods that use the same data, which consists of 100 MCI non-converters and 164 MCI converters. In both methods, MRI images were first preprocessed and registered, but in different ways. After that, features selection was performed to select the most informative voxels among all MRI voxels. Moradi used regularized logistic regression algorithm to select a subset of MRI voxels, and Tong used elastic net algorithm instead. Both methods trained feature selection algorithms with AD/NC data to learn the most discriminative voxels and then used to selected voxels from MCI data. Finally, Moradi used low density separation to calculate MRI biomarkers and to predict MCI converters/non-converters. Tong used elastic net regression to calculate grading biomarkers from MCI features, and SVM was utilized to classify MCI converters/non-converters with grading biomarker.

28For fair comparisons, both 10-fold cross validation and leave-one-out cross validation were performed on the proposed method and method of with only MRI data was used. Parameters of the compared approaches were optimized to achieve best performance. Table 5 shows the performances of three methods in 10-fold cross validation and Table 6 summarizes the performances in leave-one-out cross validations. These two tables demonstrate that the proposed method achieves the best accuracy and AUC among three methods, which means that the proposed method is more accurate in predicting MCI-to-AD conversion than other methods. The sensitivity of the proposed method is a little lower than the method of but much higher than the method of, and the specificity of the proposed method is between other two methods. Higher sensitivity means lower rate of missed diagnosis of converters, and higher specificity means lower rate of misdiagnosing non-converters as converters. Overall, the proposed method has a good balance between the sensitivity and specificity.

29The CNN has a better performance when trained with AD/NC patches rather than MCI patches, we think the reason is that the pathological changes between MCI converters and non-converters are slighter than those between AD and CN. Thus, it is more difficult for CNN to learn useful information directly from MCI data about AD-related pathological changes than from AD/NC data. The pathological changes are also hampered by inter-subject variations for MCI data. Inspired by the work in and which use information of AD and NC to help classifying MCI, we trained the CNN with the patches from AD and NC subjects and improved the performance.

30After non-rigid registration, the differences between all subject’s MRI brain image are mainly in hippocampus. So we extracted 2.5D patches only from hippocampus regions, that makes the information of other regions lost. For this reason, we included the whole brain features calculated by FreeSurfer as complementary information. The accuracy and AUC of classification are increased to 79.9 and 86.1% from 76.9 to 82.9% with the help of FreeSurfer-based features. To explore which FreeSurfer-based features contribute mostly when they are used to predict MCI-to-AD conversion, we used Lasso to select the most informative features, and the top 15 features are listed in Table 7, in which the features are almost volume and thickness average of regions related to AD. The thickness average of frontal pole is the most discriminative feature. The quantitative features of hippopotamus are not listed, indicating they contribute less than these listed features when predicting conversion. The CNN extract the deep features of hippopotamus morphology, rather than the quantitative features of hippopotamus, which are discriminative for AD diagnosis. Therefore, The CNN-based features and FreeSurfer-based features contain different useful information for classification of converters/non-converters, and they are complementary to improve the performance of classifier.

31Different from the two methods used in and, which directly used voxels as features, the proposed method employs CNN to learn the deep features from the morphology of hippopotamus, and combined CNN-based features with the globe morphology features that were computed by FreeSurfer. We believe that the learnt CNN features might be more meaningful and more discriminative than voxels. When comparing with these two methods, only MRI data was used, but the performances of these two methods were improved when combined MRI data with age and cognitive measures, so investigating the combination of the propose approach with other modality data for performance improvement is also one of our future works.

32We have also listed several deep learning-based studies in recent years for comparison in Table 8. Most of them have an accuracy of predicting conversion above 70%, especially the last three approaches (including the proposed one) have the accuracy above 80%. The best accuracy was achieved by, which uses both MRI and PET data. However, when only MRI data is used, Lu’s method declined the accuracy to 75.44%. Although an accuracy of 82.51% was also obtained with PET data, PET scanning usually suffers from contrast agents and more expensive cost than the routine MRI. In summary, our approach achieved the best performance when only MRI images were used and is expected to be improved by incorporating other modality data, e.g., PET, in the future.

33In this work, the period of predicting conversion was set to 3 years, that separates MCI subjects into MCI non-converters and MCI converters groups by the criterion who covert to AD within 3 years. But not matter what the period for prediction is, there is a disadvantage that even the classifier precisely predict a MCI non-converters who would not convert to AD within a specific period, but the conversion might still happen half year or even 1 month later. Modeling the progression of AD and predicting the time of conversion with longitudinal data are more meaningful. Our future work would investigate the usage of CNN in modeling the progression of AD.

34In this study, we have developed a framework that only use MRI data to predict the MCI-to-AD conversion, by applying CNN and other machine learning algorithms. Results show that CNN can extract discriminative features of hippocampus for prediction by learning the morphology changes of hippocampus between AD and NC. And FreeSurfer provides extra structural brain image features to improve the prediction performance as complementary information. Compared with other state-of-the-art methods, the proposed one outperforms others in higher accuracy and AUC, while keeping a good balance between the sensitivity and specificity.

35WL and XQ conceived the study, designed the experiments, analyzed the data, and wrote the whole manuscript. TT and QG provided the preprocessed data. WL, XQ, DG, XD, and MX carried out experiments. YY and GG helped to analyze the data and experiments result. MD and XQ revised the manuscript.

36The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.