Skip to content
naulon
Get started

No retraction, correction or concern found when last checked on Oct 17, 2026.

Multimodal machine learning in precision health: A scoping review

Adrienne Kline, Hanyin Wang, Yikuan Li and 6 more

npj Digital Medicine · 2022 · Review · CC BY

doi:10.1038/s41746-022-00712-8

Abstract

Machine learning is frequently being leveraged to tackle problems in the health sector including utilization for clinical decision-support. Its use has historically been focused on single modal data. Attempts to improve prediction and mimic the multimodal nature of clinical expert decision-making has been met in the biomedical field of machine learning by fusing disparate data. This review was conducted to summarize the current studies in this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA extension for Scoping Reviews to characterize multi-modal data fusion in health. Search strings were established and used in databases: PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 128 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. Early fusion was the most common data merging strategy. Notably, there was an improvement in predictive performance when using data fusion. Lacking from the papers were clear clinical deployment strategies, FDA-approval, and analysis of how using multimodal approaches from diverse sub-populations may improve biases and healthcare disparities. These findings provide a summary on multimodal data fusion as applied to health diagnosis/prognosis problems. Few papers compared the outputs of a multimodal approach with a unimodal prediction. However, those that did achieved an average increase of 6.4% in predictive accuracy. Multi-modal machine learning, while more robust in its estimations over unimodal methods, has drawbacks in its scalability and the time-consuming nature of information concatenation.

Full text

From PubMed Central, U.S. National Library of Medicine, as current on Oct 17, 2026. Paragraph numbers are ours, so an answer can point at one.

Introduction

Clinical decision support has long been an aim for those implementing algorithms and machine learning in the health sphere–. Examples of algorithmic decision supports utilize lab test values, imaging protocols or clinical (physical exam scores) hallmarks,. Some health diagnoses can be made on a single lab value or a single threshold, such as in diabetes in older adults. Other diagnoses are based on a constellation of the signs, symptoms, lab values and/or supportive imaging and are referred to as a clinical diagnosis. Oftentimes these clinical diagnoses are based on additive scoring systems that requires an admixture of positive and negative hallmarks prior to confirmatory labeling.

The modus operandi of a clinical diagnosis may fail to consider the relative weighting of these disparate data inputs and potentially non-linear relationships highlighting the limitations of human decision-making capacity. The strength of algorithmic decision-making support is that it can be used to offload such tasks, ideally yielding a more successful result. This is the promise of precision medicine. Precision medicine/health aims to create a medical model that customizes healthcare (decisions, treatments, practices etc.) that are tailored to either an individual or patient phenotype. This includes tracking patients’ health trajectories longitudinally, oftentimes incorporating genetics/epigenetics, and mathematical modeling where diagnoses and treatments incorporate this unique information. Contrast this with a one-drug-fits-all model, where there is a single treatment per disorder. Figure 1 illustrates the flow of information from hospitals/care centers that generate disparate data. It is through computational modeling and information fusion that outcomes of interest such as drug and treatment targets ultimately facilitate better decision making at the patient level in those care centers. This phenomenon has sparked an interest in fusion studies using health care data.

Undertakings to characterize this literature have been performed by Huang et al., who performed a systematic review of deep learning fusion of imaging and EHR data in health. However, it was limited to EHR and imaging data and deep learning applications. A follow-up review article included a commentary on omics and imaging data fusion. The purpose of this study is to highlight the current scope of this research domain, summarize and offer suggestions to advance the field. The current study is more inclusive in the breadth of the types of machine learning protocols used and attempts to encompass all current modalities (information types/sources).

Data fusion is underpinned by information theory and is the mechanism by which disparate data sources are merged to create an information state based on the sources’ complementarity, (Box 1). The expectation in machine learning is that data fusion efforts will result in an improvement in predictive power, and therefore provide more reliable results in potentially low validity settings. Data fusion touts the advantage that the results of modeling become inherently more robust by relying on a multitude of informational factors rather than a single type. However, the methodology of combinatory information has drawbacks; it adds complexity to specifying the model and reduces the interpretability of results,.

Data from different sources and file formats are rarely uniform, and this is especially the case with clinical data. For example, data sets can have different naming conventions, units of measure, or represent different local population biases. Care must be taken to search and correct for systematic differences between datasets and assess their degree of inter-operability. For example, Colubri et al. aggregated computed tomography (CT) and PCR lab values, by performing an intra-site normalization. This ensured that the values were comparable across sites. In doing so they discarded several potentially informative clinical variables since they were not all available in all datasets.

A balance is required to allow information that is similar to work together (harmonization) and retain data purity (information correspondence). Successful fusion uses data harmonization techniques that assure both in the quality control of the integration process. Clinical data harmonization requires multidisciplinary research among medicine, biology, and computer science. The clinical area of heart failure with preserved ejection fraction (HFpEF) saw novel applications of multiple tensor factorization formulations to integrate the deep phenotypic and trans-omic information, and this extends to other areas of precision medicine. To increase the portability of EHR-based phenotype algorithms, the Electronic Medical Records and Genomics (eMERGE) network has adopted common data models (CDMs) and standardized design patterns of the phenotype algorithm logic to integrate EHR data with genomic data and enable generalizability and scalability–.

There are three main types of data fusion that are used in machine learning; early (data-level), intermediate (joint), and late (decision-level). In the case of early fusion, multiple data sources are converted to the same information space. This often results in vectorization or numerical conversion from an alternative state, such as that performed by Chen et al. via vectorized pathology reports. Medical images possess characteristics that can undergo numerical conversion based on area, volume, and/or structural calculations. These are then concatenated with additional measurements from structured data sources and fed into an individual classifier. Canonical correlation analysis, non-negative matrix factorization,, Independent Component Analysis (ICA) and numerical feature conversion methodologies exist as common options to transform all data into the same feature space.

Intermediate data fusion occurs as a stepwise set of models and offers the greatest latitude in model architecture. For example, a 3-stage deep neural learning and fusion model was proposed by Zhou et al.. Stage 1 consists of feature selection by a soft-max classifier for independent modalities. Stages 2 and 3 constitute combining these selected features, establishing a further refined set of features, and feeding these into a Cox-nnet to perform joint latent feature representation for Alzheimer’s diagnosis. In contrast to early fusion, intermediate fusion combines the features that distinguish each type of data to produce a new representation that is more expressive than the separate representations from which it arose.

In late fusion, typically multiple models are trained where each model corresponds to an incoming data source. This is akin to ensemble learning, which offers better performance over individual models. Ensemble methods use multiple learning algorithms (typically applied to the same dataset) to obtain better predictive performance than could be obtained from any of the constituent learning algorithm alone. However, multimodal machine learning ensemble here can refer to ensemble learning within a data type or across data types. These take symbolic representations as sources and combine them to obtain a more accurate decision. Bayesian’s methods are typically employed at this level to support a voting process between the set of models into a global decision. Within late fusion there has been headway made to perform multitask deep learning–. A schematic for the 3 subtypes of data fusion is presented in Fig. 2. Attributes in the fusion techniques are shown in Table 1.

Multimodal machine learning: the area of machine learning concerned with bringing together disparate data sources to capitalize on the unique and complementary information in an algorithmic framework.

Data harmonization: using machine learning to unify different data sources to improve its quality and utilization.

Multiview machine learning: another term for multimodal machine learning.

Data fusion: the specific methodologies undertaken to perform data integration for multimodal/multiview machine learning; they come in three broad categories: early, intermediate/joint, and late.

Topic Modeling

The topic modeling displayed in Fig. 3 showcases the category, specific health ailment under investigation, and the modality type for the studies included. These were subsequently mapped to the category of the combination of information that were merged to create models for prediction/classification/clustering (Table 2). This plot should serve as a resource to fellow researchers to identify areas that are less frequent, such as dermatology, hematology, medication/drug issues such as alcohol use disorder that may offer new research horizons. Figure 4 identifies coding platforms, publishing trend and location over time, author locations and patient cohorts of the papers included in this review.

Model validation, techniques, and modalities used

Of the models used in the papers, 126/128 explicitly reported performing a validation procedure of them. The most common validation processes performed were N-fold cross validation (55),, train test split (51), leave one out cross validation (10), and external dataset (10). A cornucopia of machine learning techniques and methods were used within and across articles in this review. They have been summarized in Table 2, noting in which fusion umbrella subtype they were implemented.

Early fusion

Most papers were published using early fusion. Of those, most were published using medical imaging and EHR data,,,,,,,–,,,,,–,,–,,,,. Nearly all these papers performed numericalization of image features in essence converting them to structured data prior to processing, however, two performed matrix factorization,. A combination of EHR and text data was noted in 15 papers,,,,–,,,,,. Meng et al. created a Bidirectional Representation Learning model used latent Dirichlet allocation (LDA) on clinical notes. Cohen et al. used unigrams and bigrams in conjunction with medication usage. Zeng et al. used concept identifiers from text as input features. Nine papers used early fusion with imaging, EHR and genomic data,,,,,,,,. Doan et al. concatenated components derived from images with polygenic risk scores. Lin et al. also created aggregated scores from MRI, cerebral spinal fluid, and genetic information and brought them together into a single cohesive extreme learning machine to predict mild cognitive impairment. Tremblay et al. used a multivariate adaptive regression spline (MARS) after normalizing, removing highly correlated features. Ten papers performed fusion using imaging and genomic data,,,–,,,,. Three of these generated correlation matrices as features by vectorizing imaging parameters and correlating them with single nucleotide polymorphisms (SNPs) prior to feeding into the model,,. Three papers in this category used EHR and time series,,. Both Hernandez and Canniére et al. implemented their methods for purposes of cardiac rehabilitation and harnessed the power of support vector machines (SVMs). However, Hernandez preserved time series information by assembling ECG data into tensors that preserve the structural and temporal relationships inherent in the feature space, while Canniére performed dimensionality reduction of the time series information using t-SNE plots. Two papers comprised early fusion using imaging and time series,. There were two papers that leveraged EHR and genomic information,. Luo et al. implemented hybrid non-negative matrix factorization (HNMF) to find coherence between phenotypes and genotypes in those suffering from hypertension. One paper leveraged early fusion using imaging and text data and another used EHR, Genomics, Transcriptomics, and Insurance Claims.

Intermediate fusion

Intermediate fusion had the second highest number of papers published. 14 used imaging and EHR data,,,,,,,,,,–,,. Zihni et al. merged the output from a Multilayer Perceptron (MLP) for modeling clinical data and convolutional neural network (CNN) for modeling imaging data into a single full connected final layer to predict stroke. A very similar approach was taken by Tang et al. who used three-dimensional CNNs and merged the layers in the last layer. EHR and text data were fused together in 11 papers,,,,,,,,,,. Of these, six,,,,, used long term short term (LSTM) networks, CNNs, or knowledge-guided CNNs in their fusion of EHR and clinical notes. Chowdhury et al. used graph neural networks and autoencoders to learn meta-embeddings from structured lab test results and clinical notes,. Pivovarov et al. learned probabilistic phenotypes from clinical notes and medication/lab orders (EHR) data. Two models each employing LDA where data type was treated as a bag of elements and to bring coherence between the two models to identify unique phenotypes. Ye et al. and Shin et al. used concept identifiers via NLP and bag-of-words techniques, respectively, prior to testing a multitude of secondary models,. In general, clinical notes can provide complementary information to structured EHR data, where natural language processing (NLP) is often needed to extract such information–. A few studies were published using imaging and genomic,,. Here radiogenomics were used to diagnose attention-deficit/hyperactivity disorder (ADHD), glioblastoma survival, and dementia respectively. Polygenic risk scores were combined with MRI by Yoo et al. who used an ensemble of random forests for ADHD diagnosis. Zhou et al. fused SNPs information together with MRI and positron emission tomography (PET) for dementia diagnosis by learning latent representations (i.e., high-level features) for each modality independently. Subsequently learning joint latent feature representations for each pair of modality combination and then learning the diagnostic labels by fusing the learned joint latent feature representations from the second stage was carried out. Wijethilake used MRI and gene expression profiling, performing recursive feature elimination prior to merging into multiple models SVM, linear regression, and artificial neural network (ANN). The linear regression model outperformed the other two merged models and any single modality. Wang et al. and Zhang et al. showcased their work in merging imaging and text information,. Both used LSTM for language modeling a CNN to generate embeddings that were joined together in a dual-attention model. This is achieved by computing a context vector with attended information preserved for each modality resulting in joint learning. Seldom were articles published using: Imaging/EHR/Text, Genomic/Text, Imaging/Time series, Imaging/Text/Time series, Imaging/EHR/Genomic, Imaging/EHR/Time series, EHR/Genomic, EHR/Text/Time series.

Late fusion

A much smaller number (n = 20) of papers used late fusion. Seven of those used imaging and EHR data types,,,,,,. Both Xiong et al. and Yin et al. fed outputs into a CNN to provide a final weighting and decision,. Three papers were published using a trimodal approach: imaging, EHR and genomic,,. Xu et al. and Faris et al. published papers using EHR and text data,. Faris et al. processed clinical notes using TF-IDF, hashing vectorizer and document embeddings in conjunction with binarized clinical data. Logistic Regression (LR), Random Forest (RF), Stochastic Gradient Descent Classifier (SGD Classifier), and a Multilayer Perceptron (MLP) were applied to both sets of data independently and final outputs of the two models were combined using different schemes: ranking, summation, and multiplication. Two articles were published using imaging and time series, both of which employed CNNs, one in video information of neonates and the other in chest x-rays. However, they differed in their processing of the time series data. Salekin used a bidirectional CNN and Nishimori used a one-dimensional CNN. Far fewer papers were published using Imaging/EHR/Text, EHR/Genomic/Text, imaging/EHR, time series/, Imaging/Genomic, EHR/Genomic, and Imaging/Text.

Mixed fusion

Two papers performed multiple data fusion architectures,. Huang et al. created seven different fusion architectures. These included, early, joint, and late fusion. The architecture that performed the best was the late elastic average fusion for the diagnosis of pulmonary embolism using computed tomography and EHR data. Their Late Elastic Average Fusion leveraged an ElasticNet (linear regression with combined L1 and L2 priors that act as regularizers) for EHR variables. El-Sappagh et al. performed early and late fusion to create an interpretable Alzheimer’s diagnosis and progression detection model. Their best performing model was one that implemented instance-based explanations of the random forest classifier by using the SHapley Additive exPlanations (SHAP) feature attribution. Despite using clinical, genomic, and imaging data, the most influential feature was found to be the Mini-Mental State Examination.

Clinical relevance

Data fusion may help address sex representation and increase population diversity issues (including minority populations) in health modeling by creating a more representative dataset if one datatype contained more of one sex and another datatype contained more of the other. This reciprocal compensation ability of employing various data sets would also hold true for racial or ethnic diversities.

Less than half (37.6%) of the papers were published in a journal intended for a clinical audience. None of the papers included in the final cohort of studies had created tools for clinical use that had FDA approval. Based on the rising number of papers in this field there is a growing and global need and interest to characterize these findings.

Discussion

Returning to our research questions, we outlined from the inception of this work, we arrive at Table 3.

Many issues were raised in the papers included in this review. The most common reported limitations were cohorts from a single site, small sample sizes, retrospective data, imbalanced samples, handling of missing data, feature engineering, controlling for confounding factors, and interpretation of the models employed. Samples were most often built from a single hospital or academic medical center. Small sample sizes often lead to poor model fitting and generalizability. The median number of unique patients reported across the studies was 658 with a standard deviation of 42,600. This suggests that while some studies were able to leverage large and multi-center cohorts, a great many were not able to do so,,,.

Seldom were machine learning investigations on prospective data, an issue endemic in the field. Sample imbalances were often ignored, which results in biased models and misleading performance metrics,. Missing data were usually ignored by dropping data or imputing, if not dealt with appropriately can skew the results,,. More studies need to discuss frequencies and types of missing data–. Comparison of different imputation methods on the final results should be part of the reporting process. When performing statistical analysis, researchers usually ignored possible confounding factors such as age or gender. Doing so may have major effects on the impact of results. Such possible confounding effects should either be taken into consideration by the model, or adjusted for first, prior to reporting model results. Reasonable interpretations of the model and outputs must be presented so that clinicians find the results credible and then use them to provide guidance for treatments. However, most authors did not take the time to interpret the models for clinical audiences. Additionally, how the results may function as a clinical decision support tool. Different types of models warrant different explanations,. These limitations are highlighted where they occur in the data processing and modeling building pipeline in Fig. 5.

To expedite and facilitate this field, we have outlined several gaps for future research in this field. These are listed in Fig. 5 and explored. Medication/drug topics present an underrepresented area, with only two papers being published in this field,. Awareness of drug interaction effects is a difficult and growing issue–, particularly in geriatrics, which gave rise to Beer’s criteria. Performing multimodal machine learning may offer an earlier detection of adverse events associated with medication misuse that is a result of iatrogenic error, non-compliance, or addiction. Similar justifications as outlined above could be applied to other areas seen as ‘under saturated’ such as hematology with only one paper and nephrology having just three,,.

Augmenting clinical decision-making with ML to improve clinical research and outcomes offers positive impacts that have economic, ethical, and moral ramifications, as it can reduce suffering and save human lives. Multiple studies have now pointed out that if the data an ML model is trained on is biased this often yields bias in the predictions,. Ensuring multisite, representative data will limit model biases. We also advocate for the creation of open access pipelines/libraries to speed up data conversion to make the technology more widely available,. Improving accuracy at the expense of complex and time-consuming data transformations may mean the predictive power gained from a multimodal approach is offset by this front-end bottleneck, meaning predictions are no longer temporally relevant or useful.

While incorporating disparate data does lend itself to seemingly better predictions, as knowledge around certain diseases accumulates, data fusion in healthcare is an evolving target that warrants proactively adapting to the dynamic landscape. There is no single ML model with ubiquitous applicability. For example, it has been shown in protein-protein interactions that utilization of the XGBoost ensemble algorithm reduces noisy features, maintains the significant raw features, and prevents overfitting. Similarly, LightGBM has the advantages of faster training speed, higher efficiency, lower memory usage, better accuracy, and has been consistently outperforming other models,. Graph neural networks can synthesize new connections leading to drug discovery/targets.

In the same vein, models that permit interpretability should always be considered. For example, the Perotte et al. model was not compared with conventional simpler machine learning classifiers, and collective matrix factorization becomes inherently difficult to interpret. Contrast this with the work of Fraccaro et al. whose study of macular degeneration noted their white box performed as well as black box methods implementions.

As this field and the datasets associated mature there is work needed to address the tenets of data management: Findability, Accessibility, Interoperability, and Reuse of digital datasets (FAIR). This entails having metadata that are unique/de-identified and searchable, with open or federated access points (Findability/Accessibility), data that are shared broadly (Interoperable), and finally data that contain accurate and relevant attributes under a clear data usage agreement/license (Reusable). It is imperative there exist a clear definition of outcomes, assessment of biases and interpretability/transparency of results, and limitations inherent in its predictions.

Of crucial importance for uptake is that predictions be patient-specific and actionable at a granular level. For example, a 30-day readmission prediction algorithm, if implemented, may inform resource management and prompt additional research that may decrease the number of patients re-admitted. Linden et al. developed Deep personalized LOngitudinal convolutional RIsk model (DeepLORI) capable of creating predictions that can be interpreted on the level of individual patients. Leveraging both and clinical and empirically driven information to create meaningful and usable recommendations may improve clinician/end-user under understanding by relating to existing frameworks. Resources such as CRISP-ML provide a framework for moving use cases into more practical applications, while efforts to vie for Food and drug administration (FDA) approvals as a tool for use are encouraged to increased adoption.

Deployment of models with user interfaces annotating limitations inherent on those predictions will allow clinical decision makers to interface and implement change accordingly. Follow-through on the aforementioned tasks will push individual fields to create recommendations for subsequent real-world implementations that are relevant, actionable, and transcend regional/subpopulation differences. Limitations of this scoping review include that it is not a systematic review. Therefore, it is possible that some titles that should have been included were missed. As the primary purpose of this study was to perform scientific paper profiling on multimodal machine learning in health, a critical appraisal of individual methodological quality of the included studies was not performed. However, commentary is provided on the methodological limitations that could have affected their results and impacted their claims. This review offers comprehensive meta-data and evaluation across health domains, immaterial to the type of machine learning or the data used. This work serves as both a summary and steppingstone for future research in this field. Data fusion in health is a growing field of global interest. The topic areas of health that have high frequency relative to others were neurology and cancer, which serve to highlight opportunities for further exploration in understudied topics (hematology, dermatology). Unimodal machine learning is inherently in contrast to current routine clinical practice in which imaging, clinical or genomic data are interpreted in unison to inform accurate diagnosis and warrants further work for ease of use and implementation. Overall, it appears justified to claim that multi-modal data fusion increases predictive performance over unimodal approaches (6.4% mean improvement in AUC) and is warranted where applicable. Multimodal machine learning may be a tool leveraged in precision medicine to further subgroup patients’ and their unique health fingerprint. Furthermore, as no papers in our review sought FDA approval, we advocate for more efforts into model translation and explore necessities that facilitate that end.

A dashboard resource published in conjunction with this review article is available at: https://multimodal-ml-health.herokuapp.com/. This dashboard was created as an interactive infographic-based display of the major findings presented in this paper. To foster future work, a drop-down menu was created to help researchers filter the underlying data file of titles based on the specific overarching health topic by selection. This will facilitate the location of relevant papers.

Search strategy and selection criteria

Inclusion requirements were: (a) original research article; (b) published within the last 10 years (encompassing years 2011–2021); (c) published in English; and (d) on the topic of multi-modal or multi-view using machine learning in health for diagnostic or prognostication applications.’Multi-modal’ or’multi-view’ for our context means the multiple data sources were not of the same type. For example, while a paper using CT and MRI may be considered multi-modal imaging; however, under our criteria it would be considered uni-modal (i.e., only included imaging). Exclusions for the purposes of this review were: (a) scientific articles not published in English; (b) commentaries or editorials; or (c) other review articles. Papers were also excluded if the data were not human-derived. We also excluded papers where the fusion already occurred at the data generation stage, such as spatial transcriptomics producing integrated tissue imaging and transcriptomics data–. All papers underwent a 2-person verification for inclusion in the manuscript.

Search strings were established via literature searches and domain expertize. Additional keywords were identified based on key word co-occurrence matrices established from the abstracts of the previously included articles. Figure 6a displays the search strings, where an individual string would include one keyword from each column, this was performed for all combinations of search strings. An overview of the inclusion/exclusion process is noted in Fig. 6b and follows the standard set by PRISMA extension for scoping reviews.

Data extracted

Information garnered from the articles included title, year published, FDA approval of the tool, whether published in a clinical journal, author affiliations, number of authors, locations (continents), and abstract. Health topic(s) addressed were extracted, as well as the broader medical topic(s) that encompass the disease. For example, lung cancer would be the specific disease in question. It arises from the topics of Cancer and Respiratory according to our classification. Health topic classification was overseen and reviewed by a medical doctor to ensure accuracy. As multiple health topics often encompassed a single health disease addressed in each paper, several papers are counted twice. This is true when being mapped from the right side of the Sankey plot to the specific health disease in the middle.

We recorded and extracted the number of different modalities and the divisions (i.e., text/image vs EHR/genomic/time series) used. The objective of each paper was extracted in a 1–2 sentence summary along with the keyword (if available). Patient characterization in the studies was performed by ascertaining the number of unique patients in the cohort and patient sex (i.e., Men/women/both or not mentioned).

Computational information extracted included: (a) the coding interface(s) used in data processing/analysis, (b) machine learning type, (c) data merging technique (early, intermediate, late), and (d) types of machine learning algorithms used. Whether validation was performed (yes/no), the statistical tests run, the nature of the validation, and outcomes measures were all recorded for each paper. The significance, impact, and limitations of each paper were extracted by reviewing the primary findings and limitations as noted in the papers.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.

References 202

  1. 1.
    The ethics of machine learning-based clinical decision support: an analysis through the lens of professionalisation theory · 2021
    Not checked
  2. 2.

    Sanchez-Pinto LN, Luo Y, Churpek MM. Big data and data science in critical care. Chest. 2018;154:1239-1248.

    Not in our library10.1016/j.chest.2018.04.037

  3. 3.

    Miotto, R., Wang, F., Wang, S., Jiang, X. & Dudley, J. T. Deep learning for healthcare: Review, opportunities, and challenges. Brief Bioinform.19, 1236–1246 (2017).

    Not in our library10.1093/bib/bbx044

  4. 4.

    Timothy JWD, et al. Machine learning of threedimensional right ventricular motion enables outcome prediction in pulmonary hypertension: A cardiac MR imaging study. Radiology. 2017;283:381-390.

    Not in our library10.1148/radiol.2016161315

  5. 5.

    Gigi, F. S., Gregory, R. H., Bradley, J. N. & Jun, D. Predicting breast cancer risk using personal health data and machine learning models. PLoS One14, e0226765 (2019).

    Not in our library10.1371/journal.pone.0226765

  6. 6.

    Michael F, Justin BE-T, Elizabeth S. Clinical and public health implications of 2019 endocrine society guidelines for diagnosis of diabetes in older adults. Diabetes Care. 2020;43:1456-1461.

    Not in our library10.2337/dc19-2467

  7. 7.

    Gambhir SS, Ge TJ, Vermesh O, Spitler R. Toward achieving precision health. Sci. Transl. Med. 2018;10:eaao3612.

    Not in our library10.1126/scitranslmed.aao3612

  8. 8.

    Schüssler-Fiorenza Rose SM, et al. A longitudinal big data approach for precision health. Nat. Med. 2019;25:792-804.

    Not in our library10.1038/s41591-019-0414-6

  9. 9.

    Feero WG. Introducing “genomics and precision health”. JAMA. 2017;317:1842-1843.

    Not in our library10.1001/jama.2016.20625

  10. 10.

    Kellogg RA, Dunn J, Snyder MP. Personal omics for precision health. Circulation Res. 2018;122:1169-71.

    Not in our library10.1161/circresaha.117.310909

  11. 11.

    Thapa C, Camtepe S. Precision health data: Requirements, challenges and existing techniques for data security and privacy. Comput. Biol. Med. 2021;129:104130.

    Not in our library10.1016/j.compbiomed.2020.104130

  12. 12.

    Pranata S, et al. Precision health care elements, definitions, and strategies for patients with diabetes: A literature review. Int. J. Environ. Res. Public Health. 2021;18:6535.

    Not in our library10.3390/ijerph18126535

  13. 13.
    Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines · 2020
    No notice
  14. 14.

    Weixian, H., Kaiwen, T., Jinlong, H., Ziye, Z. & Shoubin, D. A review of fusion methods for omics and imaging data. In IEEE/ACM Trans Comput Biol Bioinform (IEEE, 2022).

    Not in our library10.1109/tcbb.2022.3143900

  15. 15.
    A Review of Data Fusion Techniques · 2013
    No notice
  16. 16.

    Alan NS, Christopher LB, Franklin EW. Revisions to the JDL data fusion model. Sens. Fusion.: Architectures, Algorithms, Appl. III. 1999;3719:430.

    Not in our library

  17. 17.

    Erik, M.-M.-R., Antonio, A. A., Ramon, F. B. & Enrique, G.-C. Improved accuracy in predicting the best sensor fusion architecture for multiple domains. Sensors21, 7007 (2021).

    Not in our library10.3390/s21217007

  18. 18.

    Ahmad FS, Luo Y, Wehbe RM, Thomas JD, Shah SJ. Advances in machine learning approaches to heart failure with preserved ejection fraction. Heart Fail Clin. 2022;18:287-300.

    Not in our library10.1016/j.hfc.2021.12.002

  19. 19.

    Erik, B. et al. Machine learning/artificial intelligence for sensor data fusion-opportunities and challenges. In IEEE Aerospace and Electronic Systems Magazines Vol. 36, 80–93 (IEEE, 2021).

    Not in our library

  20. 20.
    Machine Learning for Lung Cancer Diagnosis, Treatment, and Prognosis · 2022
    No notice
  21. 21.

    Kohane IS, et al. What every reader should know about studies using electronic health record data but may be afraid to ask. J. Med. Internet Res. 2021;23:e22219.

    Not in our library10.2196/22219

  22. 22.

    Andres C, et al. Machine-learning Prognostic Models from the 2014–16 Ebola Outbreak: Data-harmonization challenges, validation strategies, and mHealth applications. EClinicalMedicine. 2019;11:54-64.

    Not in our library10.1016/j.eclinm.2019.06.003

  23. 23.

    Afshin J, Jean Pierre P, Johanne M-P. Machine-learning-based patient-specific prediction models for knee osteoarthritis. Nat. Rev. Rheumatol. 2019;15:49-60.

    Not in our library10.1038/s41584-018-0130-5

  24. 24.

    Luo, Y., Ahmad, F. S. & Shah, S. J. Tensor factorization for precision medicine in heart failure with preserved ejection fraction. J. Cardiovasc. Transl. Res.10, 305–312 (2017).

    Not in our library10.1007/s12265-016-9727-8

  25. 25.

    Luo, Y., Wang, F. & Szolovits, P. Tensor factorization toward precision medicine. Briefings Bioinform.18, 511–514 (2016).

    Not in our library10.1093/bib/bbw026

  26. 26.

    Rasmussen L. et al. Considerations for improving the portability of electronic health record-based phenotype algorithms. In Proceedings of 2019 AMIA Annual Symposium 2019 (2019).

    Not in our library

  27. 27.

    Zhong, Y. et al. Characterizing design patterns of EHR-driven phenotype extraction algorithms. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 1143–1146 (IEEE, 2018).

    Not in our library

  28. 28.

    Rasmussen LV, et al. Solutions for unexpected challenges encountered when integrating research genomics results into the EHR. ACI Open. 2020;4:e132-e5.

    Not in our library10.1055/s-0040-1719059

  29. 29.

    Shang N, et al. Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network. J. Biomed. Inf. 2019;99:103293.

    Not in our library10.1016/j.jbi.2019.103293

  30. 30.

    Dana L, Tulay A, Christian J. Multimodal data fusion: An overview of methods, challenges, and prospects. Proc. IEEE. 2015;103:1449-1477.

    Not in our library10.1109/jproc.2015.2460697

  31. 31.

    Wei, C., Yungui, H., Brendan, B. & Simon, L. The utility of including pathology reports in improving the computational identification of patients. J. Pathol. Inform.7, 46 (2016).

    Not in our library10.4103/2153-3539.194838

  32. 32.

    Yubraj, G., Ramesh Kumar, L. & Goo Rak, K. Prediction and classification of Alzheimer’s disease based on combined features from apolipoprotein-E genotype, cerebrospinal fluid, MR, and FDG-PET imaging biomarkers. Front. Comput. Neurosci.13, 72 (2019).

    Not in our library10.3389/fncom.2019.00072

  33. 33.

    Xia An B, Xi H, Hao W, Yang W. Multimodal data analysis of Alzheimer’s disease based on clustering evolutionary random forest. IEEE J. Biomed. Health Inform. 2020;24:2973-2983.

    Not in our library10.1109/jbhi.2020.2973324

  34. 34.

    Ariana A, et al. Non-negative matrix factorization of multimodal MRI, fMRI and phenotypic data reveals differential changes in default mode subnetworks in ADHD. Neuroimage. 2014;102:207-219.

    Not in our library10.1016/j.neuroimage.2013.12.015

  35. 35.

    Chao G, Luo Y, Ding W. Recent advances in supervised dimension reduction: A survey. Mach. Learn. Knowl. Extraction. 2019;1:341-358.

    Not in our library10.3390/make1010020

  36. 36.

    Pillai PSLTY. Fusing heterogeneous data for Alzheimer’s disease classification. Stud. Health Technol. Inf. 2015;216:731-735.

    Not in our library

  37. 37.

    Tao Z, Kim Han T, Xiaofeng Z, Dinggang S. Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis. Hum. Brain Mapp. 2019;40:1001-1016.

    Not in our library10.1002/hbm.24428

  38. 38.

    Robi P. Ensemble based systems in decision making. IEEE Circuits Syst. Mag. 2006;6:21-44.

    Not in our library10.1109/mcas.2006.1688199

  39. 39.

    Francesco, C. & Marwa, M. Multimodal temporal machine learning for bipolar disorder and depression recognition. Pattern Analysis Appl.25, 493–504 (2021).

    Not in our library

  40. 40.

    Durrant-Whyte HF. Sensor models and multisensor integration. Int. J. Robot. Res. 1988;7:97-113.

    Not in our library10.1177/027836498800700608

  41. 41.

    Xu Z, et al. Identification of predictive sub-phenotypes of acute kidney injury using structured and unstructured electronic health record data with memory networks. J. Biomed. Inform. 2019;102:103361.

    Not in our library10.1016/j.jbi.2019.103361

  42. 42.
    Combining structured and unstructured data for predictive models: a deep learning approach · 2020
    No notice
  43. 43.

    Shaker E-S, Tamer A, Islam SMR, Kyung Sup K. Multimodal multitask deep learning model for Alzheimer’s disease progression detection based on time series data. Neurocomputing. 2020;412:197-215.

    Not in our library10.1016/j.neucom.2020.05.087

  44. 44.

    Haiyang, Y., Li, K. & Feng Qiang, X. Multimodal temporal-clinical note network for mortality prediction. J. Biomed. Semantics12, 3 (2021).

    Not in our library10.1186/s13326-021-00235-3

  45. 45.

    Zizhao, Z., Pingjun, C., Manish, S. & Lin, Y. TandemNet: Distilling knowledge from medical images using diagnostic reports as optional semantic references. International Conference on Medical Image Computing and Computer-Assisted Intervention, 10435, 320–328 (Springer, Cham, 2017).

    Not in our library

  46. 46.

    Xiaosong, W., Yifan, P., Le, L., Zhiyong, L. & Ronald, M. S. TieNet: Text-Image embedding network for common thorax disease classification and reporting in chest X-rays. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 9049–9058 (IEEE Computer Society, 2018).

    Not in our library

  47. 47.

    Syed Arbaaz Q, Sriparna S, Mohammed H, Gael D, Erik C. Multitask representation learning for multimodal estimation of depression level. IEEE Intell. Syst. 2019;34:45-52.

    Not in our library10.1109/mis.2019.2925204

  48. 48.

    Jordan Y, William Y, Philipp T. Multimodal skin lesion classification using deep learning. Exp. Dermatol. 2018;27:1261-1267.

    Not in our library10.1111/exd.13777

  49. 49.
    MLMDA: a machine learning approach to predict and validate MicroRNA–disease associations by integrating of heterogenous information sources · 2019
    Not checked
  50. 50.

    Sivan K, et al. Predicting risk for Alcohol Use Disorder using longitudinal data with multimodal biomarkers and family history: A machine learning study. Mol. Psychiatry. 2021;26:1133-1141.

    Not in our library10.1038/s41380-019-0534-x

  51. 51.

    Mara Ten, K et al. MRI predictors of amyloid pathology: Results from the EMIF-AD Multimodal Biomarker Discovery study. Alzheimer’s Res. Ther.10, 100 (2018).

    Not in our library10.1186/s13195-018-0428-1

  52. 52.

    Isamu, H. et al. Radiogenomics predicts the expression of microRNA-1246 in the serum of esophageal cancer patients. Sci. Rep.10, 2532 (2020).

    Not in our library10.1038/s41598-020-59500-7

  53. 53.

    Jesus JC, Jianhua Y, Daniel JM. Enhancing image analytic tools by fusing quantitative physiological values with image features. J. Digit Imaging. 2012;25:550-557.

    Not in our library10.1007/s10278-011-9449-z

  54. 54.

    Kevin Bretonnel C, et al. Methodological issues in predicting pediatric epilepsy surgery candidates through natural language processing and machine learning. Biomed. Inform. Insights. 2016;8:BII.S38308.

    Not in our library10.4137/bii.s38308

  55. 55.

    Weiming, L. et al. Predicting Alzheimer’s disease conversion from mild cognitive impairment using an extreme learning machine-based grading method with multimodal data. Front. Aging Neurosci.12, 77 (2020).

    Not in our library10.3389/fnagi.2020.00077

  56. 56.

    Micah, C. et al. Predicting rehospitalization within 2 years of initial patient admission for a major depressive episode: A multimodal machine learning approach. Transl. Psychiatry9, 285 (2019).

    Not in our library10.1038/s41398-019-0615-2

  57. 57.

    Jongin K, Boreom L. Identification of Alzheimer’s disease and mild cognitive impairment using multimodal sparse hierarchical extreme learning machine. Hum. Brain Mapp. 2018;39:3728-3741.

    Not in our library10.1002/hbm.24207

  58. 58.

    Hélène De C, et al. Wearable monitoring and interpretable machine learning can objectively track progression in patients during cardiac rehabilitation. Sensors (Switz.). 2020;20:1-15.

    Not in our library10.3390/s20123601

  59. 59.

    Tamer, A., Shaker, E.-S. & Jose, M. A. Robust hybrid deep learning models for Alzheimer’s progression detection. Knowledge-Based Syst.213, 106688 (2021).

    Not in our library

  60. 60.

    Jeungchan L, et al. Machine learning-based prediction of clinical pain using multimodal neuroimaging and autonomic metrics. Pain. 2019;160:550-560.

    Not in our library10.1097/j.pain.0000000000001417

  61. 61.

    Uttam, K., Goo Rak, K. & Horacio, R.-G. An efficient combination among sMRI, CSF, cognitive score, and APOE ϵ 4 biomarkers for classification of AD and MCI using extreme learning machine. Comput. Intell. Neurosci.2020, 8015156 (2020).

    Not in our library10.1155/2020/8015156

  62. 62.

    Bo C, Mingxia L, Heung IS, Dinggang S, Daoqiang Z. Multimodal manifold-regularized transfer learning for MCI conversion prediction. Brain Imaging Behav. 2015;9:913-926.

    Not in our library10.1007/s11682-015-9356-x

  63. 63.

    Kevin, H., Ulrike, L., Markus, M. & Katja, B-B. Separating generalized anxiety disorder from major depression using clinical, hormonal, and structural MRI data: A multimodal machine learning study. Brain Behavior7, e00633 (2017).

    Not in our library10.1002/brb3.633

  64. 64.

    Fayao L, Luping Z, Chunhua S, Jianping Y. Multiple kernel learning in the primal for multimodal alzheimer’s disease classification. IEEE J. Biomed. Health Inform. 2014;18:984-990.

    Not in our library10.1109/jbhi.2013.2285378

  65. 65.

    Diego, C.-B. et al. Robust ensemble classification methodology for I123-Ioflupane SPECT images and multiple heterogeneous biomarkers in the diagnosis of Parkinson’s disease. Front. Neuroinform.12, 53 (2018).

    Not in our library10.3389/fninf.2018.00053

  66. 66.

    Yi, Z. et al. Predicting adverse drug reactions of combined medication from heterogeneous pharmacologic databases. BMC Bioinform.19, 517 (2018).

    Not in our library10.1186/s12859-018-2520-8

  67. 67.

    Chin Po C, Susan Shur Fen G, Chi Chun L. Toward differential diagnosis of autism spectrum disorder using multimodal behavior descriptors and executive functions. Comput. Speech Lang. 2019;56:17-35.

    Not in our library10.1016/j.csl.2018.12.003

  68. 68.

    Paolo, F. et al. Combining macula clinical signs and patient characteristics for age-related macular degeneration diagnosis: A machine learning approach Retina. BMC Ophthalmol.15, 10 (2015).

    Not in our library10.1186/1471-2415-15-10

  69. 69.

    Benjamin DW, et al. Early identification of epilepsy surgery candidates: A multicenter, machine learning study. Acta Neurol. Scand. 2021;144:41-50.

    Not in our library10.1111/ane.13418

  70. 70.

    Xia An B, Wenyan Z, Lou L, Zhaoxu X. Detecting risk gene and pathogenic brain region in EMCI using a novel GERF algorithm based on brain imaging and genetic data. IEEE J. Biomed. Health Inform. 2021;25:3019-3028.

    Not in our library10.1109/jbhi.2021.3067798

  71. 71.

    Prashanth R, Sumantra Dutta R, Pravat KM, Shantanu G. High-accuracy detection of early Parkinson’s disease through multimodal features and machine learning. Int. J. Med. Inf. 2016;90:13-21.

    Not in our library10.1016/j.ijmedinf.2016.03.001

  72. 72.

    Ali, A.-M. et al. Machine learning for localizing epileptogenic-zone in the temporal lobe: Quantifying the value of multimodal clinical-semiology and imaging concordance. Front. Digital Health3, 559103 (2021).

    Not in our library10.3389/fdgth.2021.559103

  73. 73.

    Baiying L, et al. Assessment of liver fibrosis in chronic hepatitis B via multimodal data. Neurocomputing. 2017;253:169-176.

    Not in our library10.1016/j.neucom.2016.09.128

  74. 74.

    Larry, H. et al. Multimodal tensor-based method for integrative and continuous patient monitoring during postoperative cardiac care. Artif. Intell. Med.113, 102032 (2021).

    Not in our library10.1016/j.artmed.2021.102032

  75. 75.

    Ivo, D. D. et al. Predictive big data analytics: A study of Parkinson’s disease using large, complex, heterogeneous, incongruent, multi-source and incomplete observations. PLoS One11, e0157077 (2016).

    Not in our library10.1371/journal.pone.0157077

  76. 76.

    Eleftherios, T., Ioannis, S., Apostolos, H. K. & Kostas, M. Deep radiotranscriptomics of non-small cell lung carcinoma for assessing molecular and histology subtypes with a data-driven analysis. Diagnostics11, 2383 (2021).

    Not in our library10.3390/diagnostics11122383

  77. 77.

    Hua, W. et al. Identifying disease sensitive and quantitative trait-relevant biomarkers from multidimensional heterogeneous imaging genetics data via sparse multimodal multitask learning. Bioinformatics28, i127–36 (2012).

    Not in our library10.1093/bioinformatics/bts228

  78. 78.

    Xia, B., Xi, H., Yiming, X. & Hao, W. A novel CERNNE approach for predicting Parkinson’s Disease-associated genes and brain regions based on multimodal imaging genetics data. Med. Image Anal.67, 101830 (2021).

    Not in our library10.1016/j.media.2020.101830

  79. 79.

    Vijay H, et al. Predicting complications in critical care using heterogeneous clinical data. IEEE Access. 2016;4:7988-8001.

    Not in our library10.1109/access.2016.2618775

  80. 80.

    Wang H, Li Y, Khan SA, Luo Y. Prediction of breast cancer distant recurrence using natural language processing and knowledge-guided convolutional neural network. Artif. Intell. Med. 2020;110:101977.

    Not in our library10.1016/j.artmed.2020.101977

  81. 81.

    Zeng, Z. et al. Identifying breast cancer distant recurrences from electronic health records using machine learning. J. Healthcare Inform. Res.3, 283–299 (2019).

    Not in our library10.1007/s41666-019-00046-3

  82. 82.

    Kautzky, A. et al. Machine learning classification of ADHD and HC by multimodal serotonergic data. Transl. Psychiatry10, 104 (2020).

    Not in our library10.1038/s41398-020-0781-2

  83. 83.
    Distinct multivariate brain morphological patterns and their added predictive value with cognitive and polygenic risk scores in mental disorders · 2017
    Not checked
  84. 84.

    Niha, B. et al. Radiogenomic analysis of hypoxia pathway is predictive of overall survival in Glioblastoma. Sci. Rep.8, 7 (2018).

    Not in our library10.1038/s41598-017-18310-0

  85. 85.

    Jan CP, et al. Combining multimodal imaging and treatment features improves machine learning-based prognostic assessment in patients with glioblastoma multiforme. Cancer Med. 2019;8:128-136.

    Not in our library10.1002/cam4.1908

  86. 86.

    Hao Z, et al. Machine learning reveals multimodal MRI patterns predictive of isocitrate dehydrogenase and 1p/19q status in diffuse low- and high-grade gliomas. J. Neurooncol. 2019;142:299-307.

    Not in our library10.1007/s11060-019-03096-0

  87. 87.

    Namyong, P. et al. Predicting acute kidney injury in cancer patients using heterogeneous and irregular data. PLoS One13, e0199839 (2018).

    Not in our library10.1371/journal.pone.0199839

  88. 88.

    Wei Liang T, Chee Kong C, Sim Heng O, Alvin Choong Meng N. Ensemble-based regression analysis of multimodal medical data for osteopenia diagnosis. Expert Syst. Appl. 2013;40:811-819.

    Not in our library10.1016/j.eswa.2012.08.031

  89. 89.

    Sébastien T, Yasser I-M, José María M-P, Alan CE, Louis De B. Defining a multimodal signature of remote sports concussions. Eur. J. Neurosci. 2017;46:1956-1967.

    Not in our library10.1111/ejn.13583

  90. 90.

    Gianluca, B. et al. Multimodal predictive modeling of endovascular treatment outcome for acute ischemic stroke using machine-learning. Stroke51, 3541–3551 (2020).

    Not in our library10.1161/strokeaha.120.030287

  91. 91.

    Yiming, X. et al. Explainable dynamic multimodal variational autoencoder for the prediction of patients with suspected central precocious puberty. IEEE J. Biomed. Health Inform.26, 1362–1373 (2021).

    Not in our library10.1109/jbhi.2021.3103271

  92. 92.

    Alan, D. K. et al. Mixture model framework for traumatic brain injury prognosis using heterogeneous clinical and outcome data. IEEE J. Biomed. Health Inform.26, 1285–1296 (2021).

    Not in our library10.1109/jbhi.2021.3099745

  93. 93.

    Casper, R. et al. Preoperative risk stratification in endometrial cancer (ENDORISK) by a Bayesian network model: A development and validation study. PLoS Med.17, e1003111 (2020)

    Not in our library10.1371/journal.pmed.1003111

  94. 94.

    Tommaso, G. et al. SARS-COV-2 comorbidity network and outcome in hospitalized patients in Crema, Italy. PLoS One16, e0248498 (2021).

    Not in our library10.1371/journal.pone.0248498

  95. 95.

    Huan Q, et al. Machine-learning radiomics to predict early recurrence in perihilar cholangiocarcinoma after curative resection. Liver Int. 2021;41:837-850.

    Not in our library10.1111/liv.14763

  96. 96.

    Ramon, C. et al. Alzheimer’s disease risk assessment using large-scale machine learning methods. PLoS One8, e77949 (2013).

    Not in our library10.1371/journal.pone.0077949

  97. 97.
    Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Data · 2019
    No notice
  98. 98.

    Michael JD, et al. Development and validation of a novel automated Gleason grade and molecular profile that define a highly predictive prostate cancer progression algorithm-based test. Prostate Cancer Prostatic Dis. 2018;21:594-603.

    Not in our library10.1038/s41391-018-0067-4

  99. 99.

    Perotte A, Ranganath R, Hirsch JS, Blei D, Elhadad N. Risk prediction for chronic kidney disease progression using heterogeneous electronic health record data and time series analysis. J. Am. Med. Inf. Assoc. 2015;22:872-880.

    Not in our library10.1093/jamia/ocv024

  100. 100.

    Lei Y, Yalin W, Paul MT, Vaibhav AN, Jieping Y. Multi-source feature learning for joint analysis of incomplete multiple heterogeneous neuroimaging data. Neuroimage. 2012;61:622-632.

    Not in our library10.1016/j.neuroimage.2012.03.059

  101. 101.

    Yanbo, X., Siddharth, B., Shriprasad, R. D., Kevin, O. M. & Jimeng, S. RAIM: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2565–2573 (Association for Computing Machinery, 2018).

    Not in our library

  102. 102.

    Yixue H, Mohd U, Jun Y, Hossain MS, Ahmed G. Recurrent convolutional neural network based multimodal disease risk prediction. Future Gener. Computer Syst. 2019;92:76-83.

    Not in our library10.1016/j.future.2018.09.031

  103. 103.
    Artificial intelligence-enabled fully automated detection of cardiac amyloidosis using electrocardiograms and echocardiograms · 2021
    No notice
  104. 104.

    Rui, Y. et al. Richer fusion network for breast cancer classification based on multimodal data. BMC Med. Inform. Decis. Mak.21, 134 (2021).

    Not in our library10.1186/s12911-020-01340-6

  105. 105.

    Li, Y., Wang, H. & Luo, Y. A comparison of pre-trained vision-and-language models for multimodal representation learning across medical images and reports. 2020 IEEE Int. Conf. Bioinform. Biomedicine (BIBM) 2020 1999–2004 (2020).

    Not in our library

  106. 106.
    A machine learning model to predict the risk of 30-day readmissions in patients with heart failure: a retrospective analysis of electronic medical records data · 2018
    No notice
  107. 107.

    Chowdhury, S., Zhang, C., Yu, P. S. & Luo, Y. Mixed pooling multi-view attention autoencoder for representation learning in healthcare. Preprint at https://arxiv.org/abs/1910.06456 (2019).

    Not in our library

  108. 108.
    An unsupervised learning approach to identify novel signatures of health and disease from multimodal data · 2020
    Not checked
  109. 109.

    Chowdhury S, Zhang C, Yu PS, Luo Y. Med2Meta: Learning representations of medical concepts with meta-embeddings. HEALTHINF. 2020;2020:369-376.

    Not in our library

  110. 110.

    Subramanian V, Do MN, Syeda-Mahmood T. Multimodal fusion of imaging and genomics for lung cancer recurrence prediction, IEEE 17th International Symposium on Biomedical Imaging (ISBI), 804–808, (2020).

    Not in our library

  111. 111.

    Michele D, et al. Combining heterogeneous data sources for neuroimaging based diagnosis: Re-weighting and selecting what is important. Neuroimage. 2019;195:215-231.

    Not in our library10.1016/j.neuroimage.2019.01.053

  112. 112.

    Yiwen M, William S, Michael KO, Corey WA. Bidirectional representation learning from transformers using multimodal electronic health record data to predict depression. IEEE J. Biomed. Health Inform. 2021;25:3121-3129.

    Not in our library10.1109/jbhi.2021.3063721

  113. 113.

    Xing, T. et al. Elaboration of a multimodal MRI-based radiomics signature for the preoperative prediction of the histological subtype in patients with non-small-cell lung cancer. BioMed. Engineering Online19, 5 (2020).

    Not in our library10.1186/s12938-019-0744-0

  114. 114.
    Predicting MCI Status From Multimodal Language Data Using Cascaded Classifiers · 2019
    Not checked
  115. 115.

    Amir Hossein, Y. et al. Multimodal mental health analysis in social media. PLoS One15, e0226248 (2020).

    Not in our library10.1371/journal.pone.0226248

  116. 116.
    Predicting mortality in critically ill patients with diabetes using machine learning and clinical notes · 2020
    Not checked
  117. 117.

    Navodini W, Mobarakol I, Hongliang R. Radiogenomics model for overall survival prediction of glioblastoma. Med Biol. Eng. Comput. 2020;58:1767-1777.

    Not in our library10.1007/s11517-020-02179-9

  118. 118.

    Solale, T. et al. A distributed multitask multimodal approach for the prediction of Alzheimer’s disease in a longitudinal study. Neuroimage206, 116317 (2020).

    Not in our library10.1016/j.neuroimage.2019.116317

  119. 119.

    Luo Y, et al. Integrating hypertension phenotype and genotype with hybrid non-negative matrix factorization. Bioinformatics. 2019;35:1395-403.

    Not in our library10.1093/bioinformatics/bty804

  120. 120.

    Jae Hyun Y, Johanna Inhyang K, Bung Nyun K, Bumseok J. Exploring characteristic features of attention-deficit/hyperactivity disorder: Findings from multi-modal MRI and candidate genetic data. Brain Imaging Behav. 2020;14:2132-2147.

    Not in our library10.1007/s11682-019-00164-x

  121. 121.

    Chao, T., Baoyu, L., Jun, L. & Zhigao, Z. A Deep automated skeletal bone age assessment model with heterogeneous features learning. J. Med. Syst.42, 249 (2018).

    Not in our library10.1007/s10916-018-1091-6

  122. 122.

    Cheng, C. et al. Improving protein–protein interactions prediction accuracy using XGBoost feature selection and stacked ensemble classifier. Comput. Biol. Med.123 (2020).

    Not in our library10.1016/j.compbiomed.2020.103899

  123. 123.
    Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer · 2020
    Corrected
  124. 124.

    Juan Camilo V-C, et al. Multimodal assessment of Parkinson’s disease: A deep learning approach. IEEE J. Biomed. Health Inform. 2019;23:1618-1630.

    Not in our library10.1109/jbhi.2018.2866873

  125. 125.

    Ping, Z. et al. Deep-learning radiomics for discrimination conversion of Alzheimer’s disease in patients with mild cognitive impairment: A study based on 18F-FDG PET imaging. Front. Aging Neurosci.13, (2021).

    Not in our library10.3389/fnagi.2021.764872

  126. 126.

    Shin, J., Li, Y. & Luo, Y. Early prediction of mortality in critical care setting in sepsis patients using structured features and unstructured clinical notes. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 2885–2890 (IEEE, 2021).

    Not in our library

  127. 127.

    Jayachitra, V. P., Nivetha, S., Nivetha, R. & Harini, R. A cognitive IoT-based framework for effective diagnosis of COVID-19 using multimodal data. Biomed. Signal Processing Control70, 102960 (2021).

    Not in our library10.1016/j.bspc.2021.102960

  128. 128.

    Thomas, L. et al. An explainable multimodal neural network architecture for predicting epilepsy comorbidities based on administrative claims data. Front. Artificial Intelligence4, 610197 (2021).

    Not in our library10.3389/frai.2021.610197

  129. 129.

    Alan Baronio, M., Carla Diniz Lopes, B. & Silvio Cesar, C. Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data. Health Inform. Sci. Syst.9, 20 (2021)

    Not in our library10.1007/s13755-021-00151-x

  130. 130.
    Multimodal deep learning models for early detection of Alzheimer’s disease stage · 2021
    No notice
  131. 131.

    Cam Hao, H. et al. Bimodal learning via trilogy of skip-connection deep networks for diabetic retinopathy risk progression identification. Int. J. Med. Inform.132, 103926 (2019).

    Not in our library10.1016/j.ijmedinf.2019.07.005

  132. 132.

    Yucheng T, et al. Prediction of type II diabetes onset with computed tomography and electronic medical records. Lect. Notes Comput. Sci. 2020;12445:13-23.

    Not in our library10.1007/978-3-030-60946-7_2

  133. 133.

    Rui Y, et al. Integration of multimodal data for breast cancer classification using a hybrid deep learning method. Lect. Notes Computer Sci. 2019;11643:460-469.

    Not in our library10.1007/978-3-030-26763-6_44

  134. 134.

    Batuhan, B. & Mehmet, T. Improving clinical outcome predictions using convolution over medical entities with multimodal learning. Artif. Intell. Med.117, 102112 (2021).

    Not in our library10.1016/j.artmed.2021.102112

  135. 135.

    Esra, Z. et al. Multimodal fusion strategies for outcome prediction in Stroke. In HEALTHINF 2020 - 13th International Conference on Health Informatics, Proceedings; Part of 13th International Joint Conference on Biomedical Engineering Systems and Technologies, BIOSTEC 2020 421–428 (2020).

    Not in our library

  136. 136.

    Rimma P, et al. Learning probabilistic phenotypes from heterogeneous EHR data. J. Biomed. Inform. 2015;58:156-165.

    Not in our library10.1016/j.jbi.2015.10.001

  137. 137.

    Leon MA, et al. Modeling longitudinal imaging biomarkers with parametric Bayesian multi-task learning. Hum. Brain Mapp. 2019;40:3982-4000.

    Not in our library10.1002/hbm.24682

  138. 138.

    Paris Alexandros L, et al. Heterogeneity and classification of recent onset psychosis and depression: A multimodal machine learning approach. Schizophr. Bull. 2021;47:1130-1140.

    Not in our library10.1093/schbul/sbaa185

  139. 139.

    Nikolaos K, et al. Prediction models of functional outcomes for individuals in the clinical high-risk state for psychosis or with recent-onset depression: A multimodal, multisite machine learning analysis. JAMA Psychiatry. 2018;75:1156-1172.

    Not in our library10.1001/jamapsychiatry.2018.2165

  140. 140.

    Dongdong S, Minghui W, Ao L. A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data. IEEE/ACM Trans. Comput Biol. Bioinform. 2019;16:841-850.

    Not in our library10.1109/tcbb.2018.2806438

  141. 141.

    Karen, S. A. et al. A machine-learning framework for robust and reliable prediction of short- and long-term treatment response in initially antipsychotic-naïve schizophrenia patients based on multimodal neuropsychiatric data. Translational Psychiatry10, 276 (2020).

    Not in our library10.1038/s41398-020-00962-8

  142. 142.

    Sun M, et al. Early prediction of acute kidney injury in critical care setting using clinical notes and structured multivariate physiological measurements. Stud. Health Technol. Inf. 2019;264:368-372.

    Not in our library10.3233/shti190245

  143. 143.

    Dennis, S. R., Simuni, T. & Luo, Y. A predictive model for Parkinson’s disease reveals candidate gene sets for progression subtype. 2020 IEEE Int. Conf. Bioinforma. Biomedicine (BIBM) 2020, 417–420 (2020).

    Not in our library

  144. 144.
    Accurately Differentiating Between Patients With COVID-19, Patients With Other Viral Infections, and Healthy Individuals: Multimodal Late Fusion Learning Approach · 2021
    Not checked
  145. 145.

    Min C, Yixue H, Kai H, Lin W, Lu W. Disease prediction by machine learning over big data from healthcare communities. IEEE Access. 2017;5:8869-8879.

    Not in our library10.1109/access.2017.2694446

  146. 146.

    Keyang X, et al. Multimodal machine learning for automated ICD coding. Proc. Mach. Learn. Res. 2019;106:1-17.

    Not in our library

  147. 147.

    Liuqing Y, et al. Deep learning based multimodal progression modeling for Alzheimer’s disease. Stat. Biopharma. Res. 2021;13:337-343.

    Not in our library10.1080/19466315.2021.1884129

  148. 148.

    Peng L, et al. A radiogenomics signature for predicting the clinical outcome of bladder urothelial carcinoma. Eur. Radio. 2020;30:547-557.

    Not in our library10.1007/s00330-019-06371-w

  149. 149.

    Md Sirajus, S. et al. Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment. Comput. Biol. Med.129, 104150 (2021).

    Not in our library10.1016/j.compbiomed.2020.104150

  150. 150.

    Jian X, et al. Multimodal machine learning using visual fields and peripapillary circular OCT scans in detection of glaucomatous optic neuropathy. Ophthalmology. 2021;129:171-180.

    Not in our library10.1016/j.ophtha.2021.07.032

  151. 151.

    Dai Y, Yiqi Z, Yang W, Wenpu Z, Xiaoming H. Auxiliary diagnosis of heterogeneous data of Parkinson’s disease based on improved convolution neural network. Multimed. Tools Appl. 2020;79:24199-24224.

    Not in our library10.1007/s11042-020-08984-6

  152. 152.

    Makoto, N. et al. Accessory pathway analysis using a multimodal deep learning model. Sci. Rep.11, 8045 (2021).

    Not in our library10.1038/s41598-021-87631-y

  153. 153.

    Wenhuan Z, Anupam G, Daniel HH. On the application of advanced machine learning methods to analyze enhanced, multimodal data from persons infected with covid-19. Computation. 2021;9:1-15.

    Not in our library

  154. 154.

    Jeremy AT, Kit ML, Marta IG. Multi-dimensional predictions of psychotic symptoms via machine learning. Hum. Brain Mapp. 2020;41:5151-5163.

    Not in our library10.1002/hbm.25181

  155. 155.

    Hossam, F., Maria, H., Mohammad, F., Haya, E. & Alaa, A. An intelligent multimodal medical diagnosis system based on patients’ medical questions and structured symptoms for telemedicine. Inform. Med. Unlocked23, 100513 (2021).

    Not in our library

  156. 156.

    Md Ashad A, et al. A kernel machine method for detecting higher order interactions in multimodal datasets: Application to schizophrenia. J. Neurosci. Methods. 2018;309:161-174.

    Not in our library10.1016/j.jneumeth.2018.08.027

  157. 157.

    Luo Y, et al. A multidimensional precision medicine approach identifies an autism subtype characterized by dyslipidemia. Nat. Med. 2020;26:1375-1379.

    Not in our library10.1038/s41591-020-1007-0

  158. 158.
    A multilayer multimodal detection and prediction model based on explainable artificial intelligence for Alzheimer’s disease · 2021
    No notice
  159. 159.
    Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection · 2020
    No notice
  160. 160.
    Clinical text classification with rule-based features and knowledge-guided convolutional neural networks · 2019
    Not checked
  161. 161.
    Using clinical Natural Language Processing for health outcomes research: Overview and actionable suggestions for future advances · 2018
    No notice
  162. 162.

    Luo Y, Uzuner Ö, Szolovits P. Bridging semantics and syntax with graph algorithms—state-of-the-art of extracting biomedical relations. Brief. Bioinform. 2016;18:160-178.

    Not in our library10.1093/bib/bbw001

  163. 163.

    Zeng Z, Deng Y, Li X, Naumann T, Luo Y. Natural language processing for EHR-based computational phenotyping. IEEE/ACM Trans. Comput Biol. Bioinform. 2018;16:139-53.

    Not in our library10.1109/tcbb.2018.2849968

  164. 164.
    Multimodal Machine Learning Workflows for Prediction of Psychosis in Patients With Clinical High-Risk Syndromes and Recent-Onset Depression · 2021
    No notice
  165. 165.

    Petersen RC, et al. Alzheimer’s disease neuroimaging initiative (ADNI): Clinical characterization. Neurology. 2010;74:201-209.

    Not in our library10.1212/wnl.0b013e3181cb3e25

  166. 166.

    Weinstein JN, et al. The cancer genome atlas pan-cancer analysis project. Nat. Genet. 2013;45:1113.

    Not in our library10.1038/ng.2764

  167. 167.
    Key challenges for delivering clinical impact with artificial intelligence · 2019
    No notice
  168. 168.

    Michael AM, et al. Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images. J. Am. Acad. Dermatol. 2018;78:270-7.e1.

    Not in our library10.1016/j.jaad.2017.08.016

  169. 169.

    Jannin, P. G. C. & Gibaud, B. Medical Applications of NDT Data Fusion (Springer, 2001).

    Not in our library

  170. 170.
    Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond · 2022
    No notice
  171. 171.

    Zhang, Z., & Sejdić, E., Radiological images and machine learning: trends, perspectives, and prospects. Comput. Biol. Med.108, 354–370 (2019).

    Not in our library10.1016/j.compbiomed.2019.02.017

  172. 172.

    David, L., Enrico, C., Jessica, C., Parina, S. & Farah, M. How machine learning is embedded to support clinician decision making: An analysis of FDA-approved medical devices. BMJ Health Care Inform.28, e100301 (2021).

    Not in our library10.1136/bmjhci-2020-100301

  173. 173.

    Luo Y, Szolovits P, Dighe AS, Baron JM. Using machine learning to predict laboratory test results. Am. J. Clin. Pathol. 2016;145:778-788.

    Not in our library10.1093/ajcp/aqw064

  174. 174.

    Thakur, S., Choudhary, J. & Singh, D. P. Systems 435–443 (Springer, 2021).

    Not in our library

  175. 175.

    Luo Y, Szolovits P, Dighe AS, Baron JM. 3D-MICE: Integration of cross-sectional and longitudinal imputation for multi-analyte longitudinal clinical data. J. Am. Med. Inform. Assoc. (JAMIA). 2017;25:645-653.

    Not in our library10.1093/jamia/ocx133

  176. 176.

    Xue, Y., Klabjan, D. & Luo, Y. Mixture-based multiple imputation model for clinical data with a temporal dimension. 2019 IEEE International Conference on Big Data (Big Data)2019 245–252 (IEEE, 2019).

    Not in our library

  177. 177.

    Cao, W. et al. Brits: Bidirectional Recurrent Rmputation for Time Series. NeurIPS, 31 (2018).

    Not in our library

  178. 178.

    Luo Y. Evaluating the state of the art in missing data imputation for clinical data. Brief. Bioinform. 2022;23:bbab489.

    Not in our library10.1093/bib/bbab489

  179. 179.

    Zhao Q, Adeli E, Pohl KM. Training confounder-free deep learning models for medical applications. Nat. Commun. 2020;11:1-9.

    Not in our library10.1038/s41467-020-19784-9

  180. 180.

    Luo, Y. & Mao, C. ScanMap: Supervised confounding aware non-negative matrix factorization for polygenic risk modeling. In Machine Learning for Healthcare Conference; 2020: PMLR 27–45 (2020).

    Not in our library

  181. 181.

    Kalavathy R, Suresh RM. Pharmacovigilance from electronic medical records to report adverse events. J. Chem. Pharm. Sci. 2015;2015:188-191.

    Not in our library

  182. 182.

    Luo, Y. et al. Natural language processing for EHR-based pharmacovigilance: A structured review. Drug Saf. 10.1007/s40264-017-0558-6 (2017).

    Not in our library10.1007/s40264-017-0558-6

  183. 183.

    Segura Bedmar, I., Martínez, P. & Herrero Zazo, M. Semeval-2013 task 9: Extraction of drug–drug interactions from biomedical texts (ddiextraction 2013). 2013: Association for Computational Linguistics (2013).

    Not in our library

  184. 184.

    Hammann F, Drewe J. Data mining for potential adverse drug–drug interactions. Expert Opin. Drug Metab. Toxicol. 2014;10:665-671.

    Not in our library10.1517/17425255.2014.894507

  185. 185.

    Donna MF, et al. American geriatrics society 2019 updated AGS beers criteria for potentially inappropriate medication use in older adults. J. Am. Geriatr. Soc. 2019;67:674-694.

    Not in our library10.1111/jgs.15767

  186. 186.

    Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366:447-53.

    Not in our library10.1126/science.aax2342

  187. 187.

    Wang H, et al. Using machine learning to integrate sociobehavioral factors in predicting cardiovascular-related mortality risk. Stud. Health Technol. Inf. 2019;264:433-437.

    Not in our library10.3233/shti190258

  188. 188.

    Christof E. Open source software in industry. IEEE Softw. 2008;25:52-53.

    Not in our library10.1109/ms.2008.67

  189. 189.

    Robert MS. Why develop open-source software? The role of non-pecuniary benefits, monetary rewards, and open-source licence type. Oxf. Rev. Economic Policy. 2007;23:605-619.

    Not in our library10.1093/oxrep/grm034

  190. 190.

    Luo Y, Wunderink RG, Lloyd-Jones D. Proactive vs reactive machine learning in health care: Lessons from the COVID-19 pandemic. JAMA. 2022;327:623-624.

    Not in our library10.1001/jama.2021.24935

  191. 191.

    Ke, G. et al. LightGBM: A highly efficient gradient boosting decision tree. Adv. Neural Inf. Process Syst.30 (2017).

    Not in our library

  192. 192.
    Diagnosis of Diabetes Mellitus Using Gradient Boosting Machine (LightGBM) · 2021
    No notice
  193. 193.

    Xiaolei S, Mingxi L, Zeqian S. A novel cryptocurrency price trend forecasting model based on LightGBM. Financ. Res. Lett. 2020;32:101084.

    Not in our library10.1016/j.frl.2018.12.032

  194. 194.

    Dongzi J, Yiqin L, Jiancheng Q, Zhe C, Zhongshu M. SwiftIDS: Real-time intrusion detection system based on LightGBM and parallel intrusion detection mechanism. Comput. Secur. 2020;97:101984.

    Not in our library10.1016/j.cose.2020.101984

  195. 195.
    The FAIR Guiding Principles for scientific data management and stewardship · 2016
    No notice
  196. 196.
    The role of machine learning in clinical research: transforming the future of evidence generation · 2021
    Corrected
  197. 197.
    Artificial intelligence and machine learning in clinical development: a translational perspective · 2019
    No notice
  198. 198.

    Inna, K. & Simeon, S. Interpretability of machine learning solutions in public healthcare: The CRISP-ML approach. Front. Big Data4, 660206 (2021).

    Not in our library10.3389/fdata.2021.660206

  199. 199.

    Marx V. Method of the Year: Spatially resolved transcriptomics. Nat. Methods. 2021;18:9-14.

    Not in our library10.1038/s41592-020-01033-y

  200. 200.
    Statistical and machine learning methods for spatially resolved transcriptomics data analysis · 2022
    No notice
  201. 201.

    Longo SK, Guo MG, Ji AL, Khavari PA. Integrating single-cell and spatial transcriptomics to elucidate intercellular tissue dynamics. Nat. Rev. Genet. 2021;22:627-44.

    Not in our library10.1038/s41576-021-00370-8

  202. 202.

    PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann. Intern. Med.169 467–473 (2018).

    Not in our library10.7326/m18-0850

Cited by papers we hold 5

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.