Transformer-CNN: Swiss knife for QSAR modeling and interpretation

Abstract

We present SMILES-embeddings derived from the internal encoder state of a Transformer [1] model trained to canonize SMILES as a Seq2Seq problem. Using a CharNN [2] architecture upon the embeddings results in higher quality interpretable QSAR/QSPR models on diverse benchmark datasets including regression and classification tasks. The proposed Transformer-CNN method uses SMILES augmentation for training and inference, and thus the prognosis is based on an internal consensus. That both the augmentation and transfer learning are based on embeddings allows the method to provide good results for small datasets. We discuss the reasons for such effectiveness and draft future directions for the development of the method. The source code and the embeddings needed to train a QSAR model are available on https://github.com/bigchem/transformer-cnn . The repository also has a standalone program for QSAR prognosis which calculates individual atoms contributions, thus interpreting the model’s result. OCHEM [3] environment ( https://ochem.eu ) hosts the on-line implementation of the method proposed.

References 65

1. 1.

   Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. Paper presented at the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. arXiv:1706.03762

   Not in our library

2. 2.

   Zhang X, Zhao J, LeCun Y (2015) Character-level convolutional networks for text classification. arXiv e-prints. arXiv:1509.01626

   Not in our library

3. 3.

   Sushko I, Novotarskyi S, Körner R, et al. Online chemical modeling environment (OCHEM): web platform for data storage, model development and publishing of chemical information. J Comput Aided Mol Des. 2011;25:533-554.

   Not in our library10.1007/s10822-011-9440-2 (opens in a new tab)

4. 4.

   Mauri A, Consonni V, Pavan M, Todeschini R. Dragon software: an easy approach to molecular descriptor calculations. Match. 2006;56:237-248.

   Not in our library

5. 5.

   Baskin I, Varnek A. Fragment descriptors in SAR/QSAR/QSPR studies, molecular similarity analysis and in virtual screening. Chemoinformatics approaches to virtual screening. 2008;1-43.

   Not in our library

6. 6.

   Eklund M, Norinder U, Boyer S, Carlsson L. Choosing feature selection and learning algorithms in QSAR. J Chem Inf Model. 2014;54:837-843.

   Not in our library10.1021/ci400573c (opens in a new tab)

7. 7.

   Baskin II, Winkler D, Tetko IV. A renaissance of neural networks in drug discovery. Expert Opin Drug Discov. 2016;11:785-795.

   Not in our library10.1080/17460441.2016.1201262 (opens in a new tab)

8. 8.

   Duvenaud D, Maclaurin D, Aguilera-Iparraguirre J, et al (2015) Convolutional networks on graphs for learning molecular fingerprints. arXiv e-prints. arXiv:1509.09292

   Not in our library

9. 9.

   Coley CW, Barzilay R, Green WH, et al. Convolutional embedding of attributed molecular graphs for physical property prediction. J Chem Inf Model. 2017;57:1757-1772.

   Not in our library10.1021/acs.jcim.6b00601 (opens in a new tab)

10. 10.
    Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules · 2018

    Not checked
11. 11.

    Kimber TB, Engelke S, Tetko IV, et al (2018) Synergy effect between convolutional neural networks and the multiplicity of smiles for improvement of molecular prediction. arXiv e-prints. arXiv:1812.04439

    Not in our library

12. 12.

    Gilmer J, Schoenholz SS, Riley PF, et al (2017) Neural message passing for quantum chemistry. Proceedings of the 34 th International conference on machine learning, Sydney, Australia, PMLR 70. arXiv:1704.01212

    Not in our library

13. 13.

    Shang C, Liu Q, Chen K-S, et al (2018) Edge attention-based multi-relational graph convolutional networks. arXiv e-prints. arXiv:1802.04944

    Not in our library

14. 14.

    Weininger D. SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. J Chem Inf Comput Sci. 1988;28:31-36.

    Not in our library

15. 15.
    Open Babel: An open chemical toolbox · 2011

    No notice
16. 16.

    Vidal D, Thormann M, Pons M. LINGO, an efficient holographic text based method to calculate biophysical properties and intermolecular similarities. J Chem Inf Model. 2005;45:386-393.

    Not in our library10.1021/ci0496797 (opens in a new tab)

17. 17.

    Zhang X, LeCun Y (2015) Text understanding from scratch. arXiv e-prints. arXiv:1502.01710

    Not in our library

18. 18.

    Goh GB, Hodas NO, Siegel C, Vishnu A (2017) SMILES2Vec: an interpretable general-purpose deep neural network for predicting chemical properties. arXiv e-prints. arXiv:1712.02034

    Not in our library

19. 19.

    Jastrzębski S, Leśniak D, Czarnecki WM (2016) Learning to SMILE(S). arXiv e-prints. arXiv:1602.06289

    Not in our library

20. 20.

    Goh GB, Siegel C, Vishnu A, Hodas NO (2017) Using rule-based labels for weak supervised learning: a chemnet for transferable chemical property prediction. arXiv e-prints. arXiv:1712.02734

    Not in our library

21. 21.

    Zheng S, Yan X, Yang Y, Xu J. Identifying structure-property relationships through SMILES syntax analysis with self-attention mechanism. J Chem Inf Model. 2019;59:914-923.

    Not in our library10.1021/acs.jcim.8b00803 (opens in a new tab)

22. 22.

    Tetko IV, Karpov P, Bruno E, Kimber TB, Godin G. Augmentation Is What You Need! In: Tetko IV, Karpov P, Kurkova V (ed) 28th International Conference on Artificial Neural Networks Munich, Germany, 2019 Sep 17, Proceedings, Part V, Workshop and Special sessions, Springer, Cham, pp 831–835

    Not in our library

23. 23.

    Kiela D, Bottou L (2014) Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In: Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP). pp 36–45

    Not in our library

24. 24.

    Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. EMNLP

    Not in our library

25. 25.

    Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science. 2006;313:504-507.

    Not in our library10.1126/science.1127647 (opens in a new tab)

26. 26.
    InChI - the worldwide chemical structure identifier standard · 2013

    No notice
27. 27.
    Learning continuous and data-driven molecular descriptors by translating equivalent chemical representations · 2019

    No notice
28. 28.

    Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9:1735-1780.

    Not in our library10.1162/neco.1997.9.8.1735 (opens in a new tab)

29. 29.

    Schwaller P, et al. Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction. ACS Cent Sci. 2019;5:1572-1583.

    Not in our library10.1021/acscentsci.9b00576 (opens in a new tab)

30. 30.

    Karpov P, Godin G, Tetko IV. A transformer model for retrosynthesis. In: Tetko IV, Theis F, Karpov P, Kurkova V (ed) 28th International Conference on artificial neural networks, Munich, Germany, September 17–19, 2019 Proceedings, Part V, Workshop and Special sessions. Springer

    Not in our library

31. 31.

    Samek W, Müller K-R, Samek W, Montavon G, Vedaldi A, et al. Towards explainable artificial intelligence. Explainable AI: interpreting, explaining and visualizing deep learning. 2019;5-22.

    Not in our library

32. 32.

    Montavon G, Binder A, Lapuschkin S, Samek W, Montavon G, Vedaldi A, et al. Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning. 2019;193-209.

    Not in our library

33. 33.

    Tetko IV, Villa AE, Livingstone DJ. Neural network studies. 2. Variable selection. J Chem Inf Comput Sci. 1996;36:794-803.

    Not in our library10.1021/ci950204c (opens in a new tab)

34. 34.

    Gaulton A, Bellis LJ, Bento AP, et al. ChEMBL: a large-scale bioactivity database for drug discovery. Nucleic Acids Res. 2012;40:D1100-D1107.

    Not in our library10.1093/nar/gkr777 (opens in a new tab)

35. 35.

    Segler MHS, Kogej T, Tyrchan C, Waller MP (2017) Generating focussed molecule libraries for drug discovery with recurrent neural networks

    Not in our library10.1021/acscentsci.7b00512 (opens in a new tab)

36. 36.

    Gupta A, Múller AT, Huisma BJH, et al. Generative recurrent networks for de novo drug design. Mol Inform. 2018;37:1700111.

    Not in our library10.1002/minf.201700111 (opens in a new tab)

37. 37.

    Rush A (2018) The annotated transformer. In: Proceedings of workshop for NLP open source software (NLP-OSS). pp 52–60

    Not in our library

38. 38.

    Abadi M, Barham P, Chen J, et al (2016) TensorFlow: a system for large-scale machine learning

    Not in our library

39. 39.

    Landrum G RDKit: Open-source cheminformatics. https://www.rdkit.org

    Not in our library

40. 40.

    Ramsundar B, Eastman P, Walters P, Pande V. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. 2019.

    Not in our library

41. 41.

    Srivastava N, Hinton G, Krizhevsky A, et al. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15:1929-1958.

    Not in our library

42. 42.

    Srivastava RK, Greff K, Schmidhuber J (2015) Highway Networks. Paper presented at the Deep Learning Workshop, International Conference on Machine Learning, Lille, France. arXiv:1505.00387

    Not in our library

43. 43.

    Tetko IV, Karpov P, Bruno E, et al (2019) Augmentation Is What You Need!: 28th International Conference on artificial neural networks, Munich, Germany, September 17–19, 2019, Proceedings. In: Tetko IV, Kůrková V, Karpov P, Theis F (eds) Artificial neural networks and machine learning–ICANN 2019: workshop and special sessions. Springer International Publishing, Cham, pp 831–835

    Not in our library

44. 44.

    Draper NR, Smith H. Applied regression analysis. 2014.

    Not in our library

45. 45.

    Tetko IV, Sushko Y, Novotarskyi S, et al. How accurately can we predict the melting points of drug-like compounds? J Chem Inf Model. 2014;54:3320-3329.

    Not in our library10.1021/ci5005288 (opens in a new tab)

46. 46.

    Wu Z, Ramsundar B, Feinberg EN, et al. MoleculeNet: a benchmark for molecular machine learning. Chem Sci. 2018;9:513-530.

    Not in our library10.1039/c7sc02664a (opens in a new tab)

47. 47.

    Brandmaier S, Sahlin U, Tetko IV, Öberg T. PLS-optimal: a stepwise d-optimal design based on latent variables. J Chem Inf Model. 2012;52:975-983.

    Not in our library10.1021/ci3000198 (opens in a new tab)

48. 48.

    Sushko I, Novotarskyi S, Körner R, et al. Applicability domains for classification problems: benchmarking of distance to models for ames mutagenicity set. J Chem Inf Model. 2010;50:2094-2111.

    Not in our library10.1021/ci100253r (opens in a new tab)

49. 49.

    Tetko IV, Tanchuk VY, Kasheva TN, Villa AEP. Estimation of aqueous solubility of chemical compounds using e-state indices. J Chem Inf Comput Sci. 2001;41:1488-1493.

    Not in our library10.1021/ci000392t (opens in a new tab)

50. 50.

    Huuskonen JJ, Livingstone DJ, Tetko IV. Neural network modeling for estimation of partition coefficient based on atom-type electrotopological state indices. J Chem Inf Comput Sci. 2000;40:947-955.

    Not in our library10.1021/ci9904261 (opens in a new tab)

51. 51.

    Suzuki K, Nakajima H, Saito Y, et al. Janus kinase 3 (Jak3) is essential for common cytokine receptor γ chain (γc)-dependent signaling: comparative analysis of γc, Jak3, and γc and Jak3 double-deficient mice. Int Immunol. 2000;12:123-132.

    Not in our library10.1093/intimm/12.2.123 (opens in a new tab)

52. 52.

    Sutherland JJ, Weaver DF. Three-dimensional quantitative structure-activity and structure-selectivity relationships of dihydrofolate reductase inhibitors. J Comput Aided Mol Des. 2004;18:309-331.

    Not in our library10.1023/b:jcam.0000047814.85293.da (opens in a new tab)

53. 53.

    Vorberg S, Tetko IV. Modeling the biodegradability of chemical compounds using the online chemical modeling environment (OCHEM). Mol Inform. 2014;33:73-85.

    Not in our library10.1002/minf.201300030 (opens in a new tab)

54. 54.

    Novotarskyi S, Abdelaziz A, Sushko Y, et al. ToxCast EPA in vitro to in vivo challenge: insight into the rank-I model. Chem Res Toxicol. 2016;29:768-775.

    Not in our library10.1021/acs.chemrestox.5b00481 (opens in a new tab)

55. 55.

    Rybacka A, Rudén C, Tetko IV, Andersson PL. Identifying potential endocrine disruptors among industrial chemicals and their metabolites – development and evaluation of in silico tools. Chemosphere. 2015;139:372-378.

    Not in our library10.1016/j.chemosphere.2015.07.036 (opens in a new tab)

56. 56.

    Xia Z, Karpov P, Popowicz G, Tetko IV. Focused library generator: case of Mdmx inhibitors. J Comp Aided Mol Des. 2019;1:1.

    Not in our library10.1007/s10822-019-00242-8 (opens in a new tab)

57. 57.

    Chang C-C, Lin C-J. LIBSVM: A library for support vector machines. ACM Transact Int Syst Technol. 2011;2:27.

    Not in our library

58. 58.

    Breiman L. Random forests. Mach Learn. 2001;45:5-32.

    Not in our library

59. 59.

    Chen T, Guestrin C (2016) XGBoost: A scalable tree boosting system. arXiv [cs.LG]

    Not in our library

60. 60.

    Tetko IV. Associative neural network. Neural Process Lett. 2002;16:187-199.

    Not in our library

61. 61.

    Sosnin S, Karlov D, Tetko IV, Fedorov MV. Comparative study of multitask toxicity modeling on a broad chemical space. J Chem Inf Model. 2019;59:1062-1072.

    Not in our library10.1021/acs.jcim.8b00685 (opens in a new tab)

62. 62.

    Arras L, Montavon G, Müller K-R, Samek W (2017) Explaining recurrent neural network predictions in sentiment analysis. Proceedings of the 8th workshop on computational approaches to subjectivity, sentiment and social media analysis

    Not in our library

63. 63.

    Plošnik A, Vračko M, Dolenc MS. Mutagenic and carcinogenic structural alerts and their mechanisms of action. Arh Hig Rada Toksikol. 2016;67:169-182.

    Not in our library10.1515/aiht-2016-67-2801 (opens in a new tab)

64. 64.

    Xia Z, Karpov P, Popowicz G, Tetko IV. Focused library generator: case of Mdmx inhibitors. J Comput Aided Mol Des. 2019.

    Not in our library10.1007/s10822-019-00242-8 (opens in a new tab)

65. 65.

    Huuskonen J. Estimation of aqueous solubility for a diverse set of organic compounds based on molecular topology. J Chem Inf Comput Sci. 2000;40:773-777.

    Not in our library10.1021/ci9901338 (opens in a new tab)

Cited by papers we hold 6

• A review of large language models and autonomous agents in chemistry · 2025
  No notice
• Deep learning in drug discovery: an integrative review and future challenges · 2022
  No notice
• Transformer-based artificial neural networks for the conversion between chemical notations · 2021
  Not checked
• Quantitative interpretation explains machine learning models for chemical reaction prediction and uncovers bias · 2021
  Not checked
• Multi-PLI: interpretable multi‐task deep learning model for unifying protein–ligand interaction datasets · 2021
  Not checked
• State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesis · 2020
  No notice

Questions about this paper

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.