Skip to content
naulon
Get started

No retraction, correction or concern found when last checked on Oct 15, 2026.

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim and 4 more (opens in a new tab)

Bioinformatics · 2019 · CC BY

doi:10.1093/bioinformatics/btz682 (opens in a new tab)

Abstract

Motivation. Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora.

Results. We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts.

Availability and implementation. We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

Read at PubMed Central (opens in a new tab)

References 38

  1. 1.

    Alsentzer E. et al (2019 ) Publicly available clinical bert embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, MN, USA. pp. 72–78. Association for Computational Linguistics. https://www.aclweb.org/anthology/W19-1909.

    Not in our library

  2. 2.

    Bhasuran B., Natarajan J. (2018 ) Automatic extraction of gene-disease associations from literature using joint ensemble learning. PLoS One, 13, e0200699.

    Not in our library10.1371/journal.pone.0200699 (opens in a new tab)

  3. 3.
    Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research · 2015
    No notice
  4. 4.

    Devlin J. et al (2019 ) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA. pp. 4171–4186. Association for Computational Linguistics. https://www.aclweb.org/anthology/N19-1423.

    Not in our library

  5. 5.

    Doğan R.I. et al (2014 ) NCBI disease corpus: a resource for disease name recognition and concept normalization. J. Biomed. Inform., 47, 1–10.

    Not in our library10.1016/j.jbi.2013.12.006 (opens in a new tab)

  6. 6.
    LINNAEUS: A species name identification system for biomedical literature · 2010
    No notice
  7. 7.

    Giorgi J.M., Bader G.D. (2018 ) Transfer learning for biomedical named entity recognition with neural networks. Bioinformatics, 34, 4087.

    Not in our library10.1093/bioinformatics/bty449 (opens in a new tab)

  8. 8.

    Habibi M. et al (2017 ) Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics, 33, i37–i48.

    Not in our library10.1093/bioinformatics/btx228 (opens in a new tab)

  9. 9.

    Kim J.-D. et al (2004 ) Introduction to the bio-entity recognition task at JNLPBA In: Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), Geneva, Switzerland pp. 73–78. COLING. https://www.aclweb.org/anthology/W04-1213.

    Not in our library

  10. 10.
    The CHEMDNER corpus of chemicals and drugs and its annotation principles · 2015
    No notice
  11. 11.

    Krallinger M. et al (2017 ) Overview of the BioCreative VI chemical-protein interaction track. In: Proceedings of the BioCreative VI Workshop, Bethesda, MD, USA, pp. 141–146. https://academic.oup.com/database/article/doi/10.1093/database/bay073/5055578.

    Not in our library

  12. 12.

    Li J. et al (2016 ) Biocreative V CDR task corpus: a resource for chemical disease relation extraction. Database, 2016.

    Not in our library10.1093/database/baw068 (opens in a new tab)

  13. 13.

    Lim S., Kang J. (2018 ) Chemical–gene relation extraction using recursive neural network. Database, 2018.

    Not in our library10.1093/database/bay060 (opens in a new tab)

  14. 14.

    Lin C. et al (2019 ) A bert-based universal model for both within-and cross-sentence clinical temporal relation extraction. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, MN, USA. pp. 65–71. Association for Computational Linguistics. https://www.aclweb.org/anthology/W19-1908.

    Not in our library

  15. 15.

    Lou Y. et al (2017 ) A transition-based joint model for disease named entity recognition and normalization. Bioinformatics, 33, 2363–2371.

    Not in our library10.1093/bioinformatics/btx172 (opens in a new tab)

  16. 16.

    Luo L. et al (2018 ) An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition. Bioinformatics, 34, 1381–1388.

    Not in our library10.1093/bioinformatics/btx761 (opens in a new tab)

  17. 17.

    McCann B. et al (2017 ) Learned in translation: contextualized word vectors. In: Guyon,I. et al. (eds.), Advances in Neural Information Processing Systems 30, Curran Associates, Inc., pp. 6294–6305. http://papers.nips.cc/paper/7209-learned-in-translation-contextualized-word-vectors.pdf.

    Not in our library

  18. 18.

    Mikolov T. et al (2013 ) Distributed representations of words and phrases and their compositionality. In: Burges,C.J.C. (eds.), Advances in Neural Information Processing Systems 26, Curran Associates, Inc., pp. 3111–3119. http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf.

    Not in our library

  19. 19.

    Mohan S., Li D. (2019 ) Medmentions: a large biomedical corpus annotated with UMLS concepts. arXiv preprint arXiv: 1902.09476.

    Not in our library

  20. 20.
    The SPECIES and ORGANISMS Resources for Fast and Accurate Identification of Taxonomic Names in Text · 2013
    No notice
  21. 21.

    Pennington J. et al (2014 ) Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar. pp. 1532–1543. Association for Computational Linguistics. https://www.aclweb.org/anthology/D14-1162.

    Not in our library

  22. 22.

    Peters M.E. et al (2018 ) Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA pp. 2227–2237. Association for Computational Linguistics. https://www.aclweb.org/anthology/N18-1202.

    Not in our library

  23. 23.

    Pyysalo S. et al (2013 ) Distributional semantics resources for biomedical text processing. In: Proceedings of the 5th International Symposium on Languages in Biology and Medicine, Tokyo, Japan, pp. 39–43. https://academic.oup.com/bioinformatics/article/33/14/i37/3953940.

    Not in our library

  24. 24.

    Rajpurkar P. et al (2016 ) Squad: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX pp. 2383–2392. Association for Computational Linguistics. https://www.aclweb.org/anthology/D16-1264.

    Not in our library

  25. 25.

    Sachan D.S. et al (2018 ) Effective use of bidirectional language modeling for transfer learning in biomedical named entity recognition. In: Finale,D.-V. et al. (eds.), Proceedings of Machine Learning Research, Palo Alto, CA, Vol. 85, pp. 383–402. PMLR. http://proceedings.mlr.press/v85/sachan18a.html.

    Not in our library

  26. 26.
    Overview of BioCreative II gene mention recognition · 2008
    No notice
  27. 27.

    Sousa D. et al (2019 ) A silver standard corpus of human phenotype-gene relations. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN. pp. 1487–1492. Association for Computational Linguistics. https://www.aclweb.org/anthology/N19-1152.

    Not in our library

  28. 28.

    Sung N. et al (2017 ) NSML: A machine learning platform that enables you to focus on your models. arXiv preprint arXiv: 1712.05902.

    Not in our library

  29. 29.
    An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition · 2015
    No notice
  30. 30.

    Uzuner Ö. et al (2011 ) 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J. Am. Med. Inform. Assoc., 18, 552–556.

    Not in our library10.1136/amiajnl-2011-000203 (opens in a new tab)

  31. 31.

    Van Mulligen E.M. et al (2012 ) The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships. J. Biomed. Inform., 45, 879–884.

    Not in our library10.1016/j.jbi.2012.04.004 (opens in a new tab)

  32. 32.

    Vaswani A. et al (2017 ) Attention is all you need. In: Guyon,I. et al. (eds.), Advances in Neural Information Processing Systems, pp. 5998–6008. Curran Associates, Inc. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf.

    Not in our library

  33. 33.

    Wang X. et al (2018 ) Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, 35, 1745–1752.

    Not in our library10.1093/bioinformatics/bty869 (opens in a new tab)

  34. 34.

    Wiese G. et al (2017 ) Neural domain adaptation for biomedical question answering. In: Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada pp. 281–289. Association for Computational Linguistics. https://www.aclweb.org/anthology/K17-1029.

    Not in our library

  35. 35.

    Wu Y. et al (2016 ) Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.08144.

    Not in our library

  36. 36.

    Xu K. et al (2019 ) Document-level attention-based BiLSTM-CRF incorporating disease dictionary for disease named entity recognition. Comput. Biol. Med., 108, 122–132.

    Not in our library10.1016/j.compbiomed.2019.04.002 (opens in a new tab)

  37. 37.

    Yoon W. et al (2019 ) Collabonet: collaboration of deep neural networks for biomedical named entity recognition. BMC Bioinformatics, 20, 249.

    Not in our library10.1186/s12859-019-2813-6 (opens in a new tab)

  38. 38.

    Zhu H. et al (2018 ) Clinical concept extraction with contextual word embedding. NIPS Machine Learning for Health Workshop http://par.nsf.gov/biblio/10098080.

    Not in our library

Cited by papers we hold 33

and 13 more

Questions about this paper

We email you if it is retracted, withdrawn, corrected or flagged.

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.