StaResGRU-CNN with CMedLMs: A stacked residual GRU-CNN with pre-trained biomedical language models for predictive intelligence

Pin Ni, Gangmin Li, Patrick C.K. Hung, Victor Chang

Research output: Contribution to journalArticlepeer-review

Abstract

As a task requiring strong professional experience as supports, predictive biomedical intelligence cannot be separated from the support of a large amount of external domain knowledge. By using transfer learning to obtain sufficient prior experience from massive biomedical text data, it is essential to promote the performance of specific downstream predictive and decision-making task models. This is an efficient and convenient method, but it has not been fully developed for Chinese Natural Language Processing (NLP) in the biomedical field. This study proposes a Stacked Residual Gated Recurrent Unit-Convolutional Neural Networks (StaResGRU-CNN) combined with the pre-trained language models (PLMs) for biomedical text-based predictive tasks. Exploring related paradigms in biomedical NLP based on transfer learning of external expert knowledge and comparing some Chinese and English language models. We have identified some key issues that have not been developed or those present difficulties of application in the field of Chinese biomedicine. Therefore, we also propose a series of Chinese bioMedical Language Models (CMedLMs) with detailed evaluations of downstream tasks. By using transfer learning, language models are introduced with prior knowledge to improve the performance of downstream tasks and solve specific predictive NLP tasks related to the Chinese biomedical field to serve the predictive medical system better. Additionally, a free-form text Electronic Medical Record (EMR)-based Disease Diagnosis Prediction task is proposed, which is used in the evaluation of the analyzed language models together with Clinical Named Entity Recognition, Biomedical Text Classification tasks. Our experiments prove that the introduction of biomedical knowledge in the analyzed models significantly improves their performance in the predictive biomedical NLP tasks with different granularity. And our proposed model also achieved competitive performance in these predictive intelligence tasks.

Original languageEnglish
Article number107975
JournalApplied Soft Computing
Volume113
DOIs
Publication statusPublished - 22 Oct 2021

Bibliographical note

Funding Information:
This research is partly supported by VC Research (VCR 0000130) for Prof. Chang. At the same time, this study is also partially supported by the AI University Research Center (AI-URC) through the XJTLU Key Program Special Fund, China (KSF-P-02, KSF-A-17). And this work has received support from the Suzhou Bureau of Science and Technology through the Key Industrial Technology Innovation Program, China (No. SYG201840). We also appreciate Google TensorFlow Research Cloud (TFRC) for providing support in computing resources. In addition, we would like to thank all colleagues who participated in this research project, especially Ms. Yuming Li and Mr. Zhenjin Dai. We would also like to express our sincere thanks to Mr. Thomas Cilloni for providing English language support to the manuscript.

Funding Information:
This research is partly supported by VC Research ( VCR 0000130 ) for Prof. Chang. At the same time, this study is also partially supported by the AI University Research Center (AI-URC) through the XJTLU Key Program Special Fund, China ( KSF-P-02 , KSF-A-17 ). And this work has received support from the Suzhou Bureau of Science and Technology through the Key Industrial Technology Innovation Program, China (No. SYG201840 ). We also appreciate Google TensorFlow Research Cloud (TFRC) for providing support in computing resources.

Publisher Copyright:
© 2021 Elsevier B.V.

Fingerprint

Dive into the research topics of 'StaResGRU-CNN with CMedLMs: A stacked residual GRU-CNN with pre-trained biomedical language models for predictive intelligence'. Together they form a unique fingerprint.

Cite this