Leveraging Relational Knowledge with Pretrained Language Models

Abstract

Relational knowledge encoded in texts is useful in natural language processing tasks. It provides an informative training signal that could not be generated by an instance alone, generates a relational cue for explaining the relational information encoded in the models, and reveals the model’s hidden ability to utilize relational knowledge in a variety of ways. Along with the recent advance of pretrained language models, incorporating relational knowledge into the language models has garnered attention in the research community. In this dissertation, we propose three different approaches to integrate rela- tional knowledge with the pretrained language models. There exist several chal- lenges when we train or evaluate the pretrained language models with relational knowledge. (1) First, considering the contextualized embedding space formed by the pretrained language models becomes a challenge to effectively train them with relational knowledge. To this end, we design a novel mixup training methodology to tightly regularize inside and outside the manifold formed by the language models with relational knowledge. (2) Second, we address the fundamental challenge of pretrained language models, i.e., the opaqueness of their decision process, with relational knowledge. Our approach adopts the concept of optimal transport to discover implicit word-level relational knowledge between two sentences, which provides model developers useful evidence about which relationship between the words affects the final semantic relevance between the sentences. (3) Lastly, discovering the hidden ability to utilize relational knowledge in generative pretrained language models is important to elicit their potential. To effectively reveal their ability, we curate an evaluation set for measuring the generative language models’ ability to utilize other functions in code. We construct several code snippets that contain two related functions that one helps the other. With the comprehensive analysis, we show the language models’ ability to utilize auxiliary functions in multifaceted ways. In this work, we demonstrate that these approaches successfully leverage relational knowledge inside the texts with the pretrained language models through structured experiments. We believe that relational knowledge acts as a valuable source for enhancing the language model and we will continue to develop and refine them in our future research.