Speaker: Leonard Konle and Fotis Jannidis
Affiliation: Würzburg University, Germany
Title: Domain and Task Adaptive Pretraining for Language Models
Abstract: All current state-of-the-art systems in NLP utilize transformer based language models trained on massive amounts of text. This paper discusses strategies to adapt these models to historical domains and tasks, typical for research in the Computational Humanities. Using two task-specific corpora from the same domain (literary texts from the 19th Century) and Bert resp. Distilbert as baselines, we can confirm results from a recent study that continuing pretraining on the domain and the task data substantially improves task performance. Training a model from scratch using Electra is not competitive for our data sets.
