Abstract: Transfer learning has fundamentally changed the landscape of natural language processing (NLP). Many state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. When we only have limited supervision for the downstream tasks, however, due to the extremely high complexity of pre-trained models, aggressive fine-tuning often causes the fine-tuned model to overfit the training data of downstream tasks and fail to generalize to unseen data.
To address such a concern, we propose a new approach for fine-tuning of pretrained models to attain better generalization performance. Our proposed approach adopts three important ingredients: (1) Smoothness-inducing adversarial regularization, which effectively controls the complexity of the massive model; (2) Bregman proximal point optimization, which is an instance of trust-region algorithms and can prevent aggressive updating; (3) Differentiable programming, which can mitigate the undesired bias induced by conventional adversarial training algorithms. Our experiments show that the proposed approach significantly outperforms existing methods in multiple NLP tasks. In addition, our theoretical analysis provides some new insights of adversarial training for improving generalization.