Collaborative Knowledge Distillation for Continual Learning in Recommender System

Abstract

Practical recommender systems (RS) necessitate both effectiveness and efficiency. Although knowledge distillation (KD) is a promising solution, its frequent execution in continually incoming non-stationary datasets is impractical due to high training costs. To solve this problem, we propose an integrated framework that combines KD and continual learning (CL) for real-world RS. This enables collaborative evolu- tion between teacher and student models in a dynamic environment with continuous training on ever-changing datasets. Furthermore, we introduce stability and plasticity student models to prevent catastrophic forgetting, focusing on historical and recent knowledge, respectively. Therefore, both student and teacher models can be trained with multi-faceted knowledge, enhancing their training over time. Our experiments demonstrate the superiority of the proposed framework, enhancing practicality in RS and bridging the industry-research gap in real-world RS applications.