Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback
Large language model (LLM) distillation aims to transfer the capabilities of a powerful teacher to a smaller student. Direct imitation, however, can also transfer the teacher's systematic bias and errors. This challenge is particularly pronounced under covariate shift, when the teacher's reliability on target questions is uncertain and target-domain reward feedback is unavailab
Reading in Plain English mode — toggle below to switch
Large language model (LLM) distillation aims to transfer the capabilities of a powerful teacher to a smaller student. Direct imitation, however, can also transfer the teacher's systematic bias and errors. This challenge is particularly pronounced under covariate shift, when the teacher's reliability on target questions is uncertain and target-domain reward feedback is unavailab
This paper addresses a critical challenge in LLM distillation: mitigating teacher bias without relying on target-domain reward feedback, which is often unavailable. For ML engineers, this means more robust and reliable methods for creating smaller, specialized models from larger ones, especially in domains with data scarcity or privacy concerns.