Minimally Invasive Steering of Language Models
Pre-logit steering adapts a frozen language model to a test-time reward by adding vectors to its final hidden states. Unregularized reward optimization can substantially alter the output distribution and degrade generation quality. We propose Minimally Invasive Steering Vector Optimization (MISVO), which penalizes interventions using the local KL geometry of the induced token d
Reading in Plain English mode — toggle below to switch
Pre-logit steering adapts a frozen language model to a test-time reward by adding vectors to its final hidden states. Unregularized reward optimization can substantially alter the output distribution and degrade generation quality. We propose Minimally Invasive Steering Vector Optimization (MISVO), which penalizes interventions using the local KL geometry of the induced token d
This paper introduces a novel method for steering language models at test-time without significantly degrading generation quality, which is crucial for safety and alignment. ML engineers can leverage this technique to dynamically adapt model behavior to specific reward functions or user preferences during inference, enhancing control and reducing the need for costly fine-tuning.