Self-Supervised Confidence Training for Efficient Reasoning
This paper introduces a novel self-supervised confidence training method to improve the efficiency of reasoning models. By teaching models to predict their own confidence in answers at intermediate steps, it significantly reduces the length of reasoning traces without explicitly optimizing for brevity, leading to faster inference.
Reading in Plain English mode — toggle below to switch
This paper introduces a novel self-supervised confidence training method to improve the efficiency of reasoning models. By teaching models to predict their own confidence in answers at intermediate steps, it significantly reduces the length of reasoning traces without explicitly optimizing for brevity, leading to faster inference.
Imagine a student solving a complex math problem. Instead of writing out every single step until the very end, a confident student might realize they've reached the correct answer earlier and stop. This paper teaches AI models a similar 'metacognitive' skill: to internally assess their confidence and implicitly stop when they're 'sure enough', without being told to stop.
How Self-Supervised Confidence Training Works
- 11. A reasoning model generates a solution path, and its confidence in the final answer is determined.
- 22. The model is then fine-tuned to predict this confidence at various intermediate points along its own reasoning trajectory.
- 33. This confidence prediction is used as a self-supervised training target; no explicit length or stopping objectives are used.
- 44. During inference, the model operates normally, but the internal confidence mechanism implicitly guides it to generate shorter, yet accurate, reasoning paths.
L = - Σ (yᵢ log(pᵢ) + (1-yᵢ) log(1-pᵢ))
This formula represents a common loss function used to train models to predict probabilities. Here, $y_i$ is the true confidence (e.g., 1 for correct, 0 for incorrect) and $p_i$ is the model's predicted confidence. The model learns to minimize this difference.
Consider a large language model solving a multi-step mathematical word problem. With confidence training, instead of generating a very verbose, step-by-step solution that might include redundant checks, the model learns to arrive at the correct answer more directly and stop, much like an expert human solver who quickly identifies the critical steps.
Teaching models to predict their own confidence can implicitly lead to more efficient reasoning, without direct optimization for length or explicit stopping mechanisms.