Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs
Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We co
Reading in Plain English mode — toggle below to switch
Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We co
This paper highlights how vision encoders and VLMs implicitly learn conceptual information, like canonical color, even when visual cues are removed. Understanding this 'conceptual decodability' is crucial for developing more robust and interpretable multimodal models, especially for tasks requiring abstract reasoning beyond surface-level visual features.