Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal
Reading in Plain English mode — toggle below to switch
Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal
This paper offers critical insights into how image tokenizers, fundamental components of unified multimodal models, impact overall model performance and learnability. Understanding these dynamics is crucial for ML engineers designing and optimizing multimodal systems, as it directly influences model architecture choices and training strategies.