NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level aut
Reading in Plain English mode — toggle below to switch
We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level aut
This work introduces a novel approach to language modeling by incorporating Next Concept Prediction (NCP) alongside traditional next-token prediction, potentially leading to more robust and efficient models. The use of a product-quantized concept vocabulary for latent space construction offers new avenues for model compression and faster inference.