UltraBERT: Beyond the Limit of BERT Pre-Training
Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encoders underoptimized, with no single model performing competitively across both classification and retrieval tasks. In this work, we investigate how far encoder performance
Reading in Plain English mode — toggle below to switch
Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encoders underoptimized, with no single model performing competitively across both classification and retrieval tasks. In this work, we investigate how far encoder performance
This research challenges conventional wisdom in BERT pre-training, suggesting that current encoder pre-training might be under-optimized. It provides insights into compute saturation points and task-specific masking ratios, which can lead to more efficient and performant encoder models for various downstream applications.