Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Optimizer state is the largest single line item in the memory budget of mixture-of-experts (Mixture-of-Experts AI) training: on a 6.78B-parameter Mixture-of-Experts AI language model, AdamW keeps 50.6 GB of first and second moments to update 12.6 GB of bfloat16 weights. We study SkewAdam, an optimizer built on the observation that the three parameter populations of an Mixture-o
Reading in Plain English mode — toggle below to switch
Optimizer state is the largest single line item in the memory budget of mixture-of-experts (Mixture-of-Experts AI) training: on a 6.78B-parameter Mixture-of-Experts AI language model, AdamW keeps 50.6 GB of first and second moments to update 12.6 GB of bfloat16 weights. We study SkewAdam, an optimizer built on the observation that the three parameter populations of an Mixture-o
This research paper advances MoE and production ML, directly affecting how engineers build, optimize, and ship AI systems. Teams running production LLM workloads need to understand these techniques to stay competitive in hiring and architecture decisions.