Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone
Modern AI architecture design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose
Reading in Plain English mode — toggle below to switch
Modern AI architecture design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose
This research introduces a novel metric, Neural Spectral Capacity (NSC), for evaluating and designing Transformer architectures without needing to train or even instantiate the model. For ML engineers, this means a potential paradigm shift in early-stage model design, allowing for more efficient exploration of architectural choices and better resource allocation.