Common-Mode Collapse and Recovery in Direct Feedback Alignment
Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean-covariance decompos
Reading in Plain English mode — toggle below to switch
Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean-covariance decompos
Understanding the failure modes of training algorithms like Direct Feedback Alignment (DFA) is crucial for developing more robust and efficient learning systems. This paper identifies a specific collapse mechanism, offering insights into how to prevent training stalls and improve convergence, which directly impacts the reliability and performance of future AI models.