LLM Pruning Degradation in Smart Homes: Architectures & Task Complexity
This research investigates how pruning, a technique to reduce the size of large language models (LLMs), affects their performance in smart home environments, especially for tasks involving tool use. It explores different LLM architectures and pruning methods to understand when and how performance degrades, moving beyond simple accuracy metrics.
Reading in Plain English mode — toggle below to switch
This research investigates how pruning, a technique to reduce the size of large language models (LLMs), affects their performance in smart home environments, especially for tasks involving tool use. It explores different LLM architectures and pruning methods to understand when and how performance degrades, moving beyond simple accuracy metrics.
Imagine trying to make a human brain more efficient by removing some connections (neurons or synapses). If you remove too many, or the wrong ones, the brain might still function for simple tasks but struggle with complex reasoning or specific actions. This paper studies how LLMs 'brains' degrade when pruned, particularly for precise 'tool-calling' actions.
How LLM Pruning Degradation is Studied
- 11. Select diverse LLMs: Researchers chose four LLMs, including dense Transformers, dense hybrids, and Mixture-of-Experts (MoE) models, representing different architectural approaches.
- 22. Apply various pruning methods: Different techniques like depth, width, hybrid, and expert pruning were used to reduce the size of these models.
- 33. Fine-tune pruned models: After pruning, the models underwent supervised fine-tuning (SFT) to adapt them to smart home tool-calling tasks.
- 44. Evaluate performance: Over 19,500 instances from three smart home datasets were used to assess degradation, looking beyond overall accuracy at specific action components (operation, device, argument, value) and task complexity.
- 55. Analyze degradation patterns: The study observed that dense models degrade sharply after a narrow 'safe' pruning region, while MoE models are more tolerant. Degradation first impacts specific grounded actions before general intent, and aggressive pruning can lead to models refusing to act.
D = f(P, A, T)
D represents the observed degradation, P is the pruning intensity and method, A is the LLM architecture, and T is the task complexity. This formula conceptually illustrates that degradation is a function of these interacting factors.
In a smart home scenario, an LLM might be asked to 'turn on the living room lights'. If aggressively pruned, a dense model might fail to identify 'living room' as the specific device argument, or even refuse the command entirely, whereas a more robust MoE model might still correctly execute the command even with significant pruning.
Pruning impacts different LLM architectures differently, with MoE models showing higher tolerance. Degradation is not uniform; specific grounded actions are affected before general intent, and aggressive pruning can cause models to 'over-refuse' tasks.