Advancing bert pruning techniques: a conceptual framework and an ensemble method of neural subnetworks

Abstract

Large Language Models (LLMs), such as BERT (Bidirectional Encoder Representations from Transformers), have experienced rapid growth and increasing complexity, which has introduced significant challenges related to computational costs, resource consumption, and environmental impact. Although existing pruning techniques offer certain advantages, they frequently entail trade-offs, such as reduced model accuracy, increased computational overhead, and inadequate consideration of sustainability. To address these challenges, in this study we propose a conceptual framework aimed at improving the efficiency and sustainability of BERT models through innovative pruning techniques. This framework incorporates a novel multi-objective optimization algorithm that clarifies the trade-offs among model accuracy, computational expenses, inference speed, and energy usage. Furthermore, the study proposes an Ensemble of Winning Tickets, grounded on the lottery ticket hypothesis, for the pruning of BERT attention heads. Pruning techniques reduce model size while preserving performance, enabling the deployment of highly efficient and environmentally sustainable models. The experimental evaluations show how these techniques are effective to real world tasks including linguistic acceptability and sentiment analysis using the Stanford Sentiment Treebank (SST-2) and Corpus of Linguistic Acceptability (CoLA) datasets. The results show that the proposed multi objective method achieves significant model compression of BERT Base model being pruned to 70.4 million parameters gaining an accuracy of 96.3% on SST-2 dataset and F1-score of 95.2%. Furthermore, the pruned model demonstrated F1 score of 82.3% and 56% of Matthews Correlation Coefficient in CoLA dataset respectively. The findings of this study also include model size that has been reduced by 36%, with our ensemble model reaching greater performance as compared to the baseline BERT model on both SST-2 and CoLA datasets. The results show F1-score of 94% and 96%, respectively, and accuracy scores of 95% and 96% on the two datasets. The findings of this study demonstrate that advanced pruning techniques enhance model performance, computational efficiency, and environmental sustainability. This study present practical framework to researchers and practitioners to move from large, resource-heavy models to sparse, efficient, and scalable

options, thereby aiding the broader initiative to implement deep learning models in an environmentally conscious way.