Multi-layer multi-head self-attention mechanism is widely applied in modern neural language models. Attention redundancy has been observed among attention heads but has not been deeply studied in the literature. Using BERT-base model as an example, t
his paper provides a comprehensive study on attention redundancy which is helpful for model interpretation and model compression. We analyze the attention redundancy with Five-Ws and How. (What) We define and focus the study on redundancy matrices generated from pre-trained and fine-tuned BERT-base model for GLUE datasets. (How) We use both token-based and sentence-based distance functions to measure the redundancy. (Where) Clear and similar redundancy patterns (cluster structure) are observed among attention heads. (When) Redundancy patterns are similar in both pre-training and fine-tuning phases. (Who) We discover that redundancy patterns are task-agnostic. Similar redundancy patterns even exist for randomly generated token sequences. (Why'') We also evaluate influences of the pre-training dropout ratios on attention redundancy. Based on the phase-independent and task-agnostic attention redundancy patterns, we propose a simple zero-shot pruning method as a case study. Experiments on fine-tuning GLUE tasks verify its effectiveness. The comprehensive analyses on attention redundancy make model understanding and zero-shot model pruning promising.
A lot of research directed its concern to the reliability of Wireless Sensor Networks
(WSNs) used in various applications, especially in early detection of forest fires to ensure
the reliability of warning alarms sent by sensors and reduce the aver
age of false warnings.
In this research we have tried to evaluate the reliability of WSN used in early
detection of fires in Fir and cedar preserve, mainly. By designing hybrid WSN network,
similar to the terrains of the preserve and modeling it using program Opnet14.5. We have
studied several scenarios, to allow increasing malfunction of the network resulting from
fire break out and spreading: starting in allowance of 0% and comparing its results the
results of mathematical equations of reliability according to the same scenarios. In
addition, we have calculated the final availability through suggesting a mechanism to
improve WSN reliability using the redundancy, i.e add sensitive spare nodes, which
replace the damaged ones as the result of fire. The results have proved the remarkable
increasing of reliability. Also, it has been predicted of the reliability of the network
designed according to reliability of different values of the nodes used by using one of the
reliability devices "the Block Diagram".
