Entropy evolution in multilingual transformer models : An information-theoretic perspective
*Arti BansodeCorresponding authorartibansodephd@gmail.comDepartment of Computer ScienceSNDT Women’s UniversityJuhu Campus, Mumbai, Maharashtra, 400049, IndiaView full profile → , Anita Chawareanita.chaware@computersc.sndt.ac.inDepartment of Computer ScienceSNDT Women’s UniversityJuhu Campus, Mumbai, Maharashtra, 400049, IndiaView full profile →
* Corresponding author · click or hover a name for details
- Received:
- 01 Dec 2025
- Published Online:
- 30 Sep 2026
- Article type:
- Research Article
- Language:
- EN
- Article no.:
- JIOS-2353
- Pages:
- 3143–3151
Abstract
Entropy is used to quantify uncertainty in probabilistic representations. Hence, studying how uncertainty evolves across layers in LLMs can help us understand how these models process different languages. This paper presents a layer-wise entropy analysis of how information is processed across layers of four different Transformer models using parallel data in English, Hindi, Marathi, and Gujarati. The Shannon entropy is computed for each layer and several information-theoretic metrics are derived, including entropy change, CR, JSD and LDI. In addition, four scalar measures like NLC, ACD, ES, and CLAS are used to summarise overall compression behaviour. The results show that autoregressive models exhibit largely monotonic entropy compression, whereas the multilingual model shows oscillatory compression and expansion patterns. English undergoes stronger and more efficient entropy reduction than Indic Languages, while Hindi and Marathi show closely aligned entropy trajectories. These findings support the use of entropy-based analysis as a diagnostic tool for analysing architectural behaviour and cross-lingual representational disparities in transformer models.
Keywords
Subject Classifications
References
[1] C. E. Shannon, “A mathematical theory of communication,” Bell System Technical Journal, vol. 27, no. 3, pp. 379–423 (Jul. 1948), doi: 10.1002/j.1538-7305.1948.tb01338.x.
[2] N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in Proc. 2015 IEEE Information Theory Workshop (ITW), pp. 1–5 (2015), doi: 10.1109/ITW.2015.7133169.
[3] M. B. Thorat, V. P. Wawage, D. P. Yadav, P. Pushpalatha, and Y. Gandhi, “Mathematical foundations of neural network weight optimization,” Journal of Information and Optimization Sciences, vol. 47, no. 5-B, pp. 2065–2073 (2026), doi: 10.47974/JIOS-2297.
[4] R. Maheshwari and V. Kapoor, “Evaluating AI convergence and performance analysis for cross transfer entropy-based learning among time-lagged data employing cognitive computing framework,” Journal of Information and Optimization Sciences, vol. 47, no. 5-B, pp. 1931–1943 (2026), doi: 10.47974/JIOS-2285.
[5] R. Shwartz-Ziv and N. Tishby, “Opening the black box of deep neural networks via information,” arXiv preprint arXiv:1703.00810 (2017).
[6] Z. Nazari and B. Mosapour, “The entropy of hyperfuzzy sets,” Journal of Dynamical Systems and Geometric Theories, vol. 16, no. 2, pp. 173–185 (2018), doi: 10.1080/1726037X.2018.1436270.
[7] R. Ali, F. Caso, C. Irwin, and P. Liò, “Entropy-Lens: The information signature of transformer computations,” arXiv preprint arXiv:2502.16570 (2025).
[8] M. Danelli, Pruning a Transformer Using Layer-wise Batch Entropy. Innsbruck, Austria: Master’s thesis (Aug. 2024).
[9] E. Queipo-de-Llano, A. Arroyo, F. Barbero, X. Dong, M. Bronstein, Y. LeCun, and R. Shwartz-Ziv, “Attention sinks and compression valleys in LLMs are two sides of the same coin,” arXiv preprint arXiv:2510.06477 (2025).
[10] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov, “Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned,” in Proc. 57th Annu. Meeting Assoc. Comput. Linguistics (ACL), pp. 5797–5808 (2019), doi: 10.18653/v1/P19-1580.
[11] K. Ethayarajh, “How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings,” in Proc. 2019 Conf. Empirical Methods in Natural Language Processing and 9th Int. Joint Conf. Natural Language Processing (EMNLP-IJCNLP), pp. 55–65 (2019), doi: 10.18653/v1/D19-1006.
[12] V. Quétu, Z. Liao, and E. Tartaglione, “The simpler the better: An entropy-based importance metric to reduce neural networks’ depth,” in Joint European Conf. Machine Learning and Knowledge Discovery in Databases (ECML PKDD), pp. 92–108 (2024), doi: 10.1007/978-3-031-70365-2_6.
[13] J. Lin, “Divergence measures based on the Shannon entropy,” IEEE Transactions on Information Theory, vol. 37, no. 1, pp. 145–151 (Jan. 1991), doi: 10.1109/18.61115.
[14] S. Kullback and R. A. Leibler, “On information and sufficiency,” The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79–86 (Mar. 1951), doi: 10.1214/aoms/1177729694.
[15] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics (ACL), pp. 8440–8451 (2020), doi: 10.18653/v1/2020.acl-main.747.
[16] M. Gabrié, A. Manoel, C. Luneau, J. Barbier, N. Macris, F. Krzakala, and L. Zdeborová, “Entropy and mutual information in models of deep neural networks,” in Advances in Neural Information Processing Systems, vol. 31 (2018).
[17] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI, Tech. Rep., (2019).
[18] Hugging Face, “DistilGPT-2: A distilled version of OpenAI GPT-2,” (2025). [Online]. Available: Hugging Face model page [Accessed: Jan. 2025].
[19] S. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman, “GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow,” Zenodo, (2021), doi: 10.5281/zenodo.5297715.
