Leon Lang ; Pierre Baudot ; Rick Quax ; Patrick Forré - Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's Theorem

compositionality:14181 - Compositionality, January 30, 2025, Volume 7 (2025) - https://doi.org/10.46298/compositionality-7-1
Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's TheoremArticle

Authors: Leon Lang ; Pierre Baudot ; Rick Quax ; Patrick Forré

    In information theory, one major goal is to find useful functions that summarize the amount of information contained in the interaction of several random variables. Specifically, one can ask how the classical Shannon entropy, mutual information, and higher interaction information relate to each other. This is answered by Hu's theorem, which is widely known in the form of information diagrams: it relates shapes in a Venn diagram to information functions, thus establishing a bridge from set theory to information theory. In this work, we view random variables together with the joint operation as a monoid that acts by conditioning on information functions, and entropy as a function satisfying the chain rule of information. This abstract viewpoint allows to prove a generalization of Hu's theorem. It applies to Shannon and Tsallis entropy, (Tsallis) Kullback-Leibler Divergence, cross-entropy, Kolmogorov complexity, submodular information functions, and the generalization error in machine learning. Our result implies for Chaitin's Kolmogorov complexity that the interaction complexities of all degrees are in expectation close to Shannon interaction information. For well-behaved probability distributions on increasing sequence lengths, this shows that the per-bit expected interaction complexity and information asymptotically coincide, thus showing a strong bridge between algorithmic and classical information theory.


    Volume: Volume 7 (2025)
    Published on: January 30, 2025
    Imported on: March 1, 2024
    Keywords: Computer Science - Information Theory

    Consultation statistics

    This page has been seen 221 times.
    This article's PDF has been downloaded 63 times.