Entropy, cross-entropy, relative entropy: Deformation theory^(a)

¹ University of Michigan - Ann Arbor, USA
² Nagoya Institute of Technology - Nagoya, Japan

^(b) junz@umich.edu (corresponding author)
^(c) matsuzoe@nitech.ac.jp

Received: 13 November 2020
Accepted: 22 March 2021

Abstract

Attempts at generalizing Shannon entropy and Kullback-Leibler divergence (relative entropy) led to a plenthora of deformation models in theoretical physics, including q-model, κ-model, etc. Naudts and Zhang (Inf. Geom., 1 (2018) 79) established that these models can be unified under two notions: deformed ϕ-exponential family (Naudts, J., J. Inequal. Pure Appl. Math., 5 (2004) 102) and conjugate -embedding (Zhang J., Neural Comput., 16 (2004) 159) of probability functions. Conjugate -embedding has a gauge freedom which, upon its fixing, subsumes the U-model of Eguchi (Sugaku Expositions, 19 (2006) 197) proposed in a statistical machine learning context. The generalization by -entropy, -cross-entropy, -divergence, when applied to the ϕ-exponential family, yields either a Hessian structure or a conformal Hessian structure under different gauge selections —this “splitting” is the hallmark when deforming the exponential family with its dually flat (Hessian) geometry. This letter provides a unified information geometric perspective of deformation of the exponential model, with calculations for Tsallis q-model.