/-
Copyright 2026 The Formal Conjectures Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-/
module
public import Mathlib.Analysis.SpecialFunctions.BinaryEntropy
public import Mathlib.MeasureTheory.Measure.Prod
public import Mathlib.MeasureTheory.Measure.Real
public import Mathlib.Tactic.FunPropShannon entropy of finite-valued random variables
This file defines Shannon entropy and mutual information, in bits, for finite-valued random variables.
For a probability measure μ and a function U with measurable fibers, write
q_U(a) = μ.real (U ⁻¹' {a}) for the probability that U = a. For a probability q,
Real.negMulLog q is -q ln q when q > 0 and 0 when q = 0, so
H(U) = (∑ a, Real.negMulLog (q_U a)) / Real.log 2 = -∑ a, q_U(a) log₂ q_U(a),
with the convention 0 log₂ 0 = 0; dividing by Real.log 2 converts nats to bits. Mutual
information is I(U; V) = H(U) + H(V) - H(U, V).
Measurable fibers are automatic when U is measurable and its finite target has measurable
singletons, in particular for finite discrete targets. The definitions are total expressions for
arbitrary measures and functions; without normalization and measurable fibers they need not have
their usual probabilistic interpretation.
The main results are invariance under measurable pushforward and the formula for the mutual information of the coordinate projections of a finite joint law.
@[expose] public sectionopen MeasureTheoryopen scoped BigOperatorsnamespace InformationTheoryvariable {Ω : Type*} [MeasurableSpace Ω]Shannon entropy, in bits, of a finite-valued random variable.
For a probability measure μ, this is the usual Shannon entropy when every fiber
U ⁻¹' {a} is measurable. In particular, this holds when U is measurable and its target
has measurable singletons.
Zero-probability values contribute zero. The expression is also defined without these hypotheses, but then need not have a probabilistic interpretation.
noncomputable def shannonEntropyBits {α : Type*} [Fintype α]
(μ : Measure Ω) (U : Ω → α) : ℝ :=
(∑ a, Real.negMulLog (μ.real (U ⁻¹' {a}))) / Real.log 2
Mutual information, in bits, of two finite-valued random variables, defined by
I(U; V) = H(U) + H(V) - H(U, V).
For a probability measure μ, this is the usual mutual information when the fibers of
both U and V are measurable. In particular, this holds for measurable functions into
finite targets with measurable singletons.
The expression is also defined without these hypotheses, but then need not have a probabilistic interpretation.
noncomputable def mutualInformationBits {α : Type*} {β : Type*}
[Fintype α] [Fintype β] (μ : Measure Ω) (U : Ω → α) (V : Ω → β) : ℝ :=
shannonEntropyBits μ U + shannonEntropyBits μ V -
shannonEntropyBits μ (fun ω => (U ω, V ω))section Mapvariable {Ω' : Type*} {α : Type*} {β : Type*}
[MeasurableSpace Ω']
[Fintype α] [MeasurableSpace α] [MeasurableSingletonClass α]
[Fintype β] [MeasurableSpace β] [MeasurableSingletonClass β]Entropy commutes with measurable pushforward of the sample-space measure. No normalization assumption on the measure is required.
e_a Ω:Type u_1inst✝⁴:MeasurableSpace ΩΩ':Type u_2α:Type u_3inst✝³:MeasurableSpace Ω'inst✝²:Fintype αinst✝¹:MeasurableSpace αinst✝:MeasurableSingletonClass αμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αhU:Measurable Ua:αa✝:a ∈ Finset.univ⊢ (μ (T ⁻¹' U ⁻¹' {a})).toReal = (μ (U ∘ T ⁻¹' {a})).toReal
rfl All goals completed! 🐙Mutual information commutes with measurable pushforward of the sample-space measure. No normalization assumption on the measure is required.
theorem mutualInformationBits_map (μ : Measure Ω) {T : Ω → Ω'}
(hT : Measurable T) {U : Ω' → α} {V : Ω' → β}
(hU : Measurable U) (hV : Measurable V) :
mutualInformationBits (μ.map T) U V =
mutualInformationBits μ (U ∘ T) (V ∘ T) := by Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable V⊢ mutualInformationBits (Measure.map T μ) U V = mutualInformationBits μ (U ∘ T) (V ∘ T)
have hUV : Measurable (fun ω => (U ω, V ω)) := by fun_prop Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ mutualInformationBits (Measure.map T μ) U V = mutualInformationBits μ (U ∘ T) (V ∘ T) Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ mutualInformationBits (Measure.map T μ) U V = mutualInformationBits μ (U ∘ T) (V ∘ T)
unfold mutualInformationBits Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ (shannonEntropyBits (Measure.map T μ) U + shannonEntropyBits (Measure.map T μ) V -
shannonEntropyBits (Measure.map T μ) fun ω ↦ (U ω, V ω)) =
shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ fun ω ↦ ((U ∘ T) ω, (V ∘ T) ω)
rw [shannonEntropyBits_map μ hT hU, Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ (shannonEntropyBits μ (U ∘ T) + shannonEntropyBits (Measure.map T μ) V -
shannonEntropyBits (Measure.map T μ) fun ω ↦ (U ω, V ω)) =
shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ fun ω ↦ ((U ∘ T) ω, (V ∘ T) ω) Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ ((fun ω ↦ (U ω, V ω)) ∘ T) =
shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ fun ω ↦ ((U ∘ T) ω, (V ∘ T) ω) shannonEntropyBits_map μ hT hV, Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ (shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) -
shannonEntropyBits (Measure.map T μ) fun ω ↦ (U ω, V ω)) =
shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ fun ω ↦ ((U ∘ T) ω, (V ∘ T) ω) Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ ((fun ω ↦ (U ω, V ω)) ∘ T) =
shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ fun ω ↦ ((U ∘ T) ω, (V ∘ T) ω)
shannonEntropyBits_map μ hT hUV Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ ((fun ω ↦ (U ω, V ω)) ∘ T) =
shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ fun ω ↦ ((U ∘ T) ω, (V ∘ T) ω) Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ ((fun ω ↦ (U ω, V ω)) ∘ T) =
shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ fun ω ↦ ((U ∘ T) ω, (V ∘ T) ω)] Ω:Type u_1inst✝⁷:MeasurableSpace ΩΩ':Type u_2α:Type u_3β:Type u_4inst✝⁶:MeasurableSpace Ω'inst✝⁵:Fintype αinst✝⁴:MeasurableSpace αinst✝³:MeasurableSingletonClass αinst✝²:Fintype βinst✝¹:MeasurableSpace βinst✝:MeasurableSingletonClass βμ:Measure ΩT:Ω → Ω'hT:Measurable TU:Ω' → αV:Ω' → βhU:Measurable UhV:Measurable VhUV:Measurable fun ω ↦ (U ω, V ω)⊢ shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ ((fun ω ↦ (U ω, V ω)) ∘ T) =
shannonEntropyBits μ (U ∘ T) + shannonEntropyBits μ (V ∘ T) - shannonEntropyBits μ fun ω ↦ ((U ∘ T) ω, (V ∘ T) ω)
rfl All goals completed! 🐙end MapThe mutual information of the coordinate projections, written using the singleton masses of a finite measure on a finite product.
theorem mutualInformationBits_fst_snd {α : Type*} {β : Type*}
[Fintype α] [MeasurableSpace α] [MeasurableSingletonClass α]
[Fintype β] [MeasurableSpace β] [MeasurableSingletonClass β]
(ν : Measure (α × β)) [IsFiniteMeasure ν] :
mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, Real.negMulLog (∑ b, ν.real {(a, b)})) / Real.log 2 +
(∑ b, Real.negMulLog (∑ a, ν.real {(a, b)})) / Real.log 2 -
(∑ ab : α × β, Real.negMulLog (ν.real {ab})) / Real.log 2 := by α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure ν⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
classical
have hsum (s : Set (α × β)) :
ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0 := by α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure ν⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
simpa [Finset.sum_filter] using
(sum_measureReal_singleton (μ := ν) (Finset.univ.filter (fun w => w ∈ s))).symm α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
have hfst (a : α) :
ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)} := by α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure ν⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
rw [hsum α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ w, if w ∈ Prod.fst ⁻¹' {a} then ν.real {w} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ w, if w ∈ Prod.fst ⁻¹' {a} then ν.real {w} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2] α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ w, if w ∈ Prod.fst ⁻¹' {a} then ν.real {w} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
simp only [Fintype.sum_prod_type, Set.mem_preimage, Set.mem_singleton_iff] α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ x, ∑ x_1, if x = a then ν.real {(x, x_1)} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
rw [Finset.sum_eq_single a (fun x _ hx => by α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:αx:αx✝:x ∈ Finset.univhx:x ≠ a⊢ (∑ x_1, if x = a then ν.real {(x, x_1)} else 0) = 0 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ x, if a = a then ν.real {(a, x)} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2 simp [hx] All goals completed! 🐙 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ x, if a = a then ν.real {(a, x)} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2) (by α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ a ∉ Finset.univ → (∑ x, if a = a then ν.real {(a, x)} else 0) = 0 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ x, if a = a then ν.real {(a, x)} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2 simp All goals completed! 🐙 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ x, if a = a then ν.real {(a, x)} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2)] α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0a:α⊢ (∑ x, if a = a then ν.real {(a, x)} else 0) = ∑ b, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
simp α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
have hsnd (b : β) :
ν.real (Prod.snd ⁻¹' {b}) = ∑ a, ν.real {(a, b)} := by α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure ν⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}hsnd:∀ (b : β), ν.real (Prod.snd ⁻¹' {b}) = ∑ a, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
rw [hsum α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}b:β⊢ (∑ w, if w ∈ Prod.snd ⁻¹' {b} then ν.real {w} else 0) = ∑ a, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}b:β⊢ (∑ w, if w ∈ Prod.snd ⁻¹' {b} then ν.real {w} else 0) = ∑ a, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}hsnd:∀ (b : β), ν.real (Prod.snd ⁻¹' {b}) = ∑ a, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2] α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}b:β⊢ (∑ w, if w ∈ Prod.snd ⁻¹' {b} then ν.real {w} else 0) = ∑ a, ν.real {(a, b)} α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}hsnd:∀ (b : β), ν.real (Prod.snd ⁻¹' {b}) = ∑ a, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
simp [Fintype.sum_prod_type] α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}hsnd:∀ (b : β), ν.real (Prod.snd ⁻¹' {b}) = ∑ a, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2 α:Type u_2β:Type u_3inst✝⁶:Fintype αinst✝⁵:MeasurableSpace αinst✝⁴:MeasurableSingletonClass αinst✝³:Fintype βinst✝²:MeasurableSpace βinst✝¹:MeasurableSingletonClass βν:Measure (α × β)inst✝:IsFiniteMeasure νhsum:∀ (s : Set (α × β)), ν.real s = ∑ w, if w ∈ s then ν.real {w} else 0hfst:∀ (a : α), ν.real (Prod.fst ⁻¹' {a}) = ∑ b, ν.real {(a, b)}hsnd:∀ (b : β), ν.real (Prod.snd ⁻¹' {b}) = ∑ a, ν.real {(a, b)}⊢ mutualInformationBits ν Prod.fst Prod.snd =
(∑ a, (∑ b, ν.real {(a, b)}).negMulLog) / Real.log 2 + (∑ b, (∑ a, ν.real {(a, b)}).negMulLog) / Real.log 2 -
(∑ ab, (ν.real {ab}).negMulLog) / Real.log 2
simp [mutualInformationBits, shannonEntropyBits, hfst, hsnd] All goals completed! 🐙end InformationTheory