Optimal Coding based on a Mutual Information Criterion

Bernard Colin, François Dubeau, Hussein Khreibani Département de Mathématiques Université de Sherbrooke Sherbrooke J1K-2R1

Jules de Tibeiro Secteur des Sciences Université de Moncton à Shippagan Shippagan E8S-1P6

Research Report No 82

Abstract

In this paper, we use a criterion based on the notion of mutual information between random variables, in order to construct an optimal discretization of a set of continuous random variables. More precisely, we propose a simultaneous coding of the whole set of variables, which takes into account the stochastic dependence between them.

Key words: Divergence, mutual information, estimation, copula, optimal coding.

1 Introduction

In data anlysis, we frequently encounter in practice some situations where we have to deal with a great number of variables which are continuous for some of them while the others are categorical. In this case, according to the choosen data analysis model, we have to categorize these continuous variables. For instance let us suppose that we have a sample of individuals on which we observe the following variables:

X = age, Y = salary, Z = category of employment

If we want to take into account the variables simultaneously as, for example, in Multiple Correspondance Analysis (MCA), we have to put them on the same form by means of a discretization of the first two ones. Instead of the usual independent categorization of the variables X and Y, we propose using the stochastic dependence between them, in order to categorize simultaneously X and Y.

More generally let X_{1}, X_{2}, ..., X_{n} be n random variables defined on domains 𝔻_{1}, 𝔻_{2}, ..., 𝔻_{n} of a same probability space (Ω, F, ℙ). Using a mutual information criterion, we propose an optimal coding of the set (X_{1}, X_{2}, ..., X_{n}) by the means of an optimal partition of the set ×_{i=1}^{i=n}𝔻_{i} instead of the one usually obtained as the cartesian product of partitions performed on each variable.

After introducing the concepts of the generalized divergence between probability measures and the mutual information between random variables, we will give some properties of them, and we will propose a criterion for partition the set ×_{i=1}^{i=n}𝔻_{i}, which consists in minimizing the loss of mutual information between random variables, when we proceed to data reduction into classes. Finally, we will illustrate this procedure by some examples.

2 Mutual information

2.1 φ-Divergence

Let us remind the following results presented in details by, among others, Csiszár [9, 10, 11], Ali and Silvey [3], and by Zakai and Ziv [27]

Let φ(t) be any convex function from ℝ^{+}\ {0} to ℝ. In order to solve some indetermination, we adopt the following usual conventions:

φ(0) = lim_{t→0^{+}} φ(t)
0φ(\frac{0}{0}) = 0
0φ(\frac{a}{0}) = lim_{δ→0^{+}} δφ(\frac{a}{δ}) = a lim_{δ→0^{+}} δφ(\frac{1}{δ}) a > 0

The following lemma, is due to Csiszár [11]:

Lemma:

Let (X , F, μ) be any measured space (we will suppose however that the measure μ is σ-finite) and let α and β be two non negative measurable functions defined on (X , F, μ). Then:

i)

∫ 𝕀_{A}βφ (\frac{α}{β}) dμ

is defined for each A ∈ F on which α and β are integrable. Moreover if, for such a set A, ∫ I_{A}βdμ is strictly positive and if φ(t) is strictly convex at:

t_{0} = φ (\frac{∫ 𝕀_{A}αdμ}{∫ 𝕀_{A}βdμ})

one has:

ii)

∫ 𝕀_{A}βφ (\frac{α}{β}) dμ ≥ (∫ 𝕀_{A}βdμ) φ (\frac{∫ 𝕀_{A}αdμ}{∫ 𝕀_{A}βdμ}) > −∞

with equality if and only if α = t_{0}β μ − a.s. on A.■

We will assume in the following, that the equalities are considered in the “almost surely” sense.

We consider now two probability measures μ_{1}and μ_{2} defined on (X , F) such that μ_{i} ≪ μ for i = 1, 2. We define the φ-divergence or the generalized divergence between μ_{1} and μ_{2}, by the following quantity:

I_{φ} (μ_{1}, μ_{2}) = ∫ φ (\frac{dμ_{1}}{dμ_{2}} ) dμ_{2} = ∫ φ (\frac{f_{1}}{f_{2}} ) f_{2}dμ

where f_{i} = \frac{dμ_{i}}{dμ} for i = 1, 2. The above lemma ensures the existence of I_{φ} (μ_{1}, μ_{2}) and shows that I_{φ} (μ_{1}, μ_{2}) ≥ φ (1), where equality holds if and only if μ_{1} = μ_{2} in so far as φ is strictly convex at t_{0} = 1. Obviously, I_{φ} (μ_{1}, μ_{2}) does not depend on the choice of μ. Moreover one can, for homogeneous models, write this expression under the form:

I_{φ} (μ_{1}, μ_{2}) = ∫ \frac{dμ_{2}}{dμ_{1}} φ (\frac{dμ_{1}}{dμ_{2}} ) dμ_{1} = ∫ \frac{f_{2}}{f_{1}} φ (\frac{f_{1}}{f_{2}} ) f_{1}dμ

The following table presents, the main measures of φ-divergence:

φ(x)  Name
xLogx; (x − 1) Logx  Kullback and Leibler
|x − 1|  Distance in variation
x^{α}sgn (α − 1) with: 0 < α ≠ 1 Rényi’s divergence of α-order
(\sqrt{x} − 1)^{2}  Hellinger
1 − x^{α} 0 < α < 1  Chernoff
(x − 1)^{2}  χ^{2}
|1 − x^{1/m}|^{m} m > 0  Jeffreys
1 − min(x, 1)  Wald

For more details see, for instance, Goel [17], Adhikari and Joshi [2], Aczél and Daróczy [1], as well as Rényi [23]. Note that φ(1) = 0 except for Rényi’s divergence of α-order.

2.2 Mutual information and properties

Let (Ω, F, ℙ) be a probability space, and let X_{1}, X_{2}, ..., X_{k} be k random variables defined on (Ω, F, ℙ) with values in measured spaces (X_{i}, F_{i}, λ_{i}) i = 1, 2, ..., k. Note by μ_{X_{1},X_{2},...,X_{k}} and by ⊗_{i=1}^{i=k}μ_{X_{i}} the probability measures defined on the product space (×_{i=1}^{i=k}X_{i}, ⊗_{i=1}^{i=k}F_{i}, ⊗_{i=1}^{i=k}λ_{i}), respectively equal to the joint probability measure and to the product of the marginals probability measures associated with X_{1}, X_{2}, ..., X_{k} which are supposed to be absolutely continuous with respect to the product measure λ = ⊗_{i=1}^{i=k}λ_{i}. We define the φ-mutual information or the mutual information between the random variables X_{1}, X_{2}, ..., X_{k}, by:

I_{φ} (X_{1}, X_{2}, ..., X_{k}) = I_{φ} (μ_{X_{1},X_{2},...,X_{k}}, ⨂_{i=1}^{k} μ_{X_{i}})

= ∫ φ (\frac{dμ_{X_{1},X_{2},...,X_{k}}}{d (⨂_{i=1}^{k} μ_{X_{i}})}) d (⨂_{i=1}^{k} μ_{X_{i}})

= ∫ φ (\frac{f_{1}}{f_{2}} ) f_{2}dλ

where, f_{1} and f_{2} are, respectively, the probability density functions of the probability measures μ_{X_{1},X_{2},...,X_{k}} and ⊗_{i=1}^{i=k}μ_{X_{i}} with respect to the product measure λ = ⊗_{i=1}^{i=k}λ_{i}. In a lot of cases, the space X_{i} is for every i either the real line ℝ endowed with the Lebesgue measure, or a discrete space endowed with the counting measure.

Some properties of the mutual information follow (for more details and proofs, one can see, among others, Pinsker [22], Mc Eliece [20], Csiszár [9, 11], Gavurin [15]):

• If φ(1) ≥ 0, then I_{φ} (X_{1}, X_{2}, ..., X_{k}) ≥ 0 where the equality holds if and only if the random variables X_{1}, X_{2}, ..., X_{k} are independent.

• I_{φ} (X_{1}, X_{2}, ..., X_{k}) ≥ I_{φ} (X_{1}, X_{2}, ..., X_{k−1}) where the equality holds if and only if the random variable X_{k} is independent of the random variables X_{1}, X_{2}, ..., X_{k−1}.

• I_{φ} (X_{1}, X_{2}, ..., X_{k}) is convex with respect to μ_{X_{1},X_{2},...,X_{k}}.

• If for every j = 1, 2, ..., k, the applications g_{j} from (×_{i=1}^{i=k}X_{i}, ⊗_{i=1}^{i=k}F_{i}) to (Y_{j}, G_{j}) are measurable, then we have:

I_{φ} (Y_{1}, Y_{2}, ..., Y_{k}) ≤ I_{φ} (X_{1}, X_{2}, ..., X_{k})

where Y_{j} = g_{j} (X_{1}, X_{2}, ..., X_{k}).This last result, known as the “data-processing theorem”, shows that some transformation of the initial variables leads, in general, to a mutual information loss.

• I_{φ} ((X_{1}, X_{2}, ..., X_{k−1}) , X_{k}) = E^{μ^{X_{k}}} (I_{φ} (μ_{X_{1},X_{2},...,X_{k−1}|X_{k}}, μ_{X_{1},X_{2},...,X_{k−1}})) where μ_{X_{1},X_{2},...,X_{k−1}|X_{k}} is the conditional probability measure of the random variables X_{1}, X_{2}, ..., X_{k−1} given X_{k} and where E^{μ^{X_{k}}} denotes the expectation with respect to the probability measure μ_{X_{k}}.

• I_{φ} (X_{1}, X_{2}, ..., X_{k−1}, X_{k}) = E^{μ^{X_{k}}} (I_{φ} (μ_{X_{1},X_{2},...,X_{k−1}|X_{k}}, ⊗_{i=1}^{i=k−1}μ_{X_{k}})).

2.3 Approximation of the mutual information by means of a partition

2.3.1 Main results

The following properties may be considered as a generalization of those obtained, for φ(t) = tLogt, by Dobrushin [13]. Let μ_{X} be a probability measure on (X , F), where X = ×_{i=1}^{i=k}X_{i} = ℝ^{k} (in other words X_{1} = X_{2} = ... = X_{k} = ℝ), absolutely continuous with respect to the Lebesgue measure on ℝ^{k} and with a compact support S_{μ_{X}} which may be assumed of the form ×_{i=1}^{i=k} [a_{i}, b_{i}] (as we will see thereafter, this is not of real importance).

In addition let

P := ×_{i=1}^{i=k}P_{i,n_{i}} = ⋃_{j_{1}=1}^{n_{1}} ⋃_{j_{2}=1}^{n_{2}} ... ⋃_{j_{k}=1}^{n_{k}} (×_{i=1}^{i=k}γ_{ij_{i}})

be a partition of S_{μ_{X}} where for every i = 1, 2, ..., k:

P_{i,n_{i}} = {γ_{ij_{i}} = [x_{ij_{i}−1}, x_{ij_{i}}[ : j_{i} = 1, 2, ..., n_{i} − 1, γ_{in_{i}} = [x_{in_{i}−1}, x_{in_{i}}]}

is a partition (a_{i} = x_{i,0} < x_{i,1} < ... < x_{i,n_{i}} = b_{i}) of [a_{i}, b_{i}] in n_{i} elements. If for every i = 1, 2, ..., k

ξ_{i}^{P} = ∑_{j_{i}=1}^{j_{i}=n_{i}} α_{j_{i}}𝕀_{γ_{ij_{i}}} α_{j_{i}} ∈ ℝ ∀ j_{i} = 1, 2, ..., n_{i}

is a simple random variable defined on [a_{i}, b_{i}], it is easy to verify that the probability measures μ_{ξ}P_{i} and μ_{ξ_{1}^{P} ,ξ_{2}^{P} ,...,ξ_{k}^{P}} are respectively given by:

μ_{ξ}P_{i} = {μ_{X}(γ_{ij_{i}} ×_{r≠i=1}^{r=k} [a_{r}, b_{r}])} : j_{i} = 1, 2, ..., n_{i}}
= {p(γ_{ij_{i}}) : j_{i} = 1, 2, ..., n_{i}} ,

and by:

μ_{ξ_{1}^{P} ,ξ_{2}^{P} ,...,ξ_{k}^{P}} = {μ_{X} (×_{i=1}^{i=k}γ_{i,j_{i}}) ∀ j_{1}, j_{2}, ..., j_{k}}
= {p (γ_{1j_{1}}, γ_{2j_{2}}, ..., γ_{kj_{k}}) : j_{i} = 1, 2, ..., n_{i}} .

By the mean of the “data-processing theorem”, (Colin [7], Tibeiro [26]), we have:

I_{φ} (X_{1}, X_{2}, ..., X_{k}) ≥ I_{φ} (ξ_{1}^{P} , ξ_{2}^{P} , ..., ξ_{k}^{P} )

for every partition P_{1,n_{1}}, P_{2,n_{2}}, ..., P_{k,n_{k}} of [a_{1}, b_{1}] , [a_{2}, b_{2}] , ..., [a_{k}, b_{k}] and consequently the mutual information loss is given by:

I_{φ} (X_{1}, X_{2}, ..., X_{k}) − I_{φ} (ξ_{1}^{P} , ξ_{2}^{P} , ..., ξ_{k}^{P} )

Moreover it can be shown (Serfling [24]) that, under very mild conditions, I_{φ} (ξ_{1}^{P} , ξ_{2}^{P} , ..., ξ_{k}^{P} ) converges in probability to I_{φ} (X_{1}, X_{2}, ..., X_{k}) as n_{i} → ∞, for every i = 1, 2, ..., k.

Remark 1 : As it can be observed, the values taken by the randon variables are meaningless from the mutual information point of view.

Now, let us assume that the integers n_{1}, n_{2}, ..., n_{k} are given. For every i = 1, 2, ..., k, let P_{i}^{n_{i}} be the class of partitions of [a_{i}, b_{i}] in n_{i} disjoint intervals and P be the class of partitions of S_{μ_{X}} given by:

P = ×_{i=1}^{i=k}P_{i}^{n_{i}} .

Each element P of P may be considered as a vector of ℝ^{Σ_{i=1}^{i=k}(n_{i}−1)} having components :

(a_{1}, x_{1,1}, ..., x_{1,n_{1}−1}, b_{1,}a_{2}, x_{2,1}, ...x_{2,n_{2}−1,b_{2}}, ..., a_{k}, x_{k,1}, ..., x_{k,n_{k}−1,b_{k}})

under the constraints:

a_{i} < x_{i,1} < ... < x_{i,n_{i}−1} < b_{i} for every i = 1, 2, ..., k

It is then easy to verify (see, for instance, Ghurye [16], Čencov [6]), that the function of mutual in formation is non-decreasing with respect to the filter of the partitions of S_{μ_{X}}.

If, in the absolute continuous case, the support S_{μ_{X}} of μ_{X} is not compact, as for example in the case of a k-multidimensional normal distribution (S_{μ_{X}} = ℝ^{k}) or a k-multidimensional exponential distribution (S_{μ_{X}} = ℝ^{+k}), one can use a uniform representation of the probability measure μ_{X}, known as “copula” associated to μ_{X}, whose support is the unit cube [0, 1]^{k} of ℝ^{k} (see for more details Fréchet [14], Sklar [25]). By means of this uniform representation, it is then possible to reduce this case to the previous one, as shown by the following lemma:

Lemma: Let us suppose that the probability measure μ_{X} of a random vector X = (X_{1}, X_{2}, ..., X_{k}) ∈ ℝ^{k}, is absolutely continuous with respect to the Lebesgue measure on ℝ^{k}, and let F (X_{1}, X_{2}, ..., X_{k}) be the cumulative distribution function of X. Then the “uniform representation” of the probability measure μ_{X} given by:

U_{1} = F_{1}(X_{1})
U_{2} = F_{2}(X_{2})
⋮
U_{k} = F_{k}(X_{k})

where F_{i}(x_{i}) is, for i = 1, 2, ..., k, the marginal cumulative distribution function of X_{i}, is a mutual information preserving transformation.

Proof: Consider the change of variables U = T (X) defined as follow:

U_{1} = F_{1}(X_{1})
U_{2} = F_{2}(X_{2})
⋮
U_{k} = F_{k}(X_{k}) .

Then, Jacobian of the transformation is given by:

|det [(\frac{∂u_{i}}{∂x_{j}} )_{i,j}]| = det(diag(f_{1}(x_{1}), f_{2}(x_{2}), ..., f_{k}(x_{k})))

= ∏_{i=1}^{i=k} f_{i}(x_{i})

where, for i = 1, 2, ..., k, f_{i}(x_{i}) is the marginal probability density function of X_{i}, and the probability measure μ_{U} of the vector U is absolutely continuous with respect to the Lebesgue measure on [0, 1]^{k} with U_{i} uniform on [0, 1] for i = 1, 2, ..., k. Moreover, if C(u_{1}, u_{2}, ..., u_{k}) is the cumulative distribution function of U, one has:

C(u_{1}, u_{2}, ..., u_{k}) = F (F_{1}^{−1}(u_{1}), F_{2}^{−1}(u_{2}), ..., F_{k}^{−1}(u_{k}))

Therefore:

I_{φ} (X_{1}, X_{2}, ..., X_{k}) = I_{φ} (μ_{X_{1},X_{2},...,X_{k}}, ⨂_{i=1}^{k} μ_{X_{i}})

= ∫_{ℝ^{k}} φ (\frac{dμ_{X_{1},X_{2},...,X_{k}}}{d (⨂_{i=1}^{k} μ_{X_{i}})}) d (⨂_{i=1}^{k} μ_{X_{i}})

= ∫_{ℝ^{k}} φ (\frac{f(x_{1}, x_{2}, ..., x_{k})}{∏_{i=1}^{i=k} f_{i}(x_{i})}) ( ∏_{i=1}^{i=k} f_{i}(x_{i}))dx_{1}dx_{2}...dx_{k}

where f(x_{1}, x_{2}, ..., x_{k}) is the probability density function of X. Using the above transformation U = T (X), it follows that:

I_{φ} (X_{1}, X_{2}, ..., X_{k}) = ∫_{ℝ^{k}} φ (\frac{f(x_{1}, x_{2}, ..., x_{k})}{∏_{i=1}^{i=k} f_{i}(x_{i})}) ( ∏_{i=1}^{i=k} f_{i}(x_{i}))dx_{1}dx_{2}...dx_{k}

= ∫_{[0,1]^{k}} φ [(\frac{f(F_{1}^{−1}(u_{1}), F_{2}^{−1}(u_{2}), ..., F_{k}^{−1}(u_{k}))}{∏_{i=1}^{i=k} f_{i}(F_{i}^{−1}(u_{i}))})]

× ∏_{i=1}^{i=k} f_{i}(F_{i}^{−1}(u_{i})) det J^{−1}|_{T^{−1}(x)} du_{1}du_{2}...du_{k}

= ∫_{[0,1]^{k}} φ(c(u_{1}, u_{2}, ..., u_{k}))du_{1}du_{2}...du_{k}

where c(u_{1}, u_{2}, ..., u_{k}) is nothing less than the probability density function of the copula U associated to X. Since the components U_{i}, for i = 1, 2, ..., k, are uniform [0, 1] random variables, it follows that:

I_{φ} (X_{1}, X_{2}, ..., X_{k}) = ∫_{[0,1]^{k}} φ(c(u_{1}, u_{2}, ..., u_{k}))du_{1}du_{2}...du_{k}
= I_{φ} (U_{1}, U_{2}, ..., U_{k}) . ■

So, the mutual informations I_{φ} (X_{1}, X_{2}, ..., X_{k}) and I_{φ} (U_{1}, U_{2}, ..., U_{k}) are the same and consequently, it is always possible to consider the given probability measure as a probability measure on a compact support, without loss of information on the stochastic dependence between the components X_{1}, X_{2}, ..., X_{k} or U_{1}, U_{2}, ..., U_{k}.

Remark 2 : This property is a special case of the invariance of mutual information under one-to-one mapping.

2.3.2 Existence of an optimal partition

If according to the above assumption, the integers n_{1}, n_{2}, ..., n_{k} are given, then in order to obtained an optimal partition of S_{μ_{X}} (i.e a partition with a given number of elements for which the mutual information loss is minimum), we have to solve the following optimization problem:

min_{P ∈P} (I_{φ} (X_{1}, X_{2}, ..., X_{k}) − I_{φ} (ξ_{1}^{P} , ξ_{2}^{P} , ..., ξ_{k}^{P} ))

which is equivalent to:

max_{P ∈P}I_{φ} (ξ_{1}^{P} , ξ_{2}^{P} , ..., ξ_{k}^{P} ) = max_{P ∈P} ∑_{(j_{1},j_{2},...,j_{k})} (∏_{i=1}^{i=k} p(γ_{ij_{i}})φ (\frac{p (γ_{1j_{1}}, γ_{2j_{2}}, ..., γ_{kj_{k}})}{∏_{i=1}^{i=k} p(γ_{ij_{i}})}))

Whithout loss of generality, we may assume that the function I_{φ} (ξ_{1}^{P} , ξ_{2}^{P} , ..., ξ_{k}^{P} ) of the variables x_{ij_{i}} for i = 1, 2, ..., k and j_{i} = 1, 2, ..., n_{i} − 1, is defined on a compact subset of ℝ^{Σ_{i=1}^{i=k}(n_{i}−1)} of the form:

S = ×_{i=1}^{i=k}S_{i,n_{i}}

where S_{i,n_{i}} is the subset of ℝ^{n_{i}−1} given by:

a_{i} = x_{i,0} < x_{i,1} < ... < x_{i,n_{i}} = b_{i}

Since the real function I_{φ} (ξ_{1}^{P} , ξ_{2}^{P} , ..., ξ_{k}^{P} ) is continuous on a compact subset of

ℝ^{Σ_{i=1}^{i=k}(n_{i}−1)}, the existence of an optimal partition

P̃ = ×_{i=1}^{i=k}P̃_{i,n_{i}}

where:

P̃_{i,n_{i}} = (a_{i} < x_{i,1}^{∗} < ... < x_{i,n_{i}−1}^{∗} < b) i = 1, 2, ..., k

and where:

I_{φ} (ξP̃ ; 1, ξP̃ ; 2, ..., ξP̃ ; k) = max_{P ∈P}I_{φ} (ξ_{1}^{P} , ξ_{2}^{P} , ..., ξ_{k}^{P} )

is then ensured (for a complete proof and details, one may see Colin [8] and de Tibeiro [26]). Furthermore, since we deal with ∏_{i=1}^{i=k}(n_{i} − 1) variables, which is often a large number in practice, we have to use numerical methods to find a solution to the problem of optimization.

3 Estimation and asymptotic normality

In practice, we do not know the probability measure μ_{X} of the random vector X but only a number n of independent observations X^{r} for r = 1, 2, ..., n. In order to obtain an optimal partition of S_{μ_{X}} (thereafter called an “optimal coding”), we have to estimate the probability measure μ_{X}. Several cases may arise.

1. We only know the empirical measure μ_{X}^{n} = \frac{1}{n} ∑_{r=1}^{r=n} 𝕀 (X^{r}) (X^{r} ∈ ℝ^{k} for all r = 1, 2, ..., n) or, which is equivalent, the empirical probability distribution function

F_{n} (x) = \frac{1}{n} ∑_{r=1}^{r=n} 𝕀_{]−∞,x]} (X^{r}) ∀x ∈ ℝ^{k}

From the estimation point of view, this case is the same as the following one by making categories for each components of X.

2. A priori we know, for every random variables X_{i} i = 1, 2, ..., r, the set {γ_{ij_{i}}} of categories or classes of X_{i}, as for instance for sample surveys or census where the categories are given once for all. In that case, all what we know is the number of observations in each cell of the form: ×_{i=1}^{i=k}γ_{ij_{i}}.

3. In addtion to the empirical measure μ_{X}^{n}, we assume that the probability measure μ_{X} is a member of a given class (as, for instance, multivariate normal, multivariate exponential,...) and that its probability density function f (x, θ) depends on an unknown parameter θ ∈ Θ ⊂ ℝ^{m}. This third case is often more tractable than the second one and it can be reduce to the estimation of the parameter θ.

As an illustration of the implementation of the algorithm of optimal coding, we only consider the third case. Results about the first two ones, can be find, among others, in Colin [8], Darbellay [12] or Moddemeijer [21].

Under the assumption on μ_{X}, it seems natural to choose as an estimator of the mutual information, the one based on f (x, θ̂) where θ̂ is an estimator of θ. In other words, since I_{φ} (X_{1}, X_{2}, ..., X_{k}) = I_{φ} (X; θ) may be generaly considered as a continuous and differentiable function g (θ) = g (θ_{1}, θ_{2}, ..., θ_{m}) from ℝ^{m} to ℝ, we can choose as an estimator of I_{φ} (X; θ) the random variable given, with obvious notations, by:

Î_{φ} (X; θ) = ∫_{ℝ^{k}} φ (\frac{f (x_{1}, x_{2}, ..., x_{k}, θ̂)}{∏_{i=1}^{i=k} f_{i} (x_{i}, θ̂)}) (∏_{i=1}^{i=k} f_{i} (x_{i}, θ̂)) dx_{1}dx_{2}...dx_{k}

= I_{φ} (X; θ̂)

or, if μ_{U} is the copula associated to μ_{X}, by:

Î_{φ} (U; η) = I_{φ} (U; η̂) = ∫_{[0,1]_{k}} φ (c (u_{1}, u_{2}, ..., u_{k}, η̂)) du_{1}du_{2}...du_{k}

where η̂ = η(θ̂) is the estimator of the parameter η = η(θ) of the probability density function c (u_{1}, u_{2}, ..., u_{k}, η) of μ_{U}.

It is clear that the properties of the estimator Î_{φ} (X; θ) depend closely on those of the estimator θ̂ of θ. For instance, if θ̂ is the maximum likelihood estimator of θ we know (Lehmann [18], Serfling [24]) that, under mild conditions, a sequence of solutions (θ̂_{n}) of the system of likelihood equations exists and converges almost surely towards θ. Moreover, for such a sequence we have:

\sqrt{n} (θ̂_{n} − θ) →^{L} N (0, I_{X}^{−1})

where I_{X} is the Fisher information matrix relative to Θ. Hence it follows that, Î_{φ} (X; θ) is, under usual conditions of regularity on g (θ) (Serfling [24]), a maximum likelihood estimator of I_{φ} (X; θ) and that:

\sqrt{n} (I_{φ} (X, θ̂_{n}) − I_{φ} (X, θ)) →^{L} N (0, ^{t}∇g_{θ}I_{X}^{−1}∇g_{θ})

if ∇g_{θ} is different from zero

Example 1: φ(t) = tLogt

One has:

I_{φ} (X, θ) = ∫_{ℝ^{k}} φ (\frac{f (x, θ)}{∏_{i=1}^{i=k} f_{i} (x_{i}, θ)}) (∏_{i=1}^{i=k} f_{i} (x_{i}, θ)) dx_{1}dx_{2}...dx_{k}

= ∫_{ℝ^{k}} f (x, θ) Log (\frac{f (x, θ)}{∏_{i=1}^{i=k} f_{i} (x_{i}, θ)}) dx_{1}dx_{2}...dx_{k}

= ∫_{ℝ^{k}} f (x, θ) Log (f (x, θ)) dx_{1}dx_{2}...dx_{k}

− ∑_{i=1}^{i=k} ∫_{ℝ} f_{i} (x_{i}, θ) Log (f_{i} (x_{i}, θ)) dx_{i}

or:

I_{φ} (X, θ) = 𝔼 (Log (f (X, θ))) − ∑_{i=1}^{i=k} 𝔼 (Log (f_{i} (X_{i}, θ)))

Moreover, if X is a k-multivariate normal random vector N (μ, Σ) we have, using the usual notations:

I_{φ} (X, Σ) = −\frac{1}{2}Log ((2π)^{k} det Σ) − \frac{1}{2}𝔼 (^{t} (X − μ) Σ^{−1} (X − μ))

+Log ( ∏_{i=1}^{i=k} \sqrt{2πe}σ_{X_{i}})

= \frac{k}{2} + \frac{1}{2}Log (\frac{∏_{i=1}^{i=k} σ_{X_{i}}^{2}}{det Σ} ) − \frac{1}{2}𝔼 (^{t} (X − μ) Σ^{−1} (X − μ))

= \frac{1}{2}Log (\frac{∏_{i=1}^{i=k} σ_{X_{i}}^{2}}{det Σ} )

and then the estimator of I_{φ} (X, θ) is given by:

Î_{φ} (X, Σ) = I_{φ} (X, Σ̂) = \frac{1}{2}Log ( ∏_{i=1}^{i=k} σ̂_{X_{i}}^{2} (det Σ̂)^{−1})

where Σ̂ is the maximum likelihood estimator of Σ. Since Σ̂ →^{p} Σ, we obtain from the above equality:

I_{φ} (X, Σ̂) →^{p} I_{φ} (X, Σ)

If, for instance, Σ = (1 − ρ) 𝕀+ρ𝕁 (equicorrelation matrix) with ρ > 0, we have (Mardia, Kent and Bibby [19]):

det Σ = (1 − ρ)^{k−1} (1 + ρ (k − 1)) ρ ≠ 1

and then:

I_{φ} (X, ρ) = −\frac{1}{2}Log ((1 − ρ)^{k−1} (1 + ρ (k − 1)))

It can be noticed that I_{φ} (X, 0) = 0 and lim_{ρ→1}I_{φ} (X, ρ) = ∞.

Let ρ̂ be the maximum likelihood estimator of ρ then:

I_{φ} (X, ρ̂) = −\frac{1}{2}Log ((1 − ρ̂)^{k−1} (1 + ρ̂ (k − 1)))

Since (Lehmann [18]):

\sqrt{n} (ρ̂ − ρ) →^{L} N (0, \frac{(1 − ρ^{2})^{2}}{1 + ρ^{2}} )

and:

\frac{∂}{∂ρ}I_{φ} (X, ρ) = \frac{ρk (k − 1)}{2 (1 − ρ) (1 + ρ (k − 1))}

it follows that:

\sqrt{n} (I_{φ} (X, ρ̂) − I_{φ} (X, ρ)) →^{L} N (0, \frac{1}{4} \frac{ρ^{2}k^{2} (k − 1)^{2} (1 + ρ)^{2}}{(1 + ρ (k − 1))^{2} (1 + ρ^{2})})

So I_{φ} (X, ρ̂) is a consistent and asymptotically normal estimator and converges in probability towards I_{φ} (X, ρ).

Example 2: φ(t) = (t − 1)^{2}

Let us assume that X = (X_{1}, X_{2}) is distributed as a bivariate Gumbel Gm (α) with parameter α. The cumulative distribution function is given by:

F_{X_{1},X2} (x_{1}, x_{2}) = (1 − e^{−x_{1}}) (1 − e^{−x_{2}}) (1 + αe^{−x_{1}−x_{2}}) x_{1}, x_{2} ≥ 0 ; −1 ≤ α ≤ 1

The cumulative distribution function and the probability density function of the associated copula U = (U_{1}, U_{2}) are respectively given by:

F_{U_{1},U_{2}} (u_{1}, u_{2}) = u_{1}u_{2} (1 + α (1 − u_{1}) (1 − u_{2}))

and by:

f_{U_{1},U_{2}} (u_{1}, u_{2}) = 1 + α (1 − 2u_{1}) (1 − 2u_{2}) 𝕀_{[0,1]}^{2} (u_{1}, u_{2})

which is represented, for α = 0.5 just below at Fig 1

Fig 1

Then it follows that:

I_{φ} (X, α) = I_{φ} (U, α) = α^{2} ∫_{0}^{1} ∫_{0}^{1} (1 − 2u_{1})^{2} (1 − 2u_{2})^{2} du_{1}du_{2} = \frac{α^{2}}{9}

An easy calculation shows that α = 4ρ_{X_{1},X_{2}} = 3ρ_{U_{1},U_{2}}. Therefore, if ρ̂ denotes the usual estimator of ρ_{X_{1},X_{2}}, the mutual information estimator has the following form:

I_{φ} (X, α̂) = (\frac{4}{3}ρ̂)^{2}

If n is the number of observations of the random vector X we know (Serfling [24]) that:

\sqrt{n} (ρ̂ − ρ) →^{L} N (0,^{t}aΣa)

where a is a vector whose components are functions of the first and second moments of μ_{X} only and where Σ is the variance-covariance matrix of the vector:

(\overline{X}_{1}, \overline{X}_{2}, \frac{1}{n} ∑_{i=1}^{i=n} X_{1i}^{2}, \frac{1}{n} ∑_{i=1}^{i=n} X_{2i}^{2}, \frac{1}{n} ∑_{i=1}^{i=n} X_{1i}X_{2i})

From this, it follows that:

\sqrt{n} (I_{φ} (X, α̂) − \frac{16}{9}ρ^{2}) →^{L} N (0, \frac{64}{9}ρ^{2} (^{t}aΣa))

and that:

I_{φ} (X, α̂) →^{p} \frac{16}{9}ρ^{2}

As in the above example, we can say that I_{φ} (X, α̂) is a consistent and asymptotically normal estimator which converges in probability to I_{φ} (X, α).

4 Computational aspects and examples

Without loss of generality, one can consider in order to facilitate the presentation, the case of a random vector (X, Y) in ℝ^{2} with density f(x, y) whose support is [0, 1]^{2}. For each component, let respectively:

0 = x_{0} < x_{1} < x_{2}... < x_{i} < ... < x_{p} < x_{p+1} = 1

and

0 = y_{0} < y_{1} < y_{2}... < y_{j} < ... < y_{q} < y_{q+1} = 1

be two partitions of the intervals [0, 1] The joint probability measure of a rectangle [x_{i}, x_{i+1}[×[y_{j}, y_{j+1}[ is given by:

∫_{x_{i}}^{x_{i+1}}∫_{y_{j}}^{y_{j+1}}f(x, y)dxdy = p_{i,j}

while its product probability measure is express as:

(∫_{x_{i}}^{x_{i+1}}f_{1}(x)dx) (∫_{y_{j}}^{y_{j+1}}f_{2}(y)dy) = p_{i·}p_{·j}

One can easily check that: p_{i·} = ∑_{j=0}^{j=q} p_{i,j} and that : p_{·j} = ∑_{i=0}^{i=p} p_{i,j}. Moreover one has:

∑_{i=0}^{i=p} ∑_{j=0}^{j=q} p_{i,j} = ∑_{i=0}^{i=p} p_{i·} = ∑_{j=0}^{j=q} p_{·j} = 1

The approximation of the mutual information between the random variables X and Y, contained in the discrete joint probability measure {p_{ij}} for i = 0, 1, ..., p and y_{j} j = 0, 1, ..., q, is given by:

∑_{i=0}^{i=p} ∑_{j=0}^{j=q} φ (\frac{p_{i,j}}{p_{i·}p_{·j}} ) p_{i·}p_{·j}

or equivalently, using variables x_{i} and y_{j}, one has:

∑_{i=0}^{i=p} ∑_{j=0}^{j=q} φ (\frac{∫_{x_{i}}^{x_{i+1}}∫_{y_{j}}^{y_{j+1}}f(x, y)dxdy}{(∫_{x_{i}}^{x_{i+1}}f_{1}(x)dx) × (∫_{y_{j}}^{y_{j+1}}f_{2}(y)dy)}) (∫_{x_{i}}^{x_{i+1}}f_{1}(x)dx)×(∫_{y_{j}}^{y_{j+1}}f_{2}(y)dy)

For given p, q and f(x, y) (however in practice, one has often an functional estimation of f(x, y) instead of the true probabilty density), it follows that the optimisation problem may be express as:

max_{\{x_{i}\},\{y_{j}\}} ∑_{i=0}^{i=p} ∑_{j=0}^{j=q} φ (\frac{∫_{x_{i}}^{x_{i+1}}∫_{y_{j}}^{y_{j+1}}f(x, y)dxdy}{(∫_{x_{i}}^{x_{i+1}}f_{1}(x)dx) × (∫_{y_{j}}^{y_{j+1}}f_{2}(y)dy)}) (∫_{x_{i}}^{x_{i+1}}f_{1}(x)dx)×(∫_{y_{j}}^{y_{j+1}}f_{2}(y)dy)

and it is reduced to find the maximum of a continuous and differentiable function of p + q variables defined on a compact subset of ℝ^{p+q} under linear constraints. In order to solve this problem, many methods are available and for efficiency reasons one has choosen the method described in Zoutendijk [28]

Example 1:

As an illustration of the previous results, we present thereafter a short example from de Tibeiro [26].

We consider the function φ (t) = (t − 1)^{2} and a probability measure on [0, 1]^{2} which joint probability density function (with respect to the Lebesgue measure) is given by:

f (x, y) = (x + y) 𝕀_{[0,1]×[0,1]} (x, y)

Fig 2

A straightforward calculation shows that:

I_{(t−1)}^{2} (X, Y) = 9, 7 × 10^{−3}

Since I_{(t−1)}^{2} (X, Y) is a continuously differentiable function of the variables (x_{ij_{i}}), we obtained for p = q = 2 (using for instance the Maple software), the following optimal partition of [0, 1]^{2} (illustraded below):

x_{1} = y_{1} = 0, 2211 ; x_{2} = y_{2} = 0, 54

Optimal partition

Since the probability density function of the vector (X, Y) is symmetric with respect to x and y, the randon variables ξP̃ ; 1 and ξP̃ ; 2 are the same and the probabilty measures μ_{ξP̃ ; 1,ξP̃ ; 2} , μ_{ξ}P̃ ; 1 and μ_{ξ}P̃ ; 2 are respectively given by:

\begin{matrix}μ^{ξP̃ ; 1,ξP̃ ; 2} & [0, 0, 2211[ & [0, 2211, 0, 54[ & [0, 54, 1] \\ [0, 0, 2211[ & 0, 0108 & 0, 0346 & 0, 0895 \\ [0, 2211, 0, 54[ & 0, 0346 & 0, 0774 & 0, 1687 \\ [0, 54, 1] & 0, 0895 & 0, 1687 & 0, 3258 \\ μξP̃ ; 2 & 0, 1349 & 0, 2807 & 0, 5840 \end{matrix}

Let {p_{ij}}, {p_{i·}} and {p_{·j}} be respectively the probability density functions of (ξP̃ ; 1, ξP̃ ; 2), ξP̃ ; 1 and (ξP̃ ; 2). For φ (t) = (t − 1)^{2} it is easy to check that:

I_{(t−1)}^{2} (ξP̃ ; 1, ξP̃ ; 2) = ∑_{i} ∑_{j} \frac{p_{ij}^{2}}{p_{i·}p_{·j}} − 1

and after calculation one has: I_{(t−1)}^{2} (ξP̃ ; 1, ξP̃ ; 2) = 7 × 10^{−3} which represents 72 percent of the initial mutual information. As comparison, in the case of a regular partition (each class has the same width), the probability measure μ_{ξP̃ ; 1,ξP̃ ; 2} is given by:

\begin{matrix}μ^{ξ_{1}^{Preg} ,ξ_{2}^{Preg}} & [0; \frac{1}{3}[ & [\frac{1}{3}; \frac{2}{3}[ & [\frac{2}{3}; 1] \\ [0; \frac{1}{3}[ & 0, 0370 & 0, 0740 & 0, 1111 \\ [\frac{1}{3}; \frac{2}{3}[ & 0, 0740 & 0, 1111 & 0, 1481 \\ [\frac{2}{3}; 1] & 0, 1111 & 0, 1481 & 0, 1852 \\ μξ{Preg \atop 2} & 0, 2221 & 0, 3332 & 0, 4444 \end{matrix}

while this probability measure is, for an equi-partition (each class has the same frequency), equals to:

\begin{matrix}μ^{ξ_{1}^{P_{équi}} ,ξ_{2}^{P_{équi}}} & [0; 0, 4574[ & [0, 4574; 0, 7583[ & [0, 7583; 1] \\ [0; 0, 4574[ & 0, 09572 & 0, 11513 & 0, 12248 \\ [0, 4574; 0, 7583[ & 0, 11513 & 0, 11005 & 0, 10815 \\ [0, 7583; 1] & 0, 12248 & 0, 10815 & 0, 10270 \\ μξ{P_{équi} \atop 2} & 0, 3333 & 0, 3333 & 0, 3333 \end{matrix}

One can easily check that:

I_{(t−1)}^{2} (ξ_{1}^{P_{reg}}, ξ_{2}^{P_{reg}}) = 5, 44 × 10^{−3} (56% of I_{(t−1)}^{2} (X, Y)) ,

in the first case and:

I_{(t−1)}^{2} (ξ_{1}^{P_{équi}}, ξ_{2}^{P_{équi}}) = 5, 6 × 10^{−3} (58% of I_{(t−1)}^{2} (X, Y)) ,

in the second one.

An optimal partition of [0, 1]^{2} with 4 elements (p = q = 1) would give:

\begin{matrix}μ^{ξP̃ ; 1,ξP̃ ; 2} & [0, 0, 361[ & [0, 361, 1[ \\ [0, 0, 361[ & 0, 050 & 0, 205 \\ [0, 361, 1[ & 0, 205 & 0, 540 \\ μξP̃ ; 2 & 0, 255 & 0, 745 \end{matrix}

with I_{(t−1)}^{2} (ξP̃ ; 1, ξP̃ ; 2) = 5 × 10^{−3} (52% of the initial information). Finally, it is obvious that if we choose an other function φ, we will usually obtain an other optimal partition of [0, 1]^{2}.

Following, are some others examples with uneven number of classes by variable.

• p = 2, q = 3.

μ^{ξP̃ ; 1,ξP̃ ; 2}  [0, 0, 1580[ [0, 1580, 0, 3660[ [0, 3660, 0, 6397[ [0, 6397, 1] μ_{ξ}P̃ ; 1
[0, 0, 3660[  0, 0066  0, 0171  0, 0371  0, 0741 0, 1348
[0, 3660, 0, 54[  0, 0232  0, 0426  0, 0771  0, 1379 0, 2808
[0, 54, 1]  0, 0617  0, 0987  0, 1630  0, 2634 0, 5864
μ^{ξ}P̃ ; 2  0, 0915  0, 1584  0, 2772  0, 4754

For these partitions one has: I_{(t−1)}^{2} (ξP̃ ; 1, ξP̃ ; 2) = 8, 7731 × 10^{−3} (90, 4% of I_{(t−1)}^{2} (X, Y)).

• p = 3, q = 4.

(for page layout reasons, the variables X and Y has been permutated in the following table )

μ^{ξP̃ ; 1,ξP̃ ; 2}  [0, 0, 1581[ [0, 1581, 0, 3661[ [0, 3661, 0, 6397[ [0, 6397, 1] μ_{ξ}P̃ ; 2
[0, 0, 1229[  0, 0027  0, 0083  0, 0190  0, 0390 0, 0690
[0, 1229, 0, 2759[  0, 0067  0, 0147  0, 0294  0, 0562 0, 1070
[0, 2759, 0, 4666[  0, 0136  0, 0251  0, 0456  0, 0818 0, 1661
[0, 4666, 0, 7041[  0, 0249  0, 0419  0, 0708  0, 1203 0, 2579
[0, 7041, 1]  0, 0435  0, 0686  0, 1097  0, 1782 0, 4000
μ^{ξ}P̃ ; 1  0, 0914  0, 1586  0, 2745  0, 4755

In this case, one can check that I_{(t−1)}^{2} (ξP̃ ; 1, ξP̃ ; 2) = 8, 81 × 10^{−3} (90, 8% of I_{(t−1)}^{2} (X, Y)).

Example 2:

Once again, we consider the function φ (t) = (t − 1)^{2} and a probability measure on [0, 1]^{2} for which its joint probability density function (with respect to the Lebesgue measure) is given by:

f (x, y) = \frac{2}{3} (2x + y) 𝕀_{[0,1]×[0,1]} (x, y)

Fig 3

The marginal probability density functions f_{1}(x) and f_{2}(y) are then respectively given by:

f_{1}(x) = \frac{(4x + 1)}{3}𝕀_{[0,1]}(x)

and by:

f_{2}(y) = \frac{2(y + 1)}{3}𝕀_{[0,1]}(y)

Finally an easy calculation shows that:

I_{(t−1)}^{2} (X, Y) = 8, 22 × 10^{−3}

In the case, of p = 2 and q = 3, the solution is given by:

μ^{ξP̃ ; 1,ξP̃ ; 2}  [0; 0, 1892[ [0, 1892; 0, 4143[ [0, 4143; 0, 6818] [0, 6818; 1]  ξP̃ ; 1
[0; 0, 1775[  0, 0061  0, 0128  0, 0230  0, 0383 0, 0802
[0, 1775; 0, 4810[  0, 0288  0, 0437  0, 0653  0, 0965 0, 2343
[0, 4810; 1]  0, 1031  0, 1388  0, 1879  0, 2556 0, 6854
ξP̃ ; 2  0, 1380  0, 1953  0, 2762  0, 3904

After calculation, one has: I_{(t−1)}^{2} (ξP̃ ; 1, ξP̃ ; 2) = 6, 6 × 10^{−3} which represents 80 percent of the initial mutual information.

Some others examples are given below:

• p = q = 3.

μ^{ξP̃ ; 1,ξP̃ ; 2}  [0; 0, 1892[ [0, 1892; 0, 4143[ [0, 4143; 0, 6818] [0, 6818; 1]  ξP̃ ; 1
[0; 0, 1238[  0, 0034  0, 0079  0, 0148  0, 0253 0, 0514
[0, 1238; 0, 3090[  0, 0123  0, 0204  0, 0324  0, 0500 0, 1151
[0, 3090; 0, 5860]  0, 0346  0, 0497  0, 0713  0, 1020 0, 2576
]0, 5860; 1]  0, 0878  0, 1173  0, 1576  0, 2131 0, 5758
ξP̃ ; 2  0, 1381  0, 1953  0, 2761  0, 3904

with: I_{(t−1)}^{2} (ξP̃ ; 1, ξP̃ ; 2) = 7, 1×10^{−3} which represents 86 percent of I_{(t−1)}^{2} (X, Y).

• p = 3, q = 5 (In the following example ξP̃ ; 1 and ξP̃ ; 2 are permuted).

μ^{ξP̃ ; 1,ξP̃ ; 2}  [0; 0, 1238[ [0, 1238; 0, 3090[ [0, 3090; 0, 5859] [0, 5859; 1]  ξP̃ ; 2
[0; 0, 1224[  0, 0019  0, 0075  0, 0216  0, 0557 0, 0866
[0, 1224; 0, 2599[  0, 0036  0, 0106  0, 0276  0, 0674 0, 1091
[0, 2599; 0, 4142]  0, 0059  0, 0147  0, 0351  0, 0819 0, 1375
]0, 4142; 0, 5874]  0, 0089  0, 0200  0, 0446  0, 0998 0, 1733
]0, 5874; 0, 7818]  0, 0130  0, 0268  0, 0567  0, 1218 0, 2183
]0, 7818; 1]  0, 0183  0, 0357  0, 0719  0, 1492 0, 2751
ξP̃ ; 1  0, 0515  0, 1152  0, 2575  0, 5758

for which one has I_{(t−1)}^{2} (ξP̃ ; 1, ξP̃ ; 2) = 7, 8 × 10^{−3} or 95 percent of I_{(t−1)}^{2} (X, Y).

Remark 3 : If the random variables are discrete, it is more difficult to find a solution of the optimization problem and we have to use some others methods or some approximations or heuristics whose can be found, for instance, in Beasley [4]

References

[1] J. Aczél and Z. Daróczy, On measures of information and their characterizations, Academic Press New York, (1975).

[2] B.P. Adhikari and D.D. Joshi, Distance Discrimination et Résumé exhaustif, Publications de l’Institut de Statistique de l’Université de Paris, 5 (1956), 57-74.

[3] S.M. Ali and S.D. Silvey, A general class of coefficients of divergence of one distribution from another, J.Roy.Statist.Soc., B.28 (1966), 131-142.

[4] J.E. Beasley, An algorithm for set covering problems, European Journal of Operational Research, vol.31 (1987), 85-93.

[5] D. Bosq et J.P. Lecoutre, Théorie de l’estimation fonctionnelle, Economica (1987).

[6] N.N. Čencov, Statistical Decisions Rules and Optimal Inference, American Mathematical Society, Translations of Mathematical Monographs, Vol 53 (1982).

[7] B. Colin, Sur la reconstitution optimale des données, Cahiers du CERO, Vol 37 (1995), 65-95.

[8] B. Colin, J. Vaillant et M. Troupé, Information mutuelle et divergence : estimation, codage et classification, Rapport de recherche No 11, Département de Mathématiques, Université de Sherbrooke (Québec) (http://www.usherbrooke.ca/mathematiques/telechargement/) (2004).

[9] I. Csiszár, Information-type measures of difference of probability distributions and indirect observations, Studia Scientiarum Mathematicarum Hungarica, 2 (1967), 299-318.

[10] I. Csiszár, A class of measures of informativity of observation channels, Periodica Mathematica Hungarica, Vol 2 (1-4) (1972), 191-213.

[11] I. Csiszár, Information measures : A critical survey, Transaction of the seventh Prague Conference on Information theory Statistical Decision functions Random Processes. Publishing House of the Czechoslovak Academy of Sciences, Vol A Prague (1977), 73-86.

[12] G.A. Darbellay, An estimator of the mutual information based on criterion for independance, Computational Statistics & Data Analysis, 32 (1999), 1-17.

[13] R.L. Dobrushin, General formulation of Shannon’s main theorem in information theory, Amer. Math. Soc Transl. (2), Vol 33 (1963), 323-438.

[14] M. Fréchet, Sur les tableaux de corrélation dont les marges sont données, Ann. Univ. Lyon, Sér3, 14 (1951), 53-77.

[15] M.K. Gavurin, On the value of Information, Vestuik Leningrad University Series, 4 (1963), 27-34. Translation (1968), Selected Translations in Mathematical Statistics and Probability ,7 (1968), 193-202.

[16] S.G. Ghurye, Information and sufficient sub-fields, The Annals of Mathematical Statistics, Vol 38 no. 6 (1968), 2056-2066.

[17] P.K. Goël, Information measures and Bayesian Hierarcichal Models, Departement of Statistics, Purdue University, West Lafayette, Technical Report, # 81-41 (1981).

[18] E.L. Lehmann, Theory of point estimation, Wadsworth & Brooks, (1991).

[19] K.V. Mardia, J.T. Kent and J.M. Bibby, Multivariate Analysis, Academic Press Second Printing, (1980).

[20] R.J. McEliece, The theory of information coding, Encyclopedia of mathematics and its applications, Addison Wesley, (1977).

[21] R. Moddemeijer, On estimation of entropy and mutual information of continuous distributions, Signal Processing, 16 (1989), 233-248.

[22] M.S. Pinsker, Information and information stability of random variables and processes, Holden-Day, (1964).

[23] A. Rényi, On measures of entropy and information, Proceedings of the Fourth Berkeley Symposium of Mathematical Statistics and Probability, Vol 1 (1961), Berkeley : University of California Press, 547-561.

[24] R.J. Serfling, Approximation Theorems of Mathematical Statistics, Wiley, New York (1980).

[25] A. Sklar, Fonctions de répartition à n dimensions et leurs marges, Publ. Inst. Statist. Univ. Paris, 8 (1959), 229-231.

[26] J.S. de Tibeiro, Information et analyse des données, Thèse de Doctorat, Département de Mathématiques et d’Informatique, Université de Sherbrooke (Québec), Canada, (1993).

[27] J. Zakai and M. Ziv, On functionnals satisfying a data-processing theorem, IEEE Transactions, IT-19 (1973), 275-282.

[28] G. Zoutendijk, Methods of feasible directions, Elsevier, Amsterdam and D. VanNostrand, Princeton, N.J, 1960.