On improved predictive density estimation with parametric constraints

Dominique FOURDRINIER ^{∗} , Éric MARCHAND ^{†} , Ali RIGHI ^{‡} and William E. STRAWDERMAN ^{§}

October 25, 2010

Abstract

We consider the problem of predictive density estimation for normal models under Kullback-Leibler loss (KL loss) when the parameter space is constrained to a convex set. More particularly, we assume that X ∼ N_{p}(μ, v_{x}I) is observed and that we wish to estimate the density of Y ∼ N_{p}(μ, v_{y}I) under KL loss when μ is restricted to the convex set C ⊂ ℝ^{p}. We show that the best unrestricted invariant predictive density estimator p̂_{U} is dominated by the Bayes estimator p̂_{π_{C}} associated to the uniform prior π_{C} on C. We also study so called plug-in estimators, giving conditions under which domination of one estimator of the mean vector μ over another under the usual quadratic loss, translates into a domination result for certain corresponding plug-in density estimators under KL loss. Risk comparisons and domination results are also made for comparisons of plug-in estimators and Bayes predictive density estimators. Additionally, minimaxity and domination results are given for the cases where: (i) C is a cone, and (ii) C is a ball.

AMS 2010 subject classifications. Primary 62C15, 62C20, 62F10, 62H12.

Keywords and phrases: predictive estimation, risk function, quadratic loss, Kullback-Leibler loss, uniform priors, Bayes estimators, convex sets, cones, multivariate normal.

∗LITIS, EA 4108, Université de Rouen, Avenue de l’Université, BP 12, 76801 Saint-Étienne-du-Rouvray, France. The support of the ANR grant 08-EMER-002 is gratefully acknowledged.

† Département de mathématiques, Université de Sherbrooke, Sherbrooke, QC, CANADA, J1K 2R1.

The research support of NSERC of Canada is gratefully acknowledged.

‡ LMRS, UMR 6085, Université de Rouen, France Avenue de l’Université, BP 12, 76801 Saint-Étienne-du-Rouvray, France.

§ Department of Statistics, Rutgers University, 561 Hill Center, Busch Campus, Piscataway, N.J., USA 08854-8019.

1 Introduction

We consider the problem of predictive density estimation for normal models under Kullback-Leibler loss (KL loss) when the parameter space is constrained to a convex set. More precisely, let X|μ ∼ N_{p}(μ, v_{x}I) and Y |μ ∼ N_{p}(μ, v_{y}I) be two independent random vectors having a normal distribution, with common unknown mean μ that we assume to be restricted to a convex set C ⊂ ℝ^{p}. The scale parameters v_{x} and v_{y} are assumed to be known and we denote by p(x|μ, v_{x}) and p(y|μ, v_{y}) the conditional densities of X and Y.

Under the above restriction, we seek to determine efficient predictive density estimators p̂(y|x) of the density p(y|μ, v_{y}), based on observing only X = x, relative to the Kullback-Leibler loss

L(μ, p̂(y|x)) = ∫_{ℝ^{p}} p(y|μ, v_{y}) log \frac{p(y|μ, v_{y})}{p̂(y|x)}dy  (1.1)

and the associated Kullback-Leibler(KL) risk

R_{KL}(μ, p̂) = ∫_{ℝ^{p}} p(x|μ, v_{x})L(μ, p̂(y|x))dx.  (1.2)

This model was considered by George, Liang and Xu [GFX06] when the mean μ is is unrestricted, that is, μ ∈ ℝ^{p}. The reference density is the generalized Bayes predictive density p̂_{U}(y|x) associated to the noninformative prior on ℝ^{p}, π(μ) ≡ 1. Its expression may be derived from a more general result due to Aitchison (see [Ait75]), as the conditional density of Y given X = x associated with prior measure π, and given by

p(y|x) = \frac{∫_{ℜ^{p}} p(y|μ, v_{y}) p(x|μ, v_{x}) π(μ) dμ}{∫_{ℜ^{p}} p(x|μ, v_{x}) π(μ) dμ} .  (1.3)

Komaki [Kom01] noticed that expression (1.3) with π(μ) ≡ 1 reduces to

p̂_{U}(y|x) = \frac{1}{[2π(v_{x} + v_{y})]^{p/2}} exp [−\frac{1}{2} \frac{∥y − x∥^{2}}{v_{x} + v_{y}} ] ,  (1.4)

and Murray [Mur77] showed that p̂_{U} is best invariant with constant risk, under translations and non singular linear transformations. For a location family, Ng [Ng80] extended this invariance property and Liang and Barron [LB04] proved that p̂_{U} is minimax.

In the normal case, George, Liang and Xu [GFX06] gave a simple direct proof of the minimaxity of p̂_{U} in (1.4) and showed, among other results, that p̂_{U} can be improved by Bayes predictive densities p̂_{π} under superharmonic priors π and for p ≥ 3, thus adding to previous findings due to Komaki [Kom01]. We refer to a recent review of Bayesian predictive estimation by George and Xu [GX10] for further exposition and description of recent research in this area.

We will make use of the following key representation for Bayesian estimators given by George, Liang and Xu. Hereafter, we let W = (v_{y} X + v_{x} Y)/(v_{x} + v_{y}), v_{w} = (v_{x} v_{y})/(v_{x} + v_{y}), and we let m_{π}(W; v_{w}) and m_{π}(X; v_{x}) be the marginal densities of W and X respectively under prior π.

Lemma 1.1 ([GFX06] Lemma 2)

We have

p̂_{π}(y|X) = \frac{m_{π}(W; v_{w})}{m_{π}(X; v_{x})} p̂_{U}(y|X) ,  (1.5)

where p̂_{U}(·|X) is the Bayes estimator associated with the uniform prior on ℝ^{p} given by (1.4).

It is also shown that, for any prior π, the difference between the KL risks of p̂_{U}(.|x) and the Bayesian predictive density p̂_{π}(.|x) is given by

R_{KL}(μ, p̂_{U}) − R_{KL}(μ, p̂_{π}) = E_{μ,v_{w}} [log m_{π}(W; v_{w})] − E_{μ,v_{x}} [log m_{π}(X; v_{x})]  (1.6)

where E_{μ,v} stands for the expectation with respect to the normal distribution N_{p}(μ, vI).

George, Liang and Xu underlined the fact that there exists a parallel between this predictive density estimation problem and the estimation of the mean vector μ under quadratic loss ∥μ̂ − μ∥^{2} giving rise to the quadratic risk

R_{Q}^{v}(μ, μ̂) = E_{μ,v} [∥μ̂ − μ∥^{2}] .  (1.7)

More precisely, they show that the predictive density p̂_{U} plays a similar role as the standard estimator X of μ which is best invariant and minimax under the quadratic risk (1.7), but inadmissible for p > 2.

Our findings here involve the elaboration and the use of similar connections between the risks R_{Q}^{v} and R_{KL} to draw inferences regarding domination and minimaxity for predictive density estimation problems when the mean μ is restricted to a convex set C. These findings parallel, and rely on several results applicable to estimating a bounded multivariate mean with risk R_{Q} ([Ber90], [MP01], [MP02], [Har04], [TK08], [FM10], among others). In Section 2, using a result of Brown, George and Xu [BGX08], our first result formalizes a link between the two risks R_{KL} and R_{Q}^{v}, the quantities involving R_{KL} being expressed as integrated quantities involving R_{Q}^{v}. This link is used to express the risk differences between Bayes and plug-in estimators and between Bayes estimators as well.

In Section 3, various applications are given for restricted parameter spaces C. First, referring to Hartigan [Har04] who showed that the Bayes estimator of μ with respect to the uniform prior π_{C} on a convex set with a non-empty interior C dominates X under the quadratic risk (1.7), we obtain, via two different paths, a similar result for the domination of p̂_{π_{C}} over p̂_{U}. We also show that our proof of dominance for KL-loss implies dominance for quadratic, thus providing an alternative proof of Hartigan’s result. Secondly, we turn our attention to plug-in densities. When an estimator δ_{1} for μ ∈ C is dominated by a Bayes estimator μ̂_{π,v}, associated to a prior π, and to a scale factor v, we give conditions under which the Bayes predictive density estimator p̂_{π} dominates the plug-in density estimator p̂_{1}(X) ∼ N_{p}(δ_{1}(X), v_{y} I_{p}). In the case when p = 1, we apply these results to obtain improvements on the plug-in maximum likelihood estimator p̂_{mle}(X) ∼ N_{p}(δ_{mle}(X), v_{y} I_{p}).

In Section 4, we deal with the minimaxity of Bayes predictive density estimators when μ is restricted to a ball or to a cone. As a specific result, we show that, when ∥μ∥ ≤ m, the boundary uniform Bayes estimator is minimax for risk R_{KL} whenever m ≤ c_{0}(p)\sqrt{v_{w}} where c_{0}(p) is the constant given by Berry [Ber90] and Marchand and Perron [MP02]. When μ belongs to a convex cone C with non-empty interior, we prove that the unrestricted predictive density estimator p̂_{U} in (1.4) remains minimax (as it is when no restriction is assumed) for risk R_{KL} when μ ∈ C. This finding parallels the result of Tsukuma and Kubokawa [TK08] who established that X is still minimax for estimating μ under the restriction to a polyhedral cone.

In Section 5, we expand on some additional considerations concerning plug-in estimators. Section 6 contains concluding remarks, and Section 7 is an appendix with details on some of the proofs.

2 Context and preliminary results

In this section, we expand upon a link between estimation under risks R_{Q}^{v} and risk R_{KL}. The following lemma and theorem concern plug-in estimators p̂_{1} ∼ N_{p}(δ_{1}(X), v_{y}I_{p}) and Bayesian estimators p̂_{π}(·|X). Theorem 2.1 provides a useful expression for a R_{KL} risk difference in terms of integrated R_{Q}^{v} risk differences.

Lemma 2.1 For a prior π, and a plug-in estimator p̂_{1} ∼ N_{p}(δ_{1}(X), v_{y}I_{p}), we have

(a) E^{X,Y} [log (\frac{m_{π}(W; v_{w})}{m_{π}(X; v_{x})})] = \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v}(μ, X) − R_{Q}^{v}(μ, μ̂_{π,v})) dv

and

(b) E^{X,Y} [ log (\frac{p̂_{U}(Y |X)}{p̂_{1}(Y |X)}) ] = \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v}(μ, δ_{1}) − R_{Q}^{v}(μ, X)) dv ,

where p̂_{U} is, as above, the generalized Bayes estimator with respect to the flat prior on ℝ^{p}.

Proof: Part (a) is taken from Brown, George, and Xu ([BGX08], Theorem 1). For part (b), since

∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} R_{Q}^{v_{x}}(μ, δ_{1}) dv = R_{Q}^{v_{x}}(μ, δ_{1})(1/v_{w} − 1/v_{x}) = R_{Q}^{v_{x}}(μ, δ_{1})/v_{y} ,

and

∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} R_{Q}^{v}(μ, X) dv = p ∫_{v_{w}}^{v_{x}} \frac{1}{v} dv = p log((v_{x} + v_{y})/v_{y}) ,

we need to show that

E^{X,Y} [log (\frac{p̂_{U}(Y |X)}{p̂_{1}(Y |X)})] = −\frac{p}{2} log(\frac{v_{x} + v_{y}}{v_{y}}) + \frac{R_{Q}^{v_{x}}(μ, δ_{1})}{2 v_{y}} .  (2.1)

A direct expansion yields

E^{X,Y} [log (\frac{p̂_{U}(Y |X)}{p̂_{1}(Y |X)})] = E^{X,Y} [−\frac{p}{2} log(\frac{v_{x} + v_{y}}{v_{y}}) + \frac{∥Y − δ_{1}(X)∥^{2}}{2v_{y}} − \frac{∥Y − X∥^{2}}{2(v_{x} + v_{y})}]
= −\frac{p}{2} log (\frac{v_{x} + v_{y}}{v_{y}} ) + \frac{1}{2v_{y}} [E^{X,Y} (∥Y − μ∥^{2})
+ E^{X,Y} (∥δ_{1}(X) − μ∥^{2})] − \frac{p}{2},

which indeed matches (2.1).

Theorem 2.1 For a plug-in estimator p̂_{1} ∼ N_{p}(δ_{1}(X), v_{y}I_{p}) and for Bayes estimators p̂_{π} and p̂_{π′}, we have

(a) R_{KL}(μ, p̂_{1}) − R_{KL}(μ, p̂_{π}) = \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v_{x}}(μ, δ_{1}) − R_{Q}^{v}(μ, μ̂_{π,v})) dv

and

(b) R_{KL}(μ, p̂_{π′}) − R_{KL}(μ, p̂_{π}) = \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v}(μ, μ̂_{π′,v}) − R_{Q}^{v}(μ, μ̂_{π,v})) dv.

Proof: Part (b) follows immediately from part (a). From the definition of the risk R_{KL}, we have for (a):

R_{KL}(μ, p̂_{1}) − R_{KL}(μ, p̂_{π}) = E^{X,Y} [log (\frac{p̂_{π}(Y |X)}{p̂_{1}(Y |X)})] .

Upon applying (1.5), we obtain equivalently

R_{KL}(μ, p̂_{1}) − R_{KL}(μ, p̂_{π}) = E^{X,Y} [log (\frac{m_{π}(W; v_{w})}{m_{π}(X; v_{x})})] + E^{X,Y} [log (\frac{p̂_{U}(Y |X)}{p̂_{1}(Y |X)})] ,

and the result now follows from Lemma 2.1.

Note that Corollary 1 of Brown et al. [BGX08] is an integrated version of (b).

We will make use of Theorem 2.1 to obtain dominance results applicable for risk R_{KL} by working inside the above integrals and relying on the associated comparisons for the risks R_{Q}^{v}, v ∈ (v_{w}, v_{x}). As an illustration, take δ_{1}(X) = X, π to be the flat prior on ℝ^{p}. We have R_{Q}^{v_{x}}(μ, δ_{1}) = pv_{x}, μ̂_{π,v}(X) = X, R_{Q}^{v}(μ, μ̂_{π,v}) = pv, and, from part (a) of Theorem 2.1:

R_{KL}(μ, p̂_{1}) − R_{KL}(μ, p̂_{U}) = \frac{p}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (v_{x} − v) dv
= \frac{p}{2} (\frac{v_{x}}{v_{w}} − log(\frac{v_{x}}{v_{w}}) − 1) ,

which is positive. The finding that p̂_{U} dominates p̂_{1} is not new of course (e.g., Aitchison [Ait75]), but the objective here was rather to illustrate Theorem 2.1 above.

Finally, we expand on some definitions and notations with respect to convex sets and cones in ℝ^{p}. A subset C of ℝ^{p} will be called a (positively homogeneous) cone if it is closed under positive scalar multiplication, i.e. α x ∈ C when x ∈ C and α > 0 (e.g., [Roc96]). In the above, C is a cone with vertex the origin. More generally, for such a C and for any g ∈ ℝ^{p}, the set C_{g} = C + g is an affine cone with vertex g, where we adopt hereafter the notation: for α ∈ ℝ , θ ∈ ℝ^{p} and A ⊂ ℝ^{p}, αA + θ = {αa + θ | a ∈ A} .

3 Applications for restricted parameter spaces

3.1 Improving on p̂_{U}

For the problem of estimating, under risk R_{Q}^{v_{x}}, with μ restricted to a convex subset C of ℝ^{p} with a non-empty interior, but otherwise arbitrary, Hartigan [Har04] showed quite generally that the Bayes estimator with respect to a uniform prior on C dominates the estimator X. We obtain an analogous result here for the predictive density estimation problem with risk R_{KL}. A first proof follows from Hartigan’s result and Theorem 2.1. A second proof with a much different flavor circumvents the explicit use of Hartigan’s result and Theorem 2.1 and follows, surprisingly, a more direct route to establishing the result. The second part of this subsection parallels a result by Kubokawa [Kub05] in providing a class of priors for which we obtain Bayesian improvements to p̂_{U} in the univariate case with μ bounded to an interval.

Theorem 3.1 Let C ⊂ ℝ^{p} be convex set a with a non-empty interior, and let π_{C} = 1l_{C}(μ) be the noninformative prior restricted to C. Then p̂_{π_{C}} dominates p̂_{U} in (1.4) for risk R_{KL} and μ ∈ C.

First proof: We apply part (b) of Theorem 2.1 with p̂_{π′} ≡ p̂_{U}, p̂_{π} ≡ p̂_{π_{C}}, μ̂_{π′,v}(X) = X for all v, and μ̂_{π,v}(X) the Bayes estimator of μ for R_{Q}^{v} under prior π_{C}. We infer from Hartigan [Har04] that R_{Q}^{v}(μ, μ̂_{π′,v}) − R_{Q}^{v}(μ, μ̂_{π,v}) ≥ 0, for all μ ∈ C, v > 0, with equality iff C is a cone and μ is its vertex. Making use of this and Theorem 2.1, the result follows immediately.

Second proof: Using the expression of the risk difference in (1.6), the risk difference between p̂_{U} and p̂_{π_{C}} equals

△R(μ) = R_{KL}(μ, p̂_{U}) − R_{KL}(μ, p̂_{π_{C}})
= E_{μ;v_{w}} log m_{π_{C}}(W; v_{w}) − E_{μ;v_{x}} log m_{π_{C}}(X; v_{x})
= ∫_{ℝ^{p}}p(w|μ, v_{w}) log [∫_{C}p(w|θ, v_{w})dθ] dw − ∫_{ℝ^{p}}p(x|μ, v_{x}) log [∫_{C}p(x|θ, v_{x})dθ] dx

by expressing the marginal m_{π_{C}}. Applying the changes of variables z = \frac{1}{\sqrt{v_{w}}} (w − μ) , s = \frac{1}{\sqrt{v_{w}}} (θ − w) , z = \frac{1}{\sqrt{v_{x}}} (x − μ) , and s = \frac{1}{\sqrt{v_{x}}} (θ − x), we obtain

△R_{(}μ) = ∫_{ℝ^{p}} p(z|0, 1) log [\frac{∫_{C_{1}} p(s|0, 1) ds}{∫_{C_{2}} p(s|0, 1) ds}]  (3.1)

where C_{1} = \frac{1}{\sqrt{v_{w}}} \{C − μ\} − z and C_{2} = \frac{1}{\sqrt{v_{x}}} \{C − μ\} − z . We need to prove that, for any μ ∈ C, △R(μ) ≥ 0. Let μ ∈ C. Clearly it suffices to show that, for any z ∈ ℝ^{p}, we have

C_{2} = \frac{1}{\sqrt{v_{x}}} \{C − μ\} − z ⊂ C_{1} = \frac{1}{\sqrt{v_{w}}} \{C − μ\} − z .  (3.2)

This is equivalent to showing that

\frac{1}{\sqrt{v_{x}}} \{C − μ\} ⊂ \frac{1}{\sqrt{v_{w}}} \{C − μ\} .  (3.3)

Let c_{1} ∈ \frac{1}{\sqrt{v_{x}}} \{C − μ\}. Then there exists c ∈ C such that c_{1} = \frac{1}{\sqrt{v_{x}}} (c − μ). Hence we can

write c_{1} = \frac{1}{\sqrt{v_{w}}} (c′ − μ), with c′ = \begin{matrix}√ & \overline{v_{w}} \\ √ & \overline{v_{x}} \end{matrix} c + (1 − \begin{matrix}√ & \overline{v_{w}} \\ √ & \overline{v_{x}} \end{matrix}) μ. As v_{w} < v_{x}, the last quantity is a convex combination of c and μ. Since c ∈ C, μ ∈ C and C is a convex then c′ ∈ C, which implies that (3.3) is satisfied.

Finally, since {C −μ} is a convex set containing the origin, (3.3) is satisfied with equality iff C is a cone and μ is its vertex which means that △R(μ) > 0, except when C is a cone and μ is its vertex. This completes the proof.

It is worth noting that this second proof of Theorem 3.1 and part (b) of Theorem 2.1 guarantee that the domination of p̂_{π_{C}} over p̂_{U} implies Hartigan’s result, that is, the domination of μ̂_{π_{C}} over X when C is convex with a non-empty interior. Indeed, for any v_{x} > 0 and v_{y} > 0 fixed, the domination of p̂_{π_{C}} over p̂_{U} indicates that

\frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v}(μ, X) − R_{Q}^{v}(μ, μ̂_{π_{c},v})) dv ≥ 0 ,  (3.4)

for all v_{x} > v_{w} > 0. Hence, for any 0 < a < b, the integral on (a, b) of the integrand term in (3.4) is non negative, which implies that this integrand term is non negative almost everywhere. Finally, by continuity of this function in v, it follows that R_{Q}^{v}(μ, X) − R_{Q}^{v}(μ, μ̂_{π_{c},v}) ≥ 0. Hence, our method provides an independent proof of Hartigan’s result.

We now turn to the particular case where p = 1 and C is a compact interval, say [−m, m] without loss of generality. Kubokawa [Kub05] obtained a class of priors that lead to improvements on the minimum risk equivariant estimator X for risk R_{Q}^{v}. Here, in analogous manner as above, we give a parallel result for our density estimation problem with risk R_{KL}.

Theorem 3.2 Let p = 1 and μ ∈ [−m, m]. Let π be a prior with a symmetric density supported on [−m, m], which is also increasing and logconcave on [0, m]. Then p̂_{π} dominates p̂_{U} for risk R_{KL} and μ ∈ [−m, m].

Proof: Since Kubokawa [Kub05] showed that μ̂_{π,v}(X) dominates X for μ ∈ [−m, m] for all v > 0 under the given assumptions on π, the result follows directly from Theorem 2.1 with p̂_{π′} ≡ p̂_{U}.

3.2 Improving on a maximum likelihood estimator

In this subsection, we give other direct implications of Theorem 2.1 and further applications when the mean μ is restricted to a ball centered at 0 of radius m. As in above Hartigan type result, we can borrow known or easy to derive results for the analogous estimation problem with risk R_{Q}^{v}. However, in contrast, we seek to dominate here the plug-in density based on the maximum likelihood estimator for an univariate bounded normal mean, rather than the generalized Bayes estimator p̂_{U}.

Corollary 3.1 Suppose that the Bayesian estimator μ̂_{π,v} dominates a given estimator δ_{1} for μ ∈ C, and for all v ∈ (v_{w}, v_{x}) under risk R_{Q}^{v}. Suppose further that either

(A) ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v_{x}}(μ, δ_{1}) − R_{Q}^{v}(μ, δ_{1}) ) dv ≥ 0 for all μ ∈ C

or

(B) R_{Q}^{v}(μ, δ_{1}) increases in v for v ∈ (v_{w}, v_{x}), for all μ ∈ C.

Then p̂_{π}(X) improves upon p̂_{1}(X) ∼ N_{p}(δ_{1}(X), v_{y}I_{p}) under risk R_{KL} for μ ∈ C.

Proof: The proof is straightforward. First, (B) implies (A). Secondly, by virtue of Theorem 2.1 and given our assumptions, we have

R_{KL}(μ, p̂_{1}) − R_{KL}(μ, p̂_{π}) = \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v}(μ, δ_{1}) − R_{Q}^{v}(μ, μ̂_{π,v}) dv
+ \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v_{x}}(μ, δ_{1}) − R_{Q}^{v}(μ, δ_{1}) dv > 0 .

Proceeding in analogous manner, we also have the following potentially useful result as well.

Corollary 3.2 Suppose that the Bayesian estimator μ̂_{π,v_{x}} dominates a given estimator δ_{1} for μ ∈ C under risk R_{Q}^{v_{x}}. Suppose further that either

(A’) ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v_{x}}(μ, μ̂_{π,v_{x}}) − R_{Q}^{v}(μ, μ̂_{π,v}) dv ≥ 0 for all μ ∈ C

or

(B’) R_{Q}^{v}(μ, μ̂_{π,v}) increases in v for v ∈ (v_{w}, v_{x}), for all μ ∈ C.

Then p̂_{π}(X) improves upon p̂_{1}(X) ∼ N_{p}(δ_{1}(X), v_{y}I_{p}) under risk R_{KL} for μ ∈ C.

We now turn to an application of Corollary 3.1 for an univariate normal mean which is bounded to an interval, but, in view of making use of its condition (B), we require the following preliminary result.

Lemma 3.1 For estimating a univariate normal mean μ bounded to an interval [a, b], the risk R_{Q}^{v}(μ, δ_{mle}) increases in v ∈ (0, ∞) for all μ ∈ [a, b], where δ_{mle}(X) is the maximum likelihood estimator of μ.

Proof: Let X ∼ N (μ, v) with a ≤ μ ≤ b, and let ϕ(t) = (1/\sqrt{2π}) e^{−t^{2}/2} be the standard normal density function. The maximum likelihood estimator of μ is given by

δ_{mle}(X) = a1l_{(−∞,a)}(X) + X1l_{[a,b]}(X) + b1l_{(b,∞)}(X) = \{ \begin{matrix}a & if & X ≤ a \\ X & if & a ≤ X ≤ b . \\ b & if & X ≥ b \end{matrix}

Let σ = \sqrt{v}. Then the risk of δ_{mle} at μ R_{Q}^{v}(μ, δ_{mle}) equals

R(μ, σ, δ_{mle}) = E_{μ} [(δ_{mle}(X) − μ)^{2}]
= (a − μ)^{2} ∫_{−∞}^{a} \frac{1}{σ} ϕ (\frac{x − μ}{σ}) dx + (b − μ)^{2} ∫_{b}^{+∞} \frac{1}{σ} ϕ (\frac{x − μ}{σ}) dx
+ ∫_{a}^{b}(x − μ)^{2} \frac{1}{σ} ϕ (\frac{x − μ}{σ}) dx
= (a − μ)^{2} ∫_{−∞}^{(a−μ)/σ} ϕ(y)dy + (b − μ)^{2} ∫_{(b−μ)/σ}^{+∞} ϕ(y)dy
+ σ^{2} ∫_{(a−μ)/σ}^{(b−μ)/σ} y^{2}ϕ(y)dy .

By differentiating under the integral sign with respect to σ we obtain

\frac{∂}{∂σ} R(μ, σ, δ_{mle}) = \frac{(μ − a)^{3}}{σ^{2}} ϕ (\frac{a − μ}{σ}) + \frac{(b − μ)^{3}}{σ^{2}} ϕ (\frac{b − μ}{σ}) + 2σ ∫_{(a−μ)/σ}^{(b−μ)/σ} y^{2}ϕ(y)dy
+ σ^{2} \frac{(b − μ)^{2}}{σ^{2}} ϕ (\frac{b − μ}{σ}) \frac{μ − b}{σ^{2}} − σ^{2} \frac{(a − μ)^{2}}{σ^{2}} ϕ (\frac{a − μ}{σ}) \frac{μ − a}{σ}
= 2σ ∫_{(a−μ)/σ}^{(b−μ)/σ} y^{2}ϕ(y)dy
> 0 ,

after simplification. Hence R(μ, σ, δ_{mle}) increases in σ and we have the desired result.

For risk R_{Q}^{v}, μ ∈ [a, b], Marchand and Perron [MP01], as well as Casella and Strawderman [CS81] (for δ_{BU} below), showed that :

• δ_{BU}(X) dominates δ_{mle}(X) for (i) (b − a)/2 ≤ \sqrt{v};

• All Bayes estimators δ_{π}(X) with π a symmetric measure about (a + b)/2 supported on [a, b] dominate δ_{mle}(X) for (ii) (b − a)/2 ≤ c_{1}\sqrt{v};

• and δ_{FU}(X) dominates δ_{mle}(X) for (iii) (b − a)/2 ≤ c_{2}\sqrt{v};

with c_{1} ≈ 0.4837, c_{2} ≈ 0.5230, and where δ_{BU}(X) is the Bayes estimator with respect to the two-point uniform prior on {−m, m}, δ_{FU}(X) is the Bayes estimator with respect to the uniform U(a, b) prior. For the density estimating problem, we denote p̂_{BU}(X), p̂_{FU}(X) as the corresponding Bayes estimators for the boundary uniform and fully uniform priors respectively. In view of the above dominance results for risk R_{Q}^{v}, part (B) of Corollary 3.1, and Lemma 3.1, we now derive improvements on the plug-in maximum likelihood estimator p̂_{mle}(X) ∼ N_{p}(δ_{mle}(X), v_{y}I_{p}).

Corollary 3.3 For estimating p_{Y} (·|μ; ν_{y}) under risk R_{KL}, with p = 1, μ ∈ [a, b],

(a) p̂_{BU}(X) dominates p̂_{mle}(X) for (i) (b − a)/2 ≤ \sqrt{v_{w}};

(b) all Bayes estimators p̂_{π}(X), with π a symmetric measure about (a + b)/2 supported on [a, b], dominate p̂_{mle}(X) for (ii) (b − a)/2 ≤ c_{1}\sqrt{v_{w}} ;

(c) p̂_{FU}(X) dominates p̂_{mle}(X) for (iii) (b − a)/2 ≤ c_{2}\sqrt{v_{w}}.

The Bayesian estimators p̂_{π}(X), p̂_{BU}(X) and p̂_{FU}(X) may be evaluated directly by computing the predictive density p̂(·|x) as in (1.3), or via (1.5). For instance, in the case of the two-point uniform prior on {−m, m}, we obtain the predictive density p̂_{BU}(X) as being the density of a mixture of the two normal distributions N(−m, v_{y}) and N(m, v_{y}) with respective weights w(x) = (1 + e^{2 m x/v_{x}})^{−1} and 1 − w(x).

3.3 Bayesian improvements on the plug-in N_{p}(x, v_{y}I_{p}) estimator under a constraint

Here is an instructive example in relationship with Corollaries 3.1 and 3.2. Consider normal priors π_{τ} ∼ N_{p}(0, τI) and the corresponding Bayes estimators μ̂_{π_{τ} ,v}(x) = τ/(τ + v) x under squared error loss. Now, for cases where μ is constrained to the ball B_{m} = {μ ∈ ℝ^{p} : ∥μ∥ ≤ m}, one may verify that aX dominates X under risk R_{Q}^{v} iff a ∈ [(m^{2} − pv)/(m^{2} + pv), 1), which tells us that μ̂_{π_{τ} ,v}(X) lowers the risk of δ_{1}(X) on B_{m} iff τ ≥ (m^{2}/2p)−v/2. Since, R_{Q}^{v}(μ, X) = pv is increasing in v, part (B) of Corollary 3.1 tells us that p̂_{π_{τ}} (X) lowers the risk R_{KL} of the plug-in N_{p}(X, v_{y}I_{p}) whenever τ ≥ (m^{2}/2p)−v/2 for all v ∈ (v_{w}, v_{x}), that is

τ ≥ \frac{m^{2}}{2p} − \frac{v_{w}}{2} .  (3.5)

So sufficiently large prior variances lead here to improvements, with the dominance for τ → ∞ interpretable as the dominance of p̂_{U} on B_{m} for all m > 0, which is of course already known.

Alternatively, in view of applying Corollary 3.2, we begin with the weaker condition τ ≥ (m^{2}/2p) − v_{x}/2 for μ̂_{π_{τ} ,v_{x}}(X) to dominate δ_{1}(X) = X under risk R_{Q}^{v_{x}}. However, turning to condition (B’), we observe that the quadratic risk of μ̂_{π_{τ} ,v}(X), given by R_{Q}^{v}(μ, μ̂_{π,v}) = \frac{τ^{2}}{(τ+v)^{2}} pv + \frac{v^{2}}{(τ+v)^{2}} ∥μ∥^{2}, is only increasing in v ∈ (v_{w}, v_{x}) for all μ ∈ B_{m} (or μ ∈ ℝ^{p}) iff R(0, μ̂_{π,v}) is increasing in v for all v ∈ (v_{w}, v_{x}), or equivalently τ ≥ v_{x}. We hence obtain τ ≥ max(v_{x}, (m^{2}/2p) − v_{x}/2) as an alternative sufficient condition to (3.5), and the improved sufficient condition

τ ≥ min(\frac{m^{2}}{2p} − \frac{v_{w}}{2} , max(v_{x}, \frac{m^{2}}{2p} − \frac{v_{x}}{2} )) ,

for p̂_{π_{τ}} (X) to dominate the plug-in N_{p}(X, v_{y}I_{p}) under risk R_{KL} with μ ∈ B_{m}. Finally, we point out that yet a stronger result can be arrived at by working with condition (A’) of Corollary 3.2.

4 Minimax results

In this section, we derive minimaxity results for cases where μ is restricted. First, we fully exploit Section 2’s relationships between the risks R_{KL} and R_{Q}, as well as known minimaxity results for cases where the mean is restricted to a ball. Secondly, we establish that the estimator p̂_{U} remains minimax when the mean μ is restricted to a cone yielding a general result analogous to Tsukuma and Kubokawa’s result [TK08] under squared error loss.

4.1 Minimaxity when the mean is restricted to a ball

Here is a general framework which we will seek to apply in cases where the mean μ is restricted to a ball of radius m centered around the origin.

Theorem 4.1 Consider our general problems of estimating μ with risk R_{Q}^{v} and the density p_{Y} (·|μ, v_{y}) with risk R_{KL}, where μ ∈ C ⊆ ℝ^{p}. Suppose there exists a prior measure π^{∗} and a subset C_{0} of C such that π^{∗}(C_{0}) = 1; R_{Q}^{v}(μ, μ̂_{π^{∗},v}) = R̄_{v} for μ ∈ C_{0}; and sup_{μ∈C}{R_{Q}^{v}(μ, μ̂_{π^{∗},v})} = R̄_{v} for all v ∈ (v_{w}, v_{x}). Then p̂_{π^{∗}} is minimax for risk R_{KL} among all Bayesian estimators that have constant risk R_{Q}^{v} on C_{0}.

Remark 4.1 Note that these assumptions imply that, for all v ∈ (v_{w}, v_{x}), μ̂_{π^{∗},v} is minimax under R_{Q}^{v} and π^{∗} is least favorable.

Proof of Theorem 4.1: For any π′ such that R_{Q}^{v}(·, μ̂_{π′,v}) is constant on C_{0}, there exists μ_{0} ∈ C_{0} (independent of v) such that R_{Q}^{v}(μ_{0}, μ̂_{π′,v}) ≥ R̄_{v} for all v ∈ (v_{w}, v_{x}), since R̄_{v} is the Bayes risk associated with π^{∗}. Now, making use of Theorem 2.1 with p̂_{π′} ≡ p̂_{U} and the constant R_{KL} risk of p̂_{U}, we obtain indeed for any Bayes estimator p̂_{π′},

sup_{μ∈C} R_{KL}(μ, p̂_{π′}) ≥ R_{KL}(μ_{0}, p̂_{π′})
= R_{KL}(μ_{0}, p̂_{U}) + \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v}(μ_{0}, μ̂_{π′,v}) − pv) dv
≥ R_{KL}(μ, p̂_{U}) + \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} ( sup_{μ∈C}\{R_{Q}^{v}(μ, μ̂_{π^{∗},v})\} − pv) dv
≥ sup_{μ∈C} \{R_{KL}(μ, p̂_{U}) + \frac{1}{2} ∫_{v_{w}}^{v_{x}} \frac{1}{v^{2}} (R_{Q}^{v}(μ, μ̂_{π^{∗},v}) − pv) dv\}
= sup_{μ∈C} R_{KL}(μ, p̂_{π^{∗}}).

As an example, consider a constraint to a ball where μ ∈ C = {μ ∈ ℝ^{p} : ∥μ∥ ≤ m}. For this problem, Bayes estimators form a complete class and a minimax estimator can be found among orthogonally invariant estimators. Since such estimators have constant risk on spheres where ∥μ∥ = λ, a minimax estimator can be found among orthogonally invariant Bayes estimators, or equivalently among Bayes estimators with spherically symmetric priors ([Kie57]). Now take C_{0} = {μ ∈ C : ∥μ∥ = m} and π^{∗} to be the uniform prior measure on C_{0}. As studied in Berry [Ber90], Marchand and Perron [MP02], or Casella and Strawderman [CS81] (p = 1), the corresponding Bayes estimator is given by μ̂_{π^{∗},v}(x) = \frac{I_{p/2}(m∥x∥/v)}{I_{p/2−1}(m∥x∥/v)} \frac{m x}{∥x∥}, where I_{k}(·) is the modified Bessel function of order k, and μ̂_{π^{∗},v}(X) is unique minimax for risk R_{Q}^{v} with sup_{μ∈C}{R_{Q}^{v}(μ, μ̂_{π^{∗},v})} = sup_{μ∈C_{0}}{R_{Q}^{v}(μ, μ̂_{π^{∗},v})} = R̄_{v} iff m ≤ c_{0}(p)\sqrt{v}. Hence, to satisfy the assumptions of Theorem 4.1, we require simply m ≤ c_{0}(p)\sqrt{v} for all v ∈ (v_{w}, v_{x}), which yields the following.

Corollary 4.1 For estimating the density p(·|μ, v_{y}) with the constraint ∥μ∥ ≤ m, the boundary uniform Bayes estimator p̂_{π^{∗}} is minimax for risk R_{KL} whenever m ≤ c_{0}(p)\sqrt{v_{w}}, where c_{0}(p) is the constant defined and evaluated by Berry [Ber90], and Marchand and Perron [MP02].

We conclude this subsection by pointing out that, as at the end of Section 3.2, p̂_{π^{∗}} may be evaluated directly as in (1.3), or by using (1.5), and that Marchand and Perron [MP02] provide various properties of c_{0}(p), including the lower bound: \sqrt{p} ≤ c_{0}(p).

4.2 Minimaxity of p̂_{U} when the mean is restricted to a cone

In this section, we deal with the minimaxity of p̂_{U} in (1.4) when the mean μ is restricted to a cone. We point out that another potential and related approach to the problem is given by the recent work of Marchand and Strawderman [MS10].

Theorem 4.2 Let C be a convex cone, with non-empty interior. Then the unrestricted predictive density estimator p̂_{U} in (1.4) is minimax under the KL loss (1.1), when μ is restricted to C.

Proof: Let r be the constant risk of p̂_{U}. We will show that r is a limit of Bayes risks of a sequence of Bayes predictive densities p̂_{π_{k}}(y|x) with respect to a sequence of proper priors π_{k} lying in C. Let C_{k} = C ∩ B_{k} = {c ∈ C | ∥c∥ ≤ k} where B_{k} denotes the ball of radius k centered at 0. By the positive homogeneity of C, we can express C_{k} as

kC_{1} = {kc | c ∈ C_{1}}.  (4.1)

Consider, as a sequence of proper priors, the uniform distributions on the convex sets C_{k}, that is,

π_{k}(μ) = \frac{1}{λ(C_{k})}1l_{C_{k}}(μ) = \frac{1}{k^{p}} \frac{1}{λ(C_{1})}1l_{C_{k}}(μ)  (4.2)

where λ is the Lebesgue measure on ℝ^{p}.

Using (4.2) and the expression of the risk difference in (1.6), the difference of Bayes risks between p̂_{U} and p̂_{π_{k}} is

r − r(π_{k}, p̂_{π_{k}}) = ∫_{ℝ^{p}} [R_{KL}(p̂_{U}, μ) − R_{KL}(p̂_{π_{k}}, μ)] π_{k}(μ) dμ
= \frac{1}{λ(C_{k})} ∫_{C_{k}} [E_{μ,v_{w}} [log m_{π_{k}}(W; v_{w})] −E_{μ;v_{x}}[log m_{π_{k}}(X, v_{x})]]dμ . (4.3)

Now, using the second expression of π_{k}(μ) in (4.2) and expressing the marginal m_{π_{k}} in the bracketed term in (4.3), we have

r − r(π_{k}, p̂_{π_{k}}) = \frac{1}{λ(C_{1})} [L(v_{w}, k) − L(v_{x}, k)]  (4.4)

where

L(v, k) = \frac{1}{k^{p}} ∫_{C_{k}} E_{μ,v} [log ∫_{C_{k}} p(Z|a, v)da] dμ
= \frac{1}{k^{p}} ∫_{C_{k}} [∫_{ℝ^{p}} p(z|μ, v) log [∫_{C_{k}} p(z|a, v)da] dz] dμ .  (4.5)

Then with the changes of variables s = \frac{1}{\sqrt{v}}(a − z), z′ = \frac{1}{\sqrt{v}}(z − μ), and r = \frac{1}{k} μ, L(v, k) becomes

L(v, k) = \frac{1}{k^{p}} ∫_{C_{k}} [∫_{ℝ^{p}} p(z|μ, v) log [∫_{\frac{1}{\sqrt{v}} C_{k}− \frac{1}{\sqrt{v}} z} p(s|0, 1)ds] dz] dμ

= \frac{1}{k^{p}} ∫_{C_{k}} [∫_{ℝ^{p}} p(z′|0, 1) log [∫_{\frac{1}{\sqrt{v}} C_{k}− \frac{1}{\sqrt{v}} μ−z′} p(s|0, 1)ds] dz′] dμ (4.6)

= ∫_{C_{1}} [∫_{ℝ^{p}} p(z|0, 1) log ∫_{\frac{k}{\sqrt{v}} C_{1}− \frac{k}{\sqrt{v}} r−z} p(s|0, 1)ds dz] dr .  (4.7)

It remains to show that the limit of L(v, k) = L^{∗}(k/\sqrt{v}) when k goes to infinity exists and hence does not depend on v, which will imply that lim_{k→∞}(r − r(π_{k}, p̂_{π_{k}})) = 0. Let A_{k,r} =

\frac{k}{\sqrt{v}}\{C_{1} − r\}. As the interior of C is non-empty, for any interior point r of C_{1}, there exists a ball B_{ϵ} in ℝ^{p} of radius ϵ > 0, centered at 0, such that

B_{ϵ} ⊂ ℂ_{1} − r.

Then it follows that

ℝ^{p} ⊇ lim inf_{k→∞} A_{k,r} ⊇ lim inf_{k→∞} B_{\frac{k}{\sqrt{v}} ϵ} = ℝ^{p}  (4.8)

and hence

lim_{k→∞} ∫_{\frac{k}{\sqrt{v}} C_{1}− \frac{k}{\sqrt{v}} r−z} p(s|0, 1)ds = 1 .  (4.9)

Now the convexity of C, and hence of C_{1}, allows, according to Lemma 7.1, to apply the Lebesgue dominated convergence theorem to the expression in (4.7). Thus we obtain lim_{k→∞} L(k, v) = 0 for any fixed v. Therefore lim_{k→∞}(r − r(π_{k}, p̂_{π_{k}})) = 0, which establishes the

minimaxity of p̂_{U}. ^{1}

As an immediate consequence of Theorem 3.1 and Theorem 4.2, we have the following corollary.

Corollary 4.2 Let C ⊂ ℝ^{p} be convex cone with a non-empty interior, and let π_{C} = 1l_{C}(μ) be the flat prior restricted to C. Then p̂_{π_{C}} is minimax for risk R_{KL} and μ ∈ C.

Note that Theorem 4.2 holds also for affine convex cones with non-empty interiors.

Indeed, it suffices to consider the sequence of proper priors π_{k}(μ) = \frac{1}{λ(C_{k})} 1l_{C_{k}}(μ − g).

More generally, we can generalize Theorem 4.2 to cones which are not necessarily convex. The proof is analogous to the one of Theorem 4.2 and is relegated to the appendix.

Theorem 4.3 Suppose C is a finite disjoint union of (affine) convex cones, in the sense that the restriction is given by

C = ∪_{i=1}^{n} (C^{i} + g_{i})  (4.10)

where C^{1}, …, C^{n} are convex cones with non-empty interiors, and g_{1}, …, g_{n} are n fixed points in ℝ^{p} where, for any i ≠ j,

λ ((C^{i} + g_{i}) ∩ (C^{j} + g_{j})) = 0.  (4.11)

Then p̂_{U} is still minimax when μ is restricted to C.

1Note that we implicitly use the fact that the boundary of C_{1} is negligible with respect to the Lebesgue measure as a boundary of a convex subset in ℝ^{p} (cf. [Lia86]).

5 Plug-in estimators: some additional considerations

We now expand on a general phenomenon concerning plug-in estimators and how they can be improved upon within the class of normal density estimators. Below, we set G_{m}(c) = (1 − \frac{1}{c})m − log c for c > 1, m > 1, and we let c_{0}(m) denote the root of G_{m}(c) that lies in (m, ∞). It is easy to show that G_{m}(c) is positive for c ∈ (1, m], is maximized on (1, ∞) at c = m, and does indeed have a single root on (m, ∞). As above, we denote R_{Q}^{v_{x}}(μ, δ) as the risk E_{μ,v_{x}}[∥δ(X) − μ∥^{2}].

Theorem 5.1 Suppose δ(X) is an estimator of μ, μ ∈ C, such that inf_{μ∈C} R_{Q}^{v_{x}}(μ, δ) > 0. Consider the estimator p̂_{c} ∼ N_{p}(δ(X), cv_{y}I_{p}) and the plug-in version p̂_{1} ∼ N_{p}(δ(X), v_{y}I_{p}). We then have the following.

(a) For any given μ, the optimal value of c is

m(μ) = 1 + \frac{R_{Q}^{v_{x}}(μ, δ)}{pv_{y}},

and p̂_{c} improves upon p̂_{1} iff c ∈ (1, c_{0}(m(μ));

(b) For μ ∈ C, p̂_{c} dominates p̂_{1} iff c ∈ (1, c_{0}(inf_{μ∈C} m(μ)) or c = c_{0}(inf_{μ∈C} m(μ)) with m(·) not constant on C.

Proof: We have

R_{KL}(μ, p̂_{1}) − R_{KL}(μ, p̂_{c}) = E^{X,Y} [log(\frac{p̂_{c}(Y; X)}{p̂_{1}(Y; X)})]

= E^{X,Y} [log (\frac{(2πcv_{y})^{−p/2}e^{−∥Y −δ(X)∥^{2}/2cv_{y}}}{(2πv_{y})^{−p/2}e^{−∥Y −δ(X)∥^{2}/2v_{y}}} )]

= −\frac{p}{2} log c + E^{X,Y} [\frac{(1 − 1/c)}{2v_{y}}∥Y − δ(X)∥^{2}]
= \frac{p}{2}(− log c + (1 − 1/c) m(μ)),

since E^{X,Y} [∥Y − δ(X)∥^{2}] = E^{X,Y} [∥Y − μ∥^{2}] + E^{X,Y} [∥μ − δ(X)∥^{2}] = pv_{y} + R_{Q}^{v_{x}}(μ, δ). The result then follows from the above expression, the properties of G_{m}(·) and c_{0}, and since m(μ) ≥ inf_{μ∈C} m(μ) > 1 for all μ ∈ C.

Remark 5.2 In the setup of part (b) of Theorem 5.1, it may be easily derived from a similar analysis to the one above that, among estimators p̂_{c}, those with c ∈ [inf_{μ∈C} m(μ), sup_{μ∈C} m(μ)] form a complete subclass, while the estimators p̂_{c} with c ∈ [inf_{μ∈C} m(μ), c_{0}(inf_{μ∈C} m(μ))] form a complete subclass among those that dominate p̂_{1}.

Example 5.1 Take δ(X) = aX with 0 < a ≤ 1, C = ℝ^{p}. Since E_{μ,v_{x}}[∥aX − μ∥^{2}] = a^{2}pv_{x} +(1−a)^{2}μ′μ, we have m(μ) = 1+ \frac{a^{2}pv_{x}+(1−a)^{2}μ′μ}{pv_{y}} and inf_{μ∈ℝ^{p}} m(μ) = 1+a^{2} \frac{v_{x}}{v_{y}}. Part (b) of Theorem 5.1 tells us that p̂_{1} is dominated by p̂_{c} under R_{KL} iff c ∈ (1, c_{0}(1 + a^{2} \frac{v_{x}}{v_{y}})] (with c_{0}(1 + a^{2} \frac{v_{x}}{v_{y}}) > 1 + a^{2} \frac{v_{x}}{v_{y}}). For the specific case a = 1, the optimal value of c reduces to 1 + \frac{v_{x}}{v_{y}} regardless of μ, yielding the optimal choice p̂_{1+ \frac{vx}{vy}} ∼ N_{p}(X, (v_{x} + v_{y})I_{p}) and recovering its well known optimality property among the estimators p̂_{c} ∼ N_{p}(X, cv_{y}I_{p}).

Example 5.2 Take the James-Stein estimator δ(X) = (1 − \frac{(p−2) v_{x}}{X′X})X for p ≥ 3 and C = ℝ^{p}. δ(X) is minimax with inf_{μ∈ℝ^{p}} R_{Q}^{v_{x}}(μ, δ) = (p−2)v_{x} and sup_{μ∈ℝ^{p}} R_{Q}^{v_{x}}(μ, δ) = p v_{x}. Both Theorem 5.1 and Remark 5.2 then be applied with inf_{μ∈Re^{p}} m(μ) = 1 + \frac{(p−2)v_{x}}{pv_{y}} and sup_{μ∈Re^{p}} m(μ) = 1 + \frac{v_{x}}{v_{y}}. Notice that here and for any case where μ ∈ ℝ^{p}, the supremum value of m(μ) is bounded below by 1 + \frac{v_{x}}{v_{y}}, since the supremum risk of any estimator δ(X) surpasses the minimax risk which is pv_{x} and, hence, the estimator p̂_{1+ \frac{vx}{vy}} ∼ N_{p}(δ(X), (v_{x} + v_{y})I_{p}) is admissible among the estimators p̂_{c} regardless of δ(X). This provides an interesting extension of its outright optimality among the p̂_{c}’s when δ(X) = X.

Theorem 5.1 and Remark 5.2 encompass the general phenomenon that plug-in estimators N_{p}(δ(X), v_{y}I_{p}) can generally be improved upon by inflating the variance by a certain explicit factor, regardless of the efficiency (or inefficiency) of δ(X) for estimating μ. Moreover, the prescribed range of inflation is directly proportional to the risk R_{Q}^{v_{x}}(μ, δ), for all δ, and p. Alternatively, estimators p̂_{c} for which δ(X) is inadmissible for estimating μ (e.g., Example 5.2) can be improved upon as follows.

Theorem 5.2 Let c > 0, μ ∈ C, and δ_{1}(X) and δ_{2}(X) be two estimators of μ based on X. Then, the plug-in estimator p̂_{2} ∼ N_{p}(δ_{2}(X), cv_{y}I_{p}) dominates the plug-in estimator p̂_{1} ∼ N_{p}(δ_{1}(X), cv_{y}I_{p}) under R_{KL} for μ ∈ C iff δ_{2}(X) dominates δ_{1}(X) as an estimator of μ, for μ ∈ C.

Proof: The result follows directly with the decomposition

R_{KL}(μ, p̂_{2}) − R_{KL}(μ, p̂_{1}) = E^{X,Y} [log (\frac{p̂_{1}(Y; X)}{p̂_{2}(Y; X)})]
= E^{X,Y} [\frac{∥δ_{2}(X) − μ∥^{2} − ∥δ_{1}(X) − μ∥^{2}}{2cv_{y}} ] .

6 Conclusion

In [GFX06], George et al. put forth an interesting parallel between minimax conditions for estimating a multivariate normal mean vector under quadratic risks R_{Q}^{v} and minimax conditions for a predictive density estimator under Kullback-Leibler risk R_{KL}. Similar connections with regards to admissibility were also developed by Brown et al. ([BGX08]). In this paper, we obtain similar connections for dominance and minimaxity results when the mean is restricted to a convex set or to a cone. For instance, in the case where the mean is restricted to a convex set C, we prove domination of the Bayes predictive density estimator p̂_{π_{C}} with respect to the uniform prior π_{C} over the generalized Bayes predictive density p̂_{U}(y|x) associated with the flat prior on ℝ^{p}.

An essential use is made of an explicit link between a collection of risks R_{Q}^{v} and the Kullback-Leibler risk R_{KL}. It allows us to create settings where a Bayesian predictive density estimator p̂_{π} dominates a plug-in density estimator p̂_{1}(X) ∼ N_{p}(δ_{1}(X), v_{y} I_{p}). Examples including improvements on the plug-in maximum likelihood estimator p̂_{mle}(X) ∼ N_{p}(δ_{mle}(X), v_{y} I_{p}) are derived when the dimension p is 1.

Minimaxity results are also obtained when the mean is restricted to a ball as we derive conditions for which the boundary uniform Bayes estimator is minimax for risk R_{KL}. Also, when the restricted parameter space is a convex cone C with non-empty interior, we prove that the unrestricted predictive density estimator p̂_{U} remains minimax (as it is when no restriction is assumed).

7 Appendix

We expand first here on a technical lemma useful in the proof of Theorem 4.2, and we conclude with a proof of Theorem 4.3. For convenience, we denote by ϕ(s) the normal density p(s|0, 1).

Let C_{1} ⊂ B_{1} be a convex set in ℝ^{p} with non-empty interior. For t > 0, write

L^{∗}(t) = ∫_{C_{1}} ∫_{ℝ^{p}} ϕ(z) log [∫_{tC_{1}−tr−z}ϕ(s)ds] dz dr
= ∫_{C_{1}×ℝ^{p}} ϕ(z) g_{t}(r, z) d(r, z)

with

g_{t}(r, z) = log [∫_{tC_{1}−tr−z} ϕ(s)ds] .  (7.1)

We showed in the proof of Theorem 4.2 that, for any z ∈ ℝ^{p} and for any r ∈ C_{1}^{∘}, the interior of C_{1}, we have lim_{t→∞} g_{t}(r, z) = 0 , implying lim_{t→∞} L^{∗}(t) = 0 as long as dominated convergence can be applied. We now justify this.

Lemma 7.1 The function |g_{t}(·, ·)| given (7.1) is bounded by an integrable function. proof: Let 0 < t_{1} < t_{2}. Then, similarly to (3.3) in the proof of Theorem 3.1, we can see that t_{1}(C_{1} − r) ⊂ t_{2}(C_{1} − r). Hence g_{t}(r, z) is non decreasing in t > 0, for any z ∈ ℝ^{p} and any r ∈ C_{1}.

We have |g_{t}(r, z)| ≤ |g_{1}(r, z)| for all t ≥ 1 since g_{t}(r, z) ≤ 0. By applying the Jensen inequality to the logarithmic function, we obtain

|g_{1}(r, z)| = − log \{λ (C_{1} − r − z) ∫_{C_{1}−r−z} \frac{1}{λ (C_{1} − r − z)} ϕ(s) ds\}
≤ − log λ (C_{1}) − ∫_{C_{1}−r−z}\frac{1}{λ (C_{1})} log ϕ(s)ds
= − log λ (C_{1}) + \frac{1}{λ (C_{1})} ∫_{C_{1}−r−z} [\frac{∥s∥^{2}}{2} + \frac{p}{2} log(2π)] ds .

Using the inequality

∥a + b + c∥^{2} ≤ 3(∥a∥^{2} + ∥b∥^{2} + ∥c∥^{2})

for any a, b, c in ℝ^{p}, it follows that, for any s ∈ C_{1} − r − z, that is s = c_{1} − r − z with c_{1} ∈ C_{1},

∥s∥^{2} ≤ 3(∥c_{1}∥^{2} + ∥r∥^{2} + ∥z∥^{2}) ≤ 3(2 + ∥z∥^{2})

and hence

|g_{1}(r, z)| ≤ \frac{p}{2} log(2π) − log λ (C_{1}) + \frac{1}{λ (C_{1})} ∫_{C_{1}−r−z} (3 + 3\frac{∥z∥^{2}}{2} ) ds
= \frac{p}{2} log(2π) − log λ (C_{1}) + 3 + \frac{3}{2}∥z∥^{2} .

Therefore

∫_{C_{1}×ℝ^{p}} ϕ(z) |g_{1}(r, z)| d(r, z) ≤ ∫_{C_{1}×ℝ^{p}} ϕ(z) [\frac{p}{2} log 2π − log λ (C_{1}) + 3 + \frac{3}{2}∥z∥^{2}] d(r, z)
= λ(C_{1}) [\frac{p}{2} log 2π − log λ (C_{1}) + 3 + \frac{3p}{2}]
< ∞ .

proof of Theorem 4.3: Define for k ≥ 1, and i = 1, …, n, C_{k}^{i} = k C_{1}^{i} where

C_{1}^{i} = {c ∈ C^{i} | ∥c∥ ≤ 1}.

Then define

C_{k} = ∪_{i=1}^{n} (C_{k}^{i} + g_{i}) = ∪_{i=1}^{n} (kC_{1}^{i} + g_{i}) .

Note that λ(C_{k}) = k^{p}λ(C_{1}) and, for any k_{1} ≤ k_{2}, we have C_{k_{1}} ⊂ C_{k_{2}}. Using the sequence

of proper priors π_{k}(μ) = \frac{1}{k^{p}} \frac{1}{λ(C_{1})}1l_{ℂ_{k}}(μ), we have, as in the proof of Theorem 4.2,

r − r(π_{k}, p̂_{π_{k}}) = \frac{1}{λ(C_{1})} ∑_{i=1}^{n} L^{i}(v_{w}, k) − L^{i}(v_{x}, k)

with, for i = 1, … n,

L^{i}(v, k) = \frac{1}{k^{p}} ∫_{C_{k}^{i}+g_{i}} ∫_{ℝ^{p}} ϕ(z) log [∫_{\frac{1}{\sqrt{v}} C_{k}− \frac{1}{\sqrt{v}} μ−z} ϕ(s)ds] dz dμ

= \frac{1}{k^{p}} ∫_{C_{k}^{i}+g_{i}} ∫_{ℝ^{p}} ϕ(z) log [∑_{j=1}^{n} ∫_{\frac{1}{\sqrt{v}} C_{k}^{j}+ \frac{1}{\sqrt{v}} g_{j}− \frac{1}{\sqrt{v}} μ−z} ϕ(s)ds] dz dμ , thanks to (4.6)
≥ L̄^{i}(v, k) ,

where

L̄^{i}(v, k) = \frac{1}{k^{p}} ∫_{C_{k}^{i}+g_{i}}∫_{ℝ^{p}} ϕ(z) log [∫_{\frac{1}{\sqrt{v}} C_{k}^{i}+ \frac{1}{\sqrt{v}} g_{i}− \frac{1}{\sqrt{v}} μ−z} ϕ(s)ds] dz dμ

= \frac{1}{k^{p}} ∫_{C_{k}^{i}} ∫_{ℝ^{p}} ϕ(z) log [∫_{\frac{1}{\sqrt{v}} C_{k}^{i}− \frac{1}{\sqrt{v}} μ′−z} ϕ(s)ds] dz dμ′ ,

by the change of variables μ = μ′ + g_{i}.

Note that the expression of L̄^{i}(v, k) corresponds to the expression of L(v, k) in (4.6) which has been shown to satisfy lim_{k→∞} L(v, k) = 0. Hence lim_{k→∞} L̄^{i}(v, k) = 0 and from (7.2)

we get lim_{k→∞} L^{i}(v, k) ≥ 0. Since L^{i}(v, k) ≤ 0, it follows that lim_{k→∞} L^{i}(v, k) = 0. Therefore lim_{k→∞}(r − r(π_{k}, p̂_{π_{k}})) = 0, which completes the proof.

References

[Ait75] J. Aitchison. Goodness of prediction fit. Biometrika, 62:547–554, 1975.

[Ber90] C. Berry. Minimax estimation of a bounded normal mean vector. Journal of Multivariate Analysis, 35:130–139, 1990.

[BGX08] L. D. Brown, I. George, and X. Xu. Admissible predictive density estimation. Annals of Statistics, 36:1156–1170, 2008.

[CS81] G. Casella and W. E. Strawderman. Estimating a bounded normal mean. Annals of Statistics, 9:870–878, 1981.

[FM10] D. Fourdrinier and E. Marchand. On bayes estimators with uniform priors on spheres and their comparative performance with maximum likelihood estimators for estimating bounded multivariate normal means. Journal of Multivariate Analysis, 101:1390–1399, 2010.

[GFX06] E. I. George, L. Feng, and X. Xu. Improved minimax predictive densities under Kullback-Leibler loss. Annals of Statistics, 34:78–91, 2006.

[GX10] E.I. George and X. Xu. Bayesian predictive density estimation. In Frontiers of Statistical Decision Making and Bayesian Analysis. In honor of James O. Berger, pages 83–95. Springer, 2010.

[Har04] J. A. Hartigan. Uniform priors on convex sets improve risk. Statistics and Probability Letters, 67:285–288, 2004.

[Kie57] J. Kiefer. Invariance, minimax sequential estimation, and continuous time processes. Annals of Mathematical Statistics, 28:573–601, 1957.

[Kom01] F. Komaki. A shrinkage predictive distribution for multivariate normal observables. Biometrika, 88:859–864, 2001.

[Kub05] T. Kubokawa. Estimation of bounded location and scale parameters. Journal of the Japanese Statistical Society, 35:221–249, 2005.

[LB04] F. Liang and A. Barron. Exact minimax strategies for predictive density estimation, data compression and model selection. IEEE Transactions on Information Theory, 50:2708–2726, 2004.

[Lia86] R. Liang. A note on the measurability of convex sets. Arch. Math, 47:90–92, 1986.

[MP01] É. Marchand and F. Perron. Improving on the mle of a bounded normal mean. Annals of Statistics, 29:1066–1081, 2001.

[MP02] É. Marchand and F. Perron. On the minimax estimator of a bounded normal mean. Statistics and Probability Letters, 58:327–333, 2002.

[MS10] É. Marchand and W. E. Strawderman. A unified minimax result for restricted parameter spaces. Technical report, Département de Mathématiques, Université de Sherbrooke, 2010. www.usherbrooke.ca/mathematiques/recherche/publications/rapportsrecherche/, to appear in Bernoulli.

[Mur77] G. D. Murray. A note on the estimation of probability density functions. Biometrika, 64:150–152, 1977.

[Ng80] V. M. Ng. On the estimation of parametric density functions. Biometrika, 67:505–506, 1980.

[Roc96] R. T. Rockafellar. Convex Analysis. Princeton University Press, 1996.

[TK08] H. Tsukuma and T. Kubokawa. Stein’s phenomenon in estimation of means restricted to a polyhedral convex cone. Journal of Multivariate Analysis, 99:141– 164, 2008.