Wikipedia

Proof of Stein's example

Stein's example is an important result in decision theory which can be stated as

The ordinary decision rule for estimating the mean of a multivariate Gaussian distribution is inadmissible under mean squared error risk in dimension at least 3.

The following is an outline of its proof.^[1] The reader is referred to the main article for more information.

Sketched proof

The risk function of the decision rule $d({\mathbf {x}})={\mathbf {x}}$ is

R(\theta ,d)=\operatorname {E} _{\theta }[|\mathbf {\theta -X} |^{2}]

=\int ({\mathbf {\theta -x}})^{T}({\mathbf {\theta -x}})\left({\frac {1}{2\pi }}\right)^{{n/2}}e^{{(-1/2)({\mathbf {\theta -x}})^{T}({\mathbf {\theta -x}})}}m(dx)

=n.

Now consider the decision rule

d'({\mathbf {x}})={\mathbf {x}}-{\frac {\alpha }{|{\mathbf {x}}|^{2}}}{\mathbf {x}}

where $\alpha =n-2$ . We will show that $d'$ is a better decision rule than $d$ . The risk function is

R(\theta ,d')=\operatorname {E} _{\theta }\left[\left|\mathbf {\theta -X} +{\frac {\alpha }{|\mathbf {X} |^{2}}}\mathbf {X} \right|^{2}\right]

=\operatorname {E} _{\theta }\left[|\mathbf {\theta -X} |^{2}+2(\mathbf {\theta -X} )^{T}{\frac {\alpha }{|\mathbf {X} |^{2}}}\mathbf {X} +{\frac {\alpha ^{2}}{|\mathbf {X} |^{4}}}|\mathbf {X} |^{2}\right]

=\operatorname {E} _{\theta }\left[|\mathbf {\theta -X} |^{2}\right]+2\alpha \operatorname {E} _{\theta }\left[{\frac {\mathbf {(\theta -X)^{T}X} }{|\mathbf {X} |^{2}}}\right]+\alpha ^{2}\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right]

— a quadratic in $\alpha$ . We may simplify the middle term by considering a general "well-behaved" function $h:{\mathbf {x}}\mapsto h({\mathbf {x}})\in {\mathbb {R}}$ and using integration by parts. For $1\leq i\leq n$ , for any continuously differentiable $h$ growing sufficiently slowly for large $x_{i}$ we have:

\operatorname {E} _{\theta }[(\theta _{i}-X_{i})h(\mathbf {X} )|X_{j}=x_{j}(j\neq i)]=\int (\theta _{i}-x_{i})h(\mathbf {x} )\left({\frac {1}{2\pi }}\right)^{n/2}e^{-(1/2)\mathbf {(x-\theta )} ^{T}\mathbf {(x-\theta )} }m(dx_{i})

=\left[h({\mathbf {x}})\left({\frac {1}{2\pi }}\right)^{{n/2}}e^{{-(1/2){\mathbf {(x-\theta )}}^{T}{\mathbf {(x-\theta )}}}}\right]_{{x_{i}=-\infty }}^{\infty }-\int {\frac {\partial h}{\partial x_{i}}}({\mathbf {x}})\left({\frac {1}{2\pi }}\right)^{{n/2}}e^{{-(1/2){\mathbf {(x-\theta )}}^{T}{\mathbf {(x-\theta )}}}}m(dx_{i})

=-\operatorname {E} _{\theta }\left[{\frac {\partial h}{\partial x_{i}}}(\mathbf {X} )|X_{j}=x_{j}(j\neq i)\right].

Therefore,

\operatorname {E} _{\theta }[(\theta _{i}-X_{i})h(\mathbf {X} )]=-\operatorname {E} _{\theta }\left[{\frac {\partial h}{\partial x_{i}}}(\mathbf {X} )\right].

(This result is known as Stein's lemma.)

Now, we choose

h({\mathbf {x}})={\frac {x_{i}}{|{\mathbf {x}}|^{2}}}.

If $h$ met the "well-behaved" condition (it doesn't, but this can be remedied—see below), we would have

{\frac {\partial h}{\partial x_{i}}}={\frac {1}{|{\mathbf {x}}|^{2}}}-{\frac {2x_{i}^{2}}{|{\mathbf {x}}|^{4}}}

and so

\operatorname {E} _{\theta }\left[{\frac {\mathbf {(\theta -X)^{T}X} }{|\mathbf {X} |^{2}}}\right]=\sum _{i=1}^{n}\operatorname {E} _{\theta }\left[(\theta _{i}-X_{i}){\frac {X_{i}}{|\mathbf {X} |^{2}}}\right]

=-\sum _{i=1}^{n}\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}-{\frac {2X_{i}^{2}}{|\mathbf {X} |^{4}}}\right]

=-(n-2)\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right].

Then returning to the risk function of $d'$ :

R(\theta ,d')=n-2\alpha (n-2)\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right]+\alpha ^{2}\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right].

This quadratic in $\alpha$ is minimized at

\alpha =n-2,

giving

R(\theta ,d')=R(\theta ,d)-(n-2)^{2}\operatorname {E} _{\theta }\left[{\frac {1}{|\mathbf {X} |^{2}}}\right]

which of course satisfies

R(\theta ,d')<R(\theta ,d).

making $d$ an inadmissible decision rule.

It remains to justify the use of

h({\mathbf {X}})={\frac {{\mathbf {X}}}{|{\mathbf {X}}|^{2}}}.

This function is not continuously differentiable, since it is singular at ${\mathbf {x}}=0$ . However, the function

h(\mathbf {X} )={\frac {\mathbf {X} }{\varepsilon +|\mathbf {X} |^{2}}}

is continuously differentiable, and after following the algebra through and letting $\varepsilon \to 0$ , one obtains the same result.

References

^ Samworth, Richard (December 2012). "Stein's Paradox" (PDF). Eureka. 62: 38–41.

This article is copied from an article on Wikipedia® - the free encyclopedia created and edited by its online user community. The text was not checked or edited by anyone on our staff. Although the vast majority of Wikipedia® encyclopedia articles provide accurate and timely information, please do not assume the accuracy of any particular article. This article is distributed under the terms of GNU Free Documentation License.

Copyright © 2003-2025 Farlex, Inc Disclaimer

All content on this website, including dictionary, thesaurus, literature, geography, and other reference data is for informational purposes only. This information should not be considered complete, up to date, and is not intended to be used in place of a visit, consultation, or advice of a legal, medical, or any other professional.