Scoring algorithm

Scoring algorithm, also known as Fisher's scoring,^[1] is a form of Newton's method used in statistics to solve maximum likelihood equations numerically, named after Ronald Fisher.

Sketch of derivation

Let $Y_{1},\ldots ,Y_{n}$ be random variables, independent and identically distributed with twice differentiable p.d.f. $f(y;\theta )$ , and we wish to calculate the maximum likelihood estimator (M.L.E.) $\theta^*$ of $\theta$ . First, suppose we have a starting point for our algorithm $\theta _{0}$ , and consider a Taylor expansion of the score function, $V(\theta )$ , about $\theta _{0}$ :

V(\theta )\approx V(\theta _{0})-{\mathcal {J}}(\theta _{0})(\theta -\theta _{0}),\,

where

{\mathcal {J}}(\theta _{0})=-\sum _{{i=1}}^{n}\left.\nabla \nabla ^{{\top }}\right|_{{\theta =\theta _{0}}}\log f(Y_{i};\theta )

is the observed information matrix at $\theta _{0}$ . Now, setting $\theta =\theta ^{*}$ , using that $V(\theta ^{*})=0$ and rearranging gives us:

\theta ^{*}\approx \theta _{{0}}+{\mathcal {J}}^{{-1}}(\theta _{{0}})V(\theta _{{0}}).\,

We therefore use the algorithm

\theta _{{m+1}}=\theta _{{m}}+{\mathcal {J}}^{{-1}}(\theta _{{m}})V(\theta _{{m}}),\,

and under certain regularity conditions, it can be shown that $\theta _{m}\rightarrow \theta ^{*}$ .

Fisher scoring

In practice, ${\mathcal {J}}(\theta )$ is usually replaced by ${\mathcal {I}}(\theta )={\mathrm {E}}[{\mathcal {J}}(\theta )]$ , the Fisher information, thus giving us the Fisher Scoring Algorithm:

\theta _{{m+1}}=\theta _{{m}}+{\mathcal {I}}^{{-1}}(\theta _{{m}})V(\theta _{{m}})

..

References

^ Longford, Nicholas T. (1987). "A fast scoring algorithm for maximum likelihood estimation in unbalanced mixed models with nested random effects". Biometrika. 74 (4): 817–827. doi:10.1093/biomet/74.4.817.

Scoring algorithm

Sketch of derivation

Fisher scoring

See also

References

Further reading