Normal distribution

Normal distribution
From Wikipedia, the free encyclopedia
Jump to: navigation, search
Normal Probability density function
Probability density function for the normal distribution
The green line is the standard normal distribution
Cumulative distribution function
Cumulative distribution function for the normal distribution
Colors match the image above
Parameters μ location (real)
σ2 > 0 squared scale (real)
Support x \in\mathbb{R}\!
Probability density function (pdf) \frac1{\sigma\sqrt{2\pi}}\; \exp\left(-\frac{\left(x-\mu\right)^2}{2\sigma^2} \right) \!
Cumulative distribution function (cdf) \frac12 \left(1+\mathrm{erf}\,\frac{x-\mu}{\sigma\sqrt2}\right) \!
Mean μ
Median μ
Mode μ
Variance σ2
Skewness 0
Excess kurtosis 0
Entropy \ln\left(\sigma\sqrt{2\,\pi\,e}\right)\!
Moment-generating function (mgf) M_X(t)= \exp\left(\mu\,t+\frac{\sigma^2 t^2}{2}\right)
Characteristic function \chi_X(t)=\exp\left(\mu\,i\,t-\frac{\sigma^2 t^2}{2}\right)

The normal distribution, also called the Gaussian distribution, is an important family of continuous probability distributions, applicable in many fields. Each member of the family may be defined by two parameters, location and scale: the mean ("average") and standard deviation ("variability"), respectively. The standard normal distribution is the normal distribution with a mean of zero and a variance of one (the green curves in the plots to the right). Carl Friedrich Gauss became associated with this set of distributions when he analyzed astronomical data using them [1], and defined the equation of its probability density function. It is often called the bell curve because the graph of its probability density resembles a bell.

The importance of the normal distribution as a model of quantitative phenomena in the natural and behavioral sciences is due to the central limit theorem. Many psychological measurements and physical phenomena (like noise) can be approximated well by the normal distribution. While the mechanisms underlying these phenomena are often unknown, the use of the normal model can be theoretically justified by assuming that many small, independent effects are additively contributing to each observation.

The normal distribution also arises in many areas of statistics. For example, the sampling distribution of the sample mean is approximately normal, even if the distribution of the population from which the sample is taken is not normal. In addition, the normal distribution maximizes information entropy among all distributions with known mean and variance, which makes it the natural choice of underlying distribution for data summarized in terms of sample mean and variance. The normal distribution is the most widely used family of distributions in statistics and many statistical tests are based on the assumption of normality. In probability theory, normal distributions arise as the limiting distributions of several continuous and discrete families of distributions.
Contents
[hide]

* 1 History
* 2 Characterization
o 2.1 Probability density function
o 2.2 Cumulative distribution function
+ 2.2.1 Strict lower and upper bounds for the cdf
o 2.3 Generating functions
+ 2.3.1 Moment generating function
+ 2.3.2 Cumulant generating function
+ 2.3.3 Characteristic function
* 3 Properties
o 3.1 Standardizing normal random variables
o 3.2 Moments
o 3.3 Generating values for normal random variables
o 3.4 The central limit theorem
o 3.5 Infinite divisibility
o 3.6 Stability
o 3.7 Standard deviation
o 3.8 Exponential family
* 4 Complex Gaussian process
* 5 Related distributions
* 6 Descriptive and inferential statistics
o 6.1 Scores
o 6.2 Normality tests
o 6.3 Estimation of parameters
+ 6.3.1 Maximum likelihood estimation of parameters
# 6.3.1.1 Surprising generalization
+ 6.3.2 Unbiased estimation of parameters
o 6.4 Occurrence
+ 6.4.1 Photon counting
o 6.5 Measurement errors
o 6.6 Physical characteristics of biological specimens
o 6.7 Financial variables
o 6.8 Distribution in testing and intelligence
* 7 Numerical approximations of the normal distribution and its CDF
* 8 Trivia
* 9 See also
* 10 References
* 11 External links

[edit] History

The normal distribution was first introduced by Abraham de Moivre in an article in 1734, which was reprinted in the second edition of his The Doctrine of Chances, 1738 in the context of approximating certain binomial distributions for large n. His result was extended by Laplace in his book Analytical Theory of Probabilities (1812), and is now called the theorem of de Moivre-Laplace.

Laplace used the normal distribution in the analysis of errors of experiments. The important method of least squares was introduced by Legendre in 1805. Gauss, who claimed to have used the method since 1794, justified it rigorously in 1809 by assuming a normal distribution of the errors.

The name "bell curve" goes back to Jouffret who first used the term "bell surface" in 1872 for a bivariate normal with independent components. The name "normal distribution" was coined independently by Charles S. Peirce, Francis Galton and Wilhelm Lexis around 1875. This terminology unfortunately encourages the fallacy that many or all other probability distributions are not "normal". (See the discussion of "occurrence" below.)

[edit] Characterization

There are various ways to characterize a probability distribution. The most visual is the probability density function (PDF); the PDF of the normal distribution is plotted at the beginning of this article. Equivalent ways are the cumulative distribution function, the moments, the cumulants, the characteristic function, the moment-generating function, the cumulant-generating function, and Maxwell's theorem. See probability distribution for a discussion.

To indicate that a real-valued random variable X is normally distributed with mean μ and variance σ2 ≥ 0, we write

X \sim N(\mu, \sigma^2).\,\!

While it is certainly useful for certain limit theorems (e.g. asymptotic normality of estimators) and for the theory of Gaussian processes to consider the probability distribution concentrated at μ (see Dirac measure) as a normal distribution with mean μ and variance σ2 = 0, this degenerated case is often excluded from the considerations because no density with respect to the Lebesgue measure exists.

[edit] Probability density function

The continuous probability density function of the normal distribution is the Gaussian function

\varphi_{\mu,\sigma}(x) = \frac{1}{\sigma\sqrt{2\pi}} \,\exp\biggl( -\frac{(x- \mu)^2}{2\sigma^2}\biggr) = \frac{1}{\sigma} \varphi\left(\frac{x - \mu}{\sigma}\right),\quad x\in\mathbb{R},

where σ > 0 is the standard deviation, the real parameter μ is the expected value, and

\varphi(x)=\varphi_{0,1}(x)=\frac{1}{\sqrt{2\pi\,}} \, e^{-\frac{1}{2}x^2},\quad x\in\mathbb{R},

is the density function of the "standard" normal distribution, i.e., the normal distribution with μ = 0 and σ = 1. To verify that the integral of \varphi_{\mu,\sigma} over the real line is indeed equal to one, see Gaussian integral.

As a Gaussian function with the denominator of the exponent equal to 2, the standard normal density function \scriptstyle\varphi is an eigenfunction of the Fourier transform.

Some notable qualities of the probability density function:

* The density function is symmetric about its mean value μ.
* The mean μ is also its mode and median.
* The inflection points of the curve occur at one standard deviation away from the mean, i.e. at μ − σ and μ + σ.

[edit] Cumulative distribution function

The cumulative distribution function (cdf) of a probability distribution, evaluated at a number (lower-case) x, is the probability of the event that a random variable (capital) X with that distribution is less than or equal to x. The cumulative distribution function of the normal distribution is expressed in terms of the density function as follows:

\begin{align} \Phi_{\mu,\sigma}(x) &{}=\int_{-\infty}^x\varphi_{\mu,\sigma}(u)\,du\\ &{}=\frac{1}{\sigma\sqrt{2\pi}} \int_{-\infty}^x \exp \Bigl( -\frac{(u - \mu)^2}{2\sigma^2} \ \Bigr)\, du \\ &{}= \Phi\Bigl(\frac{x-\mu}{\sigma}\Bigr),\quad x\in\mathbb{R}, \end{align}

where the standard normal cdf Φ is just the general cdf evaluated with μ = 0 and σ = 1:

\Phi(x) = \Phi_{0,1}(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^x \exp\Bigl(-\frac{u^2}{2}\Bigr) \, du, \quad x\in\mathbb{R}.

The standard normal cdf can be expressed in terms of a special function called the error function, as

\Phi(x) =\frac{1}{2} \Bigl[ 1 + \operatorname{erf} \Bigl( \frac{x}{\sqrt{2}} \Bigr) \Bigr], \quad x\in\mathbb{R},

and the cdf itself can hence be expressed as

\Phi_{\mu,\sigma}(x;\mu) =\frac{1}{2} \Bigl[ 1 + \operatorname{erf} \Bigl( \frac{x-\mu}{\sigma\sqrt{2}} \Bigr) \Bigr], \quad x\in\mathbb{R}.

The inverse standard normal cumulative distribution function, or quantile function, can be expressed in terms of the inverse error function:

\Phi^{-1}(p) = \sqrt2 \;\operatorname{erf}^{-1} (2p - 1), \quad p\in(0,1),

and the inverse cumulative distribution function can hence be expressed as

\Phi_{\mu,\sigma}^{-1}(p) = \mu + \sigma\Phi^{-1}(p) = \mu + \sigma\sqrt2 \; \operatorname{erf}^{-1}(2p - 1), \quad p\in(0,1).

This quantile function is sometimes called the probit function. There is no elementary primitive for the probit function. This is not to say merely that none is known, but rather that the non-existence of such a function has been proved. Several accurate methods exist for approximating the quantile function for the normal distribution - see quantile function for a discussion and references.

The values Φ(x) may be approximated very accurately by a variety of methods, such as numerical integration, Taylor series, asymptotic series and continued fractions.

[edit] Strict lower and upper bounds for the cdf

For large x the standard normal cdf \scriptstyle\Phi(x) is close to 1 and \scriptstyle\Phi(-x)\,{=}\,1\,{-}\,\Phi(x) is close to 0. The elementary bounds

\frac{x}{1+x^2}\varphi(x)<1-\Phi(x)<\frac{\varphi(x)}{x}, \qquad x>0,

in terms of the density \scriptstyle\varphi are useful.

Using the substitution v = u2/2, the upper bound is derived as follows:

\begin{align} 1-\Phi(x) &=\int_x^\infty\varphi(u)\,du\\ &<\int_x^\infty\frac ux\varphi(u)\,du =\int_{x^2/2}^\infty\frac{e^{-v}}{x\sqrt{2\pi}}\,dv =-\biggl.\frac{e^{-v}}{x\sqrt{2\pi}}\biggr|_{x^2/2}^\infty =\frac{\varphi(x)}{x}. \end{align}

Similarly, using \scriptstyle\varphi'(u)\,{=}\,-u\,\varphi(u) and the quotient rule,

\begin{align} \Bigl(1+\frac1{x^2}\Bigr)(1-\Phi(x)) &=\int_x^\infty \Bigl(1+\frac1{x^2}\Bigr)\varphi(u)\,du\\ &>\int_x^\infty \Bigl(1+\frac1{u^2}\Bigr)\varphi(u)\,du =-\biggl.\frac{\varphi(u)}u\biggr|_x^\infty =\frac{\varphi(x)}x. \end{align}

Solving for \scriptstyle 1\,{-}\,\Phi(x)\, provides the lower bound.

[edit] Generating functions

[edit] Moment generating function

The moment generating function is defined as the expected value of exp(tX). For a normal distribution, the moment generating function is

\begin{align} M_X(t) & {} = \mathrm{E} \left[ \exp{(tX)} \right] \\ & {} = \int_{-\infty}^{\infty} \frac{1}{\sigma \sqrt{2\pi} } \exp{\left( -\frac{(x - \mu)^2}{2 \sigma^2} \right)} \exp{(tx)} \, dx \\ & {} = \exp{ \left( \mu t + \frac{\sigma^2 t^2}{2} \right)} \end{align}

as can be seen by completing the square in the exponent.

[edit] Cumulant generating function

The cumulant generating function is the logarithm of the moment generating function: g(t) = μt + σ2t2/2. Since this is a quadratic polynomial in t, only the first two cumulants are nonzero.

[edit] Characteristic function

The characteristic function is defined as the expected value of exp(itX), where i is the imaginary unit. So the characteristic function is obtained by replacing t with i t in the moment-generating function.

For a normal distribution, the characteristic function is

\begin{align} \chi_X(t;\mu,\sigma) &{} = M_X(i t) = \mathrm{E} \left[ \exp(i t X) \right] \\ &{}= \int_{-\infty}^{\infty} \frac{1}{\sigma \sqrt{2\pi}} \exp \left(- \frac{(x - \mu)^2}{2\sigma^2} \right) \exp(i t x) \, dx \\ &{}= \exp \left( i \mu t - \frac{\sigma^2 t^2}{2} \right). \end{align}

[edit] Properties

Some of the properties of the normal distribution:

1. If X \sim N(\mu, \sigma^2) and a and b are real numbers, then a X + b \sim N(a \mu + b, (a \sigma)^2) (see expected value and variance).
2. If X \sim N(\mu_X, \sigma^2_X) and Y \sim N(\mu_Y, \sigma^2_Y) are independent normal random variables, then:
* Their sum is normally distributed with U = X + Y \sim N(\mu_X + \mu_Y, \sigma^2_X + \sigma^2_Y) (proof).
* Their difference is normally distributed with V = X - Y \sim N(\mu_X - \mu_Y, \sigma^2_X + \sigma^2_Y).
* Both U and V are independent of each other. (This holds only for standard normal X and Y, verification needed.)
* The Kullback-Leibler divergence, D_{\rm KL}( X \| Y ) = { 1 \over 2 } \left( \log \left( { \sigma^2_Y \over \sigma^2_X } \right) + \frac{\sigma^2_X}{\sigma^2_Y} + \frac{\left(\mu_Y - \mu_X\right)^2}{\sigma^2_Y} - 1\right).
3. If X \sim N(0, \sigma^2_X) and Y \sim N(0, \sigma^2_Y) are independent normal random variables, then:
* Their product XY follows a distribution with density p given by

p(z) = \frac{1}{\pi\,\sigma_X\,\sigma_Y} \; K_0\left(\frac{|z|}{\sigma_X\,\sigma_Y}\right), where K0 is a modified Bessel function of the second kind.

* Their ratio follows a Cauchy distribution with X/Y \sim \mathrm{Cauchy}(0, \sigma_X/\sigma_Y). Thus the Cauchy distribution is a special kind of ratio distribution.
4. If X_1, \dots, X_n are independent standard normal variables, then X_1^2 + \cdots + X_n^2 has a chi-square distribution with n degrees of freedom.

[edit] Standardizing normal random variables

As a consequence of Property 1, it is possible to relate all normal random variables to the standard normal.

If X ~ N(μ,σ2), then

Z = \frac{X - \mu}{\sigma} \!

is a standard normal random variable: Z ~ N(0,1). An important consequence is that the cdf of a general normal distribution is therefore

\Pr(X \le x) = \Phi \left( \frac{x-\mu}{\sigma} \right) = \frac{1}{2} \left( 1 + \operatorname{erf} \left( \frac{x-\mu}{\sigma\sqrt{2}} \right) \right) .

Conversely, if Z is a standard normal distribution, Z ~ N(0,1), then

X = σZ + μ

is a normal random variable with mean μ and variance σ2.

The standard normal distribution has been tabulated (usually in the form of value of the cumulative distribution function Φ), and the other normal distributions are the simple transformations, as described above, of the standard one. Therefore, one can use tabulated values of the cdf of the standard normal distribution to find values of the cdf of a general normal distribution.

[edit] Moments

Some of the first few moments of the normal distribution are:
Number Raw moment Central moment Cumulant
0 1 1
1 μ 0 μ
2 μ2 + σ2 σ2 σ2
3 μ3 + 3μσ2 0 0
4 μ4 + 6μ2σ2 + 3σ4 3σ4 0

All cumulants of the normal distribution beyond the second are zero.

[edit] Generating values for normal random variables

For computer simulations, it is often useful to generate values that have a normal distribution. There are several methods and the most basic is to invert the standard normal cdf. More efficient methods are also known, one such method being the Box-Muller transform. An even faster algorithm is the ziggurat algorithm.

The Box-Muller algorithm says that, if you have two numbers a and b uniformly distributed on (0, 1], (e.g. the output from a random number generator), then two standard normally distributed random variables are c and d, where:

c = \sqrt{- 2 \ln a} \cdot \cos(2 \pi b)

d = \sqrt{- 2 \ln a} \cdot \sin(2 \pi b)

This is because the chi-square distribution with two degrees of freedom (see property 4 above) is an easily-generated exponential random variable.

[edit] The central limit theorem

Main article: central limit theorem

Plot of the pdf of a normal distribution with μ = 12 and σ = 3, approximating the pdf of a binomial distribution with n = 48 and p = 1/4
Plot of the pdf of a normal distribution with μ = 12 and σ = 3, approximating the pdf of a binomial distribution with n = 48 and p = 1/4

The normal distribution has the very important property that under certain conditions, the distribution of a sum of a large number of identically distributed independent variables is approximately normal. This is the central limit theorem.

The practical importance of the central limit theorem is that the normal distribution can be used as an approximation to some other distributions.

* A binomial distribution with parameters n and p is approximately normal for large n and p not too close to 1 or 0 (some books recommend using this approximation only if np and n(1 − p) are both at least 5; in this case, a continuity correction should be applied).

The approximating normal distribution has mean μ = np and variance σ2 = np(1 − p).

* A Poisson distribution with parameter λ is approximately normal for large λ.

The approximating normal distribution has mean μ = λ and variance σ2 = λ.

Whether these approximations are sufficiently accurate depends on the purpose for which they are needed, and the rate of convergence to the normal distribution. It is typically the case that such approximations are less accurate in the tails of the distribution.

[edit] Infinite divisibility

The normal distributions are infinitely divisible probability distributions.

[edit] Stability

The normal distributions are strictly stable probability distributions.

[edit] Standard deviation
Dark blue is less than one standard deviation from the mean. For the normal distribution, this accounts for about 68% of the set (dark blue) while two standard deviations from the mean (medium and dark blue) account for about 95% and three standard deviations (light, medium, and dark blue) account for about 99.7%.
Dark blue is less than one standard deviation from the mean. For the normal distribution, this accounts for about 68% of the set (dark blue) while two standard deviations from the mean (medium and dark blue) account for about 95% and three standard deviations (light, medium, and dark blue) account for about 99.7%.

About 68% of values drawn from a standard normal distribution are within 1 standard deviation away from the mean; about 95% of the values are within two standard deviations and about 99.7% lie within 3 standard deviations. This is known as the "68-95-99.7 rule" or the "empirical rule."

To be more precise, the area under the curve between − nσ and nσ is

\mathrm{erf}\left(n/\sqrt{2}\right)\,

where erf(x) is the error function. To six decimal places the values of the 1,2 and 3 sigma points are 0.682689, 0.954499, 0.997300 respectively.

[edit] Exponential family

The Normal distribution is a two-parameter exponential family with natural parameters μ and 1 / σ2, and natural statistics X and X2.

[edit] Complex Gaussian process

Consider complex Gaussian random variable,

Z=X+iY\,

where X\, and Y\, are real and independent Gaussian variables with equal variances \sigma_r^2\,. The pdf of the joint variables is then

\frac{1}{2\,\pi\,\sigma_r^2} e^{-\frac{x^2+y^2}{2 \sigma_r ^2}}

Because \sigma_z=\sqrt{2}\sigma_r, the resulting PDF for the complex Gaussian variable Z\, is

\frac{1}{\pi\,\sigma_z^2} e^{-\frac{|z|^2}{\sigma_z^2}}.

[edit] Related distributions

* R \sim \mathrm{Rayleigh}(\sigma^2) is a Rayleigh distribution if R = \sqrt{X^2 + Y^2} where X \sim N(0, \sigma^2) and Y \sim N(0, \sigma^2) are two independent normal distributions.
* Y \sim \chi_{\nu}^2 is a chi-square distribution with ν degrees of freedom if Y = \sum_{k=1}^{\nu} X_k^2 where X_k \sim N(0,1) for k=1,\dots,\nu and are independent.
* Y \sim \mathrm{Cauchy}(\mu = 0, \theta = 1) is a Cauchy distribution if Y = X1 / X2 for X_1 \sim N(0,1) and X_2 \sim N(0,1) are two independent normal distributions.
* Y \sim \mbox{Log-N}(\mu, \sigma^2) is a log-normal distribution if Y = eX and X \sim N(\mu, \sigma^2).
* Relation to Lévy skew alpha-stable distribution: if X\sim \textrm{Levy-S}\alpha\textrm{S}(2,\beta,\sigma/\sqrt{2},\mu) then X \sim N(\mu,\sigma^2).
* Truncated normal distribution. If, X \sim N(\mu, \sigma^2) then, truncating below at A and above at B will lead to a random variable with mean E(X)=\mu + \frac{\sigma(\varphi_1-\varphi_2)}{T}, where T=\Phi\left(\frac{B-\mu}{\sigma}\right)-\Phi\left(\frac{A-\mu}{\sigma}\right) and \varphi_1 = \varphi\left(\frac{A-\mu}{\sigma}\right) and \varphi_2 = \varphi\left(\frac{B-\mu}{\sigma}\right), where \varphi is the probability density function of a standard normal random variable.
* If X is a random variable with a normal distribution, and Y = | X | , then Y has a folded normal distribution.

[edit] Descriptive and inferential statistics

[edit] Scores

Many scores are derived from the normal distribution, including percentile ranks ("percentiles"), normal curve equivalents, stanines, z-scores, and T-scores. Additionally, a number of behavioral statistical procedures are based on the assumption that scores are normally distributed; for example, t-tests and ANOVAs (see below). Bell curve grading assigns relative grades based on a normal distribution of scores.
This short section requires expansion.

[edit] Normality tests

Main article: normality test

Normality tests check a given set of data for similarity to the normal distribution. The null hypothesis is that the data set is similar to the normal distribution, therefore a sufficiently small P-value indicates non-normal data.

* Kolmogorov-Smirnov test
* Lilliefors test
* Anderson-Darling test
* Ryan-Joiner test
* Shapiro-Wilk test
* Normal probability plot (rankit plot)
* Jarque-Bera test

[edit] Estimation of parameters

[edit] Maximum likelihood estimation of parameters

Suppose

X_1,\dots,X_n

are independent and each is normally distributed with expectation μ and variance σ2 > 0. In the language of statisticians, the observed values of these n random variables make up a "sample of size n from a normally distributed population." It is desired to estimate the "population mean" μ and the "population standard deviation" σ, based on the observed values of this sample. The continuous joint probability density function of these n independent random variables is

\begin{align}f(x_1,\dots,x_n;\mu,\sigma) &= \prod_{i=1}^n \varphi_{\mu,\sigma}(x_i)\\ &=\frac1{(\sigma\sqrt{2\pi})^n}\prod_{i=1}^n \exp\biggl(-{1 \over 2} \Bigl({x_i-\mu \over \sigma}\Bigr)^2\biggr), \quad(x_1,\ldots,x_n)\in\mathbb{R}^n. \end{align}

As a function of μ and σ, the likelihood function based on the observations X1, ..., Xn is

L(\mu,\sigma) = \frac C{\sigma^n} \exp\left(-{\sum_{i=1}^n (X_i-\mu)^2 \over 2\sigma^2}\right), \quad\mu\in\mathbb{R},\ \sigma>0,

with some constant C > 0 (which in general would be even allowed to depend on X1, ..., Xn, but will vanish anyway when partial derivatives of the log-likelihood function with respect to the parameters are computed, see below).

In the method of maximum likelihood, the values of μ and σ that maximize the likelihood function are taken as estimates of the population parameters μ and σ.

Usually in maximizing a function of two variables, one might consider partial derivatives. But here we will exploit the fact that the value of μ that maximizes the likelihood function with σ fixed does not depend on σ. Therefore, we can find that value of μ, then substitute it for μ in the likelihood function, and finally find the value of σ that maximizes the resulting expression.

It is evident that the likelihood function is a decreasing function of the sum

\sum_{i=1}^n (X_i-\mu)^2. \,\!

So we want the value of μ that minimizes this sum. Let

\overline{X}_n=(X_1+\cdots+X_n)/n

be the "sample mean" based on the n observations. Observe that

\begin{align} \sum_{i=1}^n (X_i-\mu)^2 &=\sum_{i=1}^n\bigl((X_i-\overline{X}_n)+(\overline{X}_n-\mu)\bigr)^2\\ &=\sum_{i=1}^n(X_i-\overline{X}_n)^2 + 2(\overline{X}_n-\mu)\underbrace{\sum_{i=1}^n (X_i-\overline{X}_n)}_{=\,0} + \sum_{i=1}^n (\overline{X}_n-\mu)^2\\ &=\sum_{i=1}^n(X_i-\overline{X}_n)^2 + n(\overline{X}_n-\mu)^2. \end{align}

Only the last term depends on μ and it is minimized by

\widehat{\mu}_n=\overline{X}_n.

That is the maximum-likelihood estimate of μ based on the n observations X1, ..., Xn. When we substitute that estimate for μ into the likelihood function, we get

L(\overline{X}_n,\sigma) = \frac C{\sigma^n} \exp\biggl(-{\sum_{i=1}^n (X_i-\overline{X}_n)^2 \over 2\sigma^2}\biggr), \quad\sigma>0.

It is conventional to denote the "log-likelihood function", i.e., the logarithm of the likelihood function, by a lower-case \ell, and we have

\ell(\overline{X}_n,\sigma)=\log C-n\log\sigma-{\sum_{i=1}^n(X_i-\overline{X}_n)^2 \over 2\sigma^2}, \quad\sigma>0,

and then

\begin{align} {\partial \over \partial\sigma}\ell(\overline{X}_n,\sigma) &=-{n \over \sigma} +{\sum_{i=1}^n (X_i-\overline{X}_n)^2 \over \sigma^3}\\ &=-{n \over \sigma^3}\biggl(\sigma^2-{1 \over n}\sum_{i=1}^n (X_i-\overline{X}_n)^2 \biggr), \quad\sigma>0. \end{align}

This derivative is positive, zero, or negative according as σ2 is between 0 and

\hat\sigma_n^2:={1 \over n}\sum_{i=1}^n(X_i-\overline{X}_n)^2,

or equal to that quantity, or greater than that quantity. (If there is just one observation, meaning that n = 1, or if X1 = ... = Xn, which only happens with probability zero, then \hat\sigma{}_n^2=0 by this formula, reflecting the fact that in these cases the likelihood function is unbounded as σ decreases to zero.)

Consequently this average of squares of residuals is the maximum-likelihood estimate of σ2, and its square root is the maximum-likelihood estimate of σ based on the n observations. This estimator \hat\sigma{}_n^2 is biased, but has a smaller mean squared error than the usual unbiased estimator, which is n/(n − 1) times this estimator.

[edit] Surprising generalization

The derivation of the maximum-likelihood estimator of the covariance matrix of a multivariate normal distribution is subtle. It involves the spectral theorem and the reason it can be better to view a scalar as the trace of a 1×1 matrix than as a mere scalar. See estimation of covariance matrices.

[edit] Unbiased estimation of parameters

The maximum likelihood estimator of the population mean μ from a sample is an unbiased estimator of the mean, as is the variance when the mean of the population is known a priori. However, if we are faced with a sample and have no knowledge of the mean or the variance of the population from which it is drawn, the unbiased estimator of the variance σ2 is:

S^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \overline{X})^2.

This "sample variance" follows a Gamma distribution if all X are independent identically distributed (iid):

S^2 \sim \operatorname{Gamma}\left(\frac{n-1}{2},\frac{2 \sigma^2}{n-1}\right).

[edit] Occurrence

Approximately normal distributions occur in many situations, as a result of the central limit theorem. When there is reason to suspect the presence of a large number of small effects acting additively and independently, it is reasonable to assume that observations will be normal. There are statistical methods to empirically test that assumption, for example the Kolmogorov-Smirnov test.

Effects can also act as multiplicative (rather than additive) modifications. In that case, the assumption of normality is not justified, and it is the logarithm of the variable of interest that is normally distributed. The distribution of the directly observed variable is then called log-normal.

Finally, if there is a single external influence which has a large effect on the variable under consideration, the assumption of normality is not justified either. This is true even if, when the external variable is held constant, the resulting marginal distributions are indeed normal. The full distribution will be a superposition of normal variables, which is not in general normal. This is related to the theory of errors (see below).

To summarize, here is a list of situations where approximate normality is sometimes assumed. For a fuller discussion, see below.

* In counting problems (so the central limit theorem includes a discrete-to-continuum approximation) where reproductive random variables are involved, such as
o Binomial random variables, associated to yes/no questions;
o Poisson random variables, associated to rare events;
* In physiological measurements of biological specimens:
o The logarithm of measures of size of living tissue (length, height, skin area, weight);
o The length of inert appendages (hair, claws, nails, teeth) of biological specimens, in the direction of growth; presumably the thickness of tree bark also falls under this category;
o Other physiological measures may be normally distributed, but there is no reason to expect that a priori;
* Measurement errors are often assumed to be normally distributed, and any deviation from normality is considered something which should be explained;
* Financial variables
o Changes in the logarithm of exchange rates, price indices, and stock market indices; these variables behave like compound interest, not like simple interest, and so are multiplicative;
o Other financial variables may be normally distributed, but there is no reason to expect that a priori;
* Light intensity
o The intensity of laser light is normally distributed;
o Thermal light has a Bose-Einstein distribution on very short time scales, and a normal distribution on longer timescales due to the central limit theorem.

Of relevance to biology and economics is the fact that complex systems tend to display power laws rather than normality.

[edit] Photon counting

Light intensity from a single source varies with time, as thermal fluctuations can be observed if the light is analyzed at sufficiently high time resolution. The intensity is usually assumed to be normally distributed. Quantum mechanics interprets measurements of light intensity as photon counting. The natural assumption in this setting is the Poisson distribution. When light intensity is integrated over times longer than the coherence time and is large, the Poisson-to-normal limit is appropriate.

[edit] Measurement errors

Normality is the central assumption of the mathematical theory of errors. Similarly, in statistical model-fitting, an indicator of goodness of fit is that the residuals (as the errors are called in that setting) be independent and normally distributed. The assumption is that any deviation from normality needs to be explained. In that sense, both in model-fitting and in the theory of errors, normality is the only observation that need not be explained, being expected. However, if the original data are not normally distributed (for instance if they follow a Cauchy distribution), then the residuals will also not be normally distributed. This fact is usually ignored in practice.

Repeated measurements of the same quantity are expected to yield results which are clustered around a particular value. If all major sources of errors have been taken into account, it is assumed that the remaining error must be the result of a large number of very small additive effects, and hence normal. Deviations from normality are interpreted as indications of systematic errors which have not been taken into account. Whether this assumption is valid is debatable.

[edit] Physical characteristics of biological specimens

The sizes of full-grown animals is approximately lognormal. The evidence and an explanation based on models of growth was first published in the 1932 book Problems of Relative Growth by Julian Huxley.

Differences in size due to sexual dimorphism, or other polymorphisms like the worker/soldier/queen division in social insects, further make the distribution of sizes deviate from lognormality.

The assumption that linear size of biological specimens is normal (rather than lognormal) leads to a non-normal distribution of weight (since weight or volume is roughly proportional to the 2nd or 3rd power of length, and Gaussian distributions are only preserved by linear transformations), and conversely assuming that weight is normal leads to non-normal lengths. This is a problem, because there is no a priori reason why one of length, or body mass, and not the other, should be normally distributed. Lognormal distributions, on the other hand, are preserved by powers so the "problem" goes away if lognormality is assumed.

On the other hand, there are some biological measures where normality is assumed, such as blood pressure of adult humans. This is supposed to be normally distributed, but only after separating males and females into different populations (each of which is normally distributed).


[edit] Financial variables

Because of the exponential nature of inflation, financial indicators such as stock values, or commodity prices make good examples of multiplicative behavior. As such, periodic changes in them (for example, yearly changes) should not be expected to be normal, but perhaps lognormal. This was the theory proposed in 1900 by Louis Bachelier. However, Benoît Mandelbrot, the popularizer of fractals, showed that even the assumption of lognormality is flawed--the changes in logarithm over short periods (such as a day) are approximated well by distributions that do not have a finite variance, and therefore the central limit theorem does not apply. Rather, the sum of many such changes gives log-Levy distributions.

[edit] Distribution in testing and intelligence

Sometimes, the difficulty and number of questions on an IQ test is selected in order to yield normal distributed results. Or else, the raw test scores are converted to IQ values by fitting them to the normal distribution. In either case, it is the deliberate result of test construction or score interpretation that leads to IQ scores being normally distributed for the majority of the population. However, the question whether intelligence itself is normally distributed is more involved, because intelligence is a latent variable, therefore its distribution cannot be observed directly.

[edit] Numerical approximations of the normal distribution and its CDF

The normal distribution is widely used in scientific and statistical computing. Therefore, it has been implemented in various ways.

The GNU Scientific Library calculates values of the standard normal CDF using piecewise approximations by rational functions. Another approximation method uses third-degree polynomials on intervals [1].

Generation of deviates from the unit normal is normally done using the Box-Muller method of choosing an angle uniformly and a radius exponential and then transforming to (normally distributed) x and y coordinates. If log, cos or sin are expensive then a simple alternative is to simply sum 12 uniform [−1/2, 1/2] deviates. This is equivalent to a twelfth-order polynomial approximation to the normal distribution and is quite usable in many applications.

A method that is much faster than the Box-Muller transform but which is still exact is the so called Ziggurat algorithm developed by George Marsaglia. In about 97% of all cases it uses only two random numbers, one random integer and one random uniform, one multiplication and an if-test. Only in 3% of the cases where the combination of those two falls outside the "core of the ziggurat" a kind of rejection sampling using logarithms, exponentials and more uniform random numbers has to be employed.

There is also some investigation into the connection between the fast Hadamard transform and the normal distribution since the transform employs just addition and subtraction and by the central limit theorem random numbers from almost any distribution will be transformed into the normal distribution. In this regard a series of Hadamard transforms can be combined with random permutations to turn arbitrary data sets into a normally distributed data.

In Microsoft Excel the function NORMSINV() calculates the cdf of the standard normal distribution.

[edit] Trivia

* The last series of the 10 Deutsche Mark banknotes featured Carl Friedrich Gauss and a graph and formula of the normal probability density function.

[edit] See also

* Behrens-Fisher problem
* Data transformation (statistics) - simple techniques to transform data into normal distribution
* Erdős-Kac theorem, on the occurrence of the normal distribution in number theory
* Gaussian blur, convolution using the normal distribution as a kernel
* Gaussian function
* Iannis Xenakis, Gaussian distribution in music.
* Lognormal distribution
* Multivariate normal distribution
* Matrix normal distribution
* Normal-gamma distribution
* Normally distributed and uncorrelated does not imply independent (an example of two normally distributed uncorrelated random variables that are not independent; this cannot happen in the presence of joint normality)
* Probit function
* Sample size
* Student's t-distribution
* Gaussian process
o Wiener process
o Brownian bridge
o Ornstein-Uhlenbeck process
* Inverse Gaussian distribution

[edit] References

* John Aldrich. Earliest Uses of Symbols in Probability and Statistics. Electronic document, retrieved March 20, 2005. (See "Symbols associated with the Normal Distribution".)
* Abraham de Moivre (1738). The Doctrine of Chances.
* Stephen Jay Gould (1981). The Mismeasure of Man. First edition. W. W. Norton. ISBN 0-393-01489-4 .
* Havil, 2003. Gamma, Exploring Euler's Constant, Princeton, NJ: Princeton University Press, p. 157.
* R. J. Herrnstein and Charles Murray (1994). The Bell Curve: Intelligence and Class Structure in American Life. Free Press. ISBN 0-02-914673-9 .
* Pierre-Simon Laplace (1812). Analytical Theory of Probabilities.
* Jeff Miller, John Aldrich, et al. Earliest Known Uses of Some of the Words of Mathematics. In particular, the entries for "bell-shaped and bell curve", "normal" (distribution), "Gaussian", and "Error, law of error, theory of errors, etc.". Electronic documents, retrieved December 13, 2005.
* S. M. Stigler (1999). Statistics on the Table, chapter 22. Harvard University Press. (History of the term "normal distribution".)
* Eric W. Weisstein et al. Normal Distribution at MathWorld. Electronic document, retrieved March 20, 2005.
* Marvin Zelen and Norman C. Severo (1964). Probability Functions. Chapter 26 of Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, ed, by Milton Abramowitz and Irene A. Stegun. National Bureau of Standards.

[edit] External links

* Calculating the Cumulative Normal distribution, C++, VBA, sitmo.com
* Java Applet on Normal Distributions
* Interactive Distribution Modeler (incl. Normal Distribution).
* Free Area Under the Normal Curve Calculator from Daniel Soper's Free Statistics Calculators website. Computes the cumulative area under the normal curve (i.e., the cumulative probability), given a z-score.
* PlanetMath: normal random variable
* GNU Scientific Library – Reference Manual – The Gaussian Distribution
* Distribution Calculator – Calculates probabilities and critical values for normal, t, chi-square and F-distribution.
* Public Domain Normal Distribution Table
* Is normal distribution due to Karl Gauss? Euler, his family of gamma functions, and place in history of statistics
* Maxwell demons: Simulating probability distributions with functions of propositional calculus
* Normal distribution table
* An algorithm for computing the inverse normal cumulative distribution function by Peter J. Acklam – has examples for several programming languages
* [2]Intuitive derivation.

Image:Bvn-small.png Probability distributions [ view • talk • edit ]
Univariate Multivariate
Discrete: Benford • Bernoulli • binomial • Boltzmann • categorical • compound Poisson • discrete phase-type • degenerate • Gauss-Kuzmin • geometric • hypergeometric • logarithmic • negative binomial • parabolic fractal • Poisson • Rademacher • Skellam • uniform • Yule-Simon • zeta • Zipf • Zipf-Mandelbrot Ewens • multinomial • multivariate Polya
Continuous: Beta • Beta prime • Cauchy • chi-square • Dirac delta function • Coxian • Erlang • exponential • exponential power • F • fading • Fermi-Dirac • Fisher's z • Fisher-Tippett • Gamma • generalized extreme value • generalized hyperbolic • generalized inverse Gaussian • Half-Logistic • Hotelling's T-square • hyperbolic secant • hyper-exponential • hypoexponential • inverse chi-square (scaled inverse chi-square) • inverse Gaussian • inverse gamma (scaled inverse gamma) • Kumaraswamy • Landau • Laplace • Lévy • Lévy skew alpha-stable • logistic • log-normal • Maxwell-Boltzmann • Maxwell speed • Nakagami • normal (Gaussian) • normal-gamma • normal inverse Gaussian • Pareto • Pearson • phase-type • polar • raised cosine • Rayleigh • relativistic Breit-Wigner • Rice • shifted Gompertz • Student's t • triangular • truncated normal • type-1 Gumbel • type-2 Gumbel • uniform • Variance-Gamma • Voigt • von Mises • Weibull • Wigner semicircle • Wilks' lambda Dirichlet • Generalized Dirichlet distribution . inverse-Wishart • Kent • matrix normal • multivariate normal • multivariate Student • von Mises-Fisher • Wigner quasi • Wishart
Miscellaneous: Cantor • conditional • equilibrium • exponential family • infinitely divisible • location-scale family • marginal • maximum entropy • posterior • prior • quasi • sampling • singular
Retrieved from "http://en.wikipedia.org/wiki/Normal_distribution"

Categories: Articles with sections needing expansion | Continuous distributions
Views

* Article
* Discussion
* Edit this page
* History

Personal tools

* Sign in / create account

Navigation

* Main page
* Contents
* Featured content
* Current events
* Random article

interaction

* About Wikipedia
* Community portal
* Recent changes
* Contact Wikipedia
* Donate to Wikipedia
* Help

Search

Toolbox

* What links here
* Related changes
* Upload file
* Special pages
* Printable version
* Permanent link
* Cite this article

In other languages

* العربية
* Česky
* Cymraeg
* Català
* Dansk
* Deutsch
* Español
* Esperanto
* فارسی
* Français
* Galego
* 한국어
* Hrvatski
* Bahasa Indonesia
* Íslenska
* Italiano
* עברית
* Latviešu
* Lietuvių
* Magyar
* Nederlands
* 日本語
* Polski
* Português
* Русский
* Simple English
* Basa Sunda
* Suomi
* Svenska
* Tiếng Việt
* Українська
* 中文

Powered by MediaWiki
Wikimedia Foundation

* This page was last modified 22:37, 23 August 2007.
* All text is available under the terms of the GNU Free Documentation License. (See Copyrights for details.)
Wikipedia® is a registered trademark of the Wikimedia Foundation, Inc., a US-registered 501(c)(3) tax-deductible nonprofit charity.
* Privacy policy
* About Wikipedia
* Disclaimers