1. Summary #

To determine the true value $X$ from multiple measurements $x_1, x_2, \cdots, x_N$, the true value can be estimated with uncertainty as follows:

$$ \begin{align} \label{eq1} X \to \bar{x} \pm \Delta\bar{x} \end{align} $$

Here, $\bar{x}$ is the most probable value (or mean) of $x_1, x_2, \cdots, x_N$, and $\Delta\bar{x}$ is the uncertainty in the mean, given by:

$$ \begin{align} \bar{x} &= \frac{1}{N}\sum_{i=1}^N x_i \label{eq2}\\ \Delta\bar{x} &= \frac{\Delta x}{\sqrt{N}}=\frac{1}{{\sqrt{N}}}\sqrt{\frac{1}{N-1}\sum_{i=1}^N (x_i-\bar{x})^2} \label{eq3} \end{align} $$

The meaning of “uncertainty” is that, assuming the measurements deviate purely probabilistically around the true value, the probability that the true value lies within $\bar{x} \pm \Delta\bar{x}$ is approximately 68.3% (the $1\sigma$ range).

From equation \eqref{eq3}, an important property of uncertainty emerges: to reduce the uncertainty by a factor of $1/10$, the total number of data points must be increased by a factor of 100.

2. Explanation #

This discussion follows the Gauss error theory¹², which assumes that the variation in measurements is purely probabilistic. Specifically, measurements are assumed to follow a Gaussian distribution $p(x)$ centered around the true value $X$:

$$ \begin{align} \label{eq4} p(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{1}{2}\left(\frac{x-X}{\sigma}\right)^2\right]} \end{align} $$

$$ \begin{align} \label{eq5} \int_{-\infty}^{\infty} p(x) dx=1 \end{align} $$

Here, $\sigma$ is the standard deviation, and $\sigma^2$ is the variance.

2.1. Derivation of the Most Probable Value $\bar{x}$ #

The most probable value represents the most likely value of the measurement target. We calculate this value from the measurement results based on the probability distribution $p(x)$.

For a system with probability distribution $p(x)$, the probability of obtaining a measurement $x_i$ is $p(x_i)$. The probability of obtaining results $x_1, x_2, \cdots, x_N$ from multiple measurements is the product of their individual probabilities:

$$ \begin{align} \label{eq6} p(x_1)p(x_2)\cdots p(x_N) &\propto \prod_{i=1}^N \exp{\left[-\frac{1}{2}\left(\frac{x_i-X}{\sigma}\right)^2\right]} \\ &=\exp{\left[-\frac{1}{2\sigma^2}\sum_{i=1}^N\left(x_i-X\right)^2\right]} \label{eq7} \end{align} $$

In practice, multiple measurements yield results that maximize the probability in equation \eqref{eq7}. From this assumption, we determine $X$ such that equation \eqref{eq7} is maximized. This corresponds to minimizing:

$$ \begin{align} \label{eq8} \sum_{i=1}^N\left(x_i-X\right)^2 \end{align} $$

The value of $X$ minimizing \eqref{eq8} is adopted as the most probable value of the measurement target. Taking the derivative with respect to $X$ and setting it to zero gives:

$$ \begin{align} \label{eq9} \frac{\partial}{\partial X}\sum_{i=1}^N\left(x_i-X\right)^2 = -2\sum_{i=1}^N\left(x_i-X\right) \\ \sum_{i=1}^N\left(x_i-X\right)\Bigr|_{X=X_0}=0 \end{align} $$

Thus, the most probable value $X = X_0$ satisfies:

$$ \begin{gather} \label{eq10} \sum_{i=1}^N \left(x_i-X_0 \right)=0 \\ \sum_{i=1}^N x_i-N X_0 =0 \\ \to X_0 = \bar{x}\equiv \frac{1}{N}\sum_{i=1}^N x_i \end{gather} $$

This shows that the most probable value $X_0$ can be calculated as the arithmetic mean of the measurements ${x_i}$.

2.2. Derivation of the Uncertainty of the Most Probable Value $\Delta\bar{x}$ #

Next, we consider the uncertainty (variance) of measurements. Since we are working with a Gaussian distribution, it suffices to express the standard deviation $\sigma$ in terms of the measurements ${x_i}$.

The standard deviation $\sigma$ can be computed as follows if the true value $X$ is known:

$$ \begin{align} \label{eq11} \sigma^2 = \frac{1}{N}\sum_{i=1}^N (x_i-X)^2 \end{align} $$

However, since the true value $X$ is unknown, we aim to estimate it under this condition. To simplify the analysis, we consider the variance $\sigma^2$ instead of $\sigma$ itself.

From the prior discussion, the most probable value $X_0$ was shown to be the arithmetic mean $\bar{x}$ of ${x_i}$. Using this, we reformulate the sum in \eqref{eq11} as follows:

$$ \begin{align} \sum_{i=1}^N (x_i-X)^2 &=\sum_{i=1}^N \left[{(x_i-\bar{x})+(\bar{x}-X)}\right]^2 \label{eq13}\\ &= \sum_{i=1}^N (x_i-\bar{x})^2 + \sum_{i=1}^N(\bar{x}-X)^2+ 2(\bar{x}-X)\sum_{i=1}^N (x_i-\bar{x}) \label{eq14}\\ &= \sum_{i=1}^N (x_i-\bar{x})^2 + N(\bar{x}-X)^2 \label{eq15} \\ &= \sum_{i=1}^N (x_i-\bar{x})^2 + \frac{1}{N}\left[\sum_{i=1}^N(x_i-X)\right]^2 \label{eq16} \\ &= \sum_{i=1}^N (x_i-\bar{x})^2 + \frac{1}{N}\sum_{i=1}^N(x_i-X)^2+ \frac{1}{N}{\sum\sum}_{i\ne j}(x_i-X)(x_j-X) \label{eq17}\\ &\approx \sum_{i=1}^N (x_i-\bar{x})^2 + \frac{1}{N}\sum_{i=1}^N(x_i-X)^2 \label{eq18} \end{align} $$

In \eqref{eq14}, the third term vanishes due to the definition of the mean. From \eqref{eq17} to \eqref{eq18}, the term ${\sum\sum}_{i\ne j}(x_i-X)(x_j-X)$ is ignored because it fluctuates around zero due to its probabilistic nature and the independence of different terms.

Numerical experiments illustrate the following:

	Value of each term in equation \eqref{eq17} for a Gaussian distribution with mean $\mu=0$ and variance $\sigma=3$
Overall
Zoomed

The red line in equation \eqref{eq18}, representing the third term, occasionally exceeds the second term. This might suggest that the approximation of neglecting the third term is invalid. However, since the second and third terms ultimately cannot be evaluated without knowing the true value $X$, they should be treated as expectations. Considering the expectations of the second and third terms, the second term has a finite value (approximately 10 in the above figure), while the third term approaches zero. Thus, it is reasonable to neglect the third term in the transition from \eqref{eq17} to \eqref{eq18}.

By solving for $\sigma^2 = \frac{1}{N}\sum_{i=1}^N (x_i-X)^2$ in \eqref{eq18}, we obtain:

$$ \begin{align} \sigma^2&=\frac{1}{N}\sum_{i=1}^N (x_i-X)^2 \label{eq19} \\ &\approx \frac{1}{N-1}\sum_{i=1}^N (x_i-\bar{x})^2 \label{eq20} \end{align} $$

The left-hand side is referred to as the standard deviation, and the right-hand side as the unbiased standard deviation. When the true value is unknown, the calculation should follow the right-hand side of \eqref{eq19}.

Note that $\Delta x = \sigma$, derived from \eqref{eq20}, represents the uncertainty of each measurement $x_i$, not the uncertainty of the most probable value $\bar{x}$. The uncertainty of $\bar{x}$, $\Delta \bar{x}$, is computed as follows:

The mean $\bar{x}$ of ${x_i}$ is restated as:

$$ \begin{align} \bar{x} = \frac{1}{N}\sum_{i=1}^N x_i \label{eq21} \end{align} $$

For a general function $f=f(x,y,z,\cdots)$ with uncertainties $\Delta x, \Delta y, \Delta z, \cdots$ in its arguments $x, y, z, \cdots$, the uncertainty $\Delta f$ is given by:

$$ \begin{align} \Delta f = \sqrt{\left(\frac{\partial f}{\partial x}\right)^2(\Delta x)^2+\left(\frac{\partial f}{\partial y}\right)^2(\Delta y)^2+\left(\frac{\partial f}{\partial z}\right)^2(\Delta z)^2+\cdots} \label{eq22} \end{align} $$

Here, the partial derivatives are evaluated at the most probable values $\bar{x}, \bar{y}, \bar{z}, \cdots$. This result arises from approximating the solution around the most probable value using a linear expansion (Taylor expansion up to first order).

Applying \eqref{eq22} to \eqref{eq21}, the uncertainty of the mean $\Delta \bar{x}$ is calculated as:

$$ \begin{align} \Delta \bar{x} &= \sqrt{\left(\frac{\partial \bar{x}}{\partial x_1}\right)^2(\Delta x_1)^2 + \left(\frac{\partial \bar{x}}{\partial x_2}\right)^2(\Delta x_2)^2+\cdots} \\ &= \sqrt{\frac{1}{N^2}\sigma^2 + \frac{1}{N^2}\sigma^2 + \cdots + } \\ &= \frac{\sigma}{\sqrt{N}}\label{eq23} \end{align} $$

Here, we used $\frac{\partial \bar{x}}{\partial x_j} = \frac{1}{N}$.

2.3. Summary #

From multiple measurements $x_1, x_2, \cdots, x_N$, the most probable value, including its uncertainty, is expressed as:

$$ \begin{align} \bar{x} \pm \Delta\bar{x} \end{align} $$

Here, $\bar{x}$ is the most probable value (or mean), and $\Delta\bar{x}$ is its uncertainty, given as:

$$ \begin{align} \bar{x} &= \frac{1}{N}\sum_{i=1}^N x_i\\ \Delta\bar{x} &= \frac{\Delta x}{\sqrt{N}}=\frac{1}{{\sqrt{N}}}\sqrt{\frac{1}{N-1}\sum_{i=1}^N (x_i-\bar{x})^2} \end{align} $$

The uncertainty indicates that the true value lies within this range with a probability of approximately 68.3% (commonly referred to as the $1\sigma$ range).

3. Numerical Experiment #

3.1. Example #

We generate random numbers following a normal distribution with a mean $\mu=0$ and standard deviation $\sigma=3$, and analyze these data to estimate the most probable value (mean) and its uncertainty. The specific distribution is shown in the figure below:

The above data were generated in Python using the command samples = np.random.normal(mu, sigma, N).

3.2. Case $N \leq 100$ (Step Size 1) #

3.2.1. Deviation and Uncertainty #

Deviation	Uncertainty
Correct calculation is shown as the blue line	$\sigma/\sqrt{N}$, where $\sigma$ is from \eqref{eq20}.

3.2.2. Difference from Standard Deviation #

Comparison of the deviation using the true value (right-hand side of \eqref{eq19}) with deviations computed using $1/N$ (incorrect) and $1/(N-1)$ (correct, \eqref{eq20}):

	Comparison of Differences Between Standard and Unbiased Standard Deviations
	Correct calculation is shown as the blue line (|Unbiased Std Dev - Std Dev|)

Focusing on areas with larger differences, the correct calculation (blue line) is slightly more stable and smaller on average compared to the incorrect calculation (orange line), though the difference is not dramatic.

3.2.3. Most Probable Value with Error Bars #

Including uncertainty as error bars, the results are shown below:

	The probability of the true value falling within the error bars is $68.3%$.

3.3. Case $N \leq 100000$ (Step Size 100) #

3.3.1. Deviation and Uncertainty #

Deviation	Uncertainty
Correct calculation is shown as the blue line (\eqref{eq20})	$\sigma/\sqrt{N}$, where $\sigma$ is from \eqref{eq20}.
	Uncertainty shown on a logarithmic scale:
	Note: A 100-fold increase in data points reduces uncertainty by one order of magnitude, following $1/\sqrt{N}$.

3.3.2. Difference from Standard Deviation #

	Comparison of Differences Between Standard and Unbiased Standard Deviations
	Correct calculation is shown as the blue line (|Unbiased Std Dev - Std Dev|)

3.3.3. Most Probable Value with Error Bars #

Including uncertainty as error bars, the results are shown below:

	The probability of the true value falling within the error bars is $68.3%$.

3.4. Code #

Below is the Python code used to generate the above figures.

import numpy as np
from plot_xy import plot_xy
import matplotlib.pyplot as plt

# Specify mean and variance
mu = 0  # Mean
sigma = 3  # Standard deviation

start = 2
step = 1
count = 998
N_points = [start + i * step for i in range(count)]

np.random.seed(5)

means = []  # \bar{x} = \frac{1}{N}\sum_{i=1}^{N}x_i
biased_deviations = []  # \sqrt{ \frac{1}{N} \sum_{i=1}^{N}(x_i-mu)^2 }
unbiased_deviations = []  # \sqrt{ \frac{1}{N-1} \sum_{i=1}^{N}(x_i-\bar{x})^2 }
uncertainties_of_mean = []  # \frac{1}{\sqrt{N}} \times \sqrt{ \frac{1}{N-1} \sum_{i=1}^{N}(x_i-\bar{x})^2 }
for N in N_points:
    # Generate N random numbers following a normal distribution with mean \mu and standard deviation \sigma.
    samples = np.random.normal(mu, sigma, N)

    # Calculate the mean and unbiased variance of the generated random numbers.
    sample_mean = np.mean(samples)  # Mean
    biased_deviation = np.sqrt(np.mean((samples - mu) ** 2))  # Biased deviation (using true value)
    unbiased_deviation = np.sqrt(np.var(samples, ddof=1))  # Unbiased deviation

    print(f"N:{N}, Sample : {sample_mean:.6f}, {unbiased_deviation:.6f}, {biased_deviation:.6f}")

    means.append(sample_mean)
    biased_deviations.append(biased_deviation)
    unbiased_deviations.append(unbiased_deviation)
    uncertainties_of_mean.append(unbiased_deviation / np.sqrt(N))

# -------------------------------------

# - Show histgram
count, bins, ignored = plt.hist(samples, bins=50, density=True)
# Calculate the center of bins
bin_centers = 0.5 * (bins[1:] + bins[:-1])
# Plot the line connecting the centers of the bins
plt.plot(bin_centers, count, "-")
plt.title("Histogram of Normally Distributed Numbers")
plt.xlabel("Points")
plt.ylabel("Frequency")
plt.legend()
plt.show()
plt.clf()

# - Show mean with error bar
plt.errorbar(N_points, means, yerr=uncertainties_of_mean, elinewidth=1, fmt=".-", ms=3, label="with error", capsize=2, capthick=1)
plt.xlabel("Points")
plt.ylabel("Most probable value")
plt.legend()
plt.show()
plt.clf()

# - Deviation
plt.plot(N_points, biased_deviations, ".-", ms=3, label="biased deviation")
plt.plot(N_points, unbiased_deviations, ".-", ms=3, label="unbiased deviation")
plt.xlabel("Points")
plt.ylabel("Deviation")
plt.legend()
plt.show()
plt.clf()