We use MathJax
Many situations arise where a random variable can be defined in terms of the sum of other random variables. The most important of these situations is the estimation of a population mean from a sample mean. Therefore, we need some results about the properties of sums of random variables.
For any two random variables $X$ and $Y$, the expected value of the sum of those variables will be equal to the sum of their expected values.
| $E(X + Y) = E(X) + E(Y)$ |
The proof, for both the discrete and continuous cases, is rather straightforward. The only essential observations are that the order of the summations (or integrals) can be swapped, and that marginal functions occur midway through the proof.
\begin{align} E(X + Y) &= \sum\limits_x \sum\limits_y (x+y) P_{XY}(x,y) \\ &= \sum\limits_x \sum\limits_y x P_{XY}(x,y) + \sum\limits_y \sum\limits_x y P_{XY}(x,y) \\ &= \sum\limits_x x P_X (x) + y P_Y (y) \\ &= E(X) + E(Y) \end{align} \begin{align} E(X + Y) &= \int_x \int_y (x+y) f_{XY}(x,y) \, \mathrm{d}x \\ &= \int_x \int_y x f_{XY}(x,y) \, \mathrm{d}y \mathrm{d}x + \int_y \int_x y f_{XY}(x,y) \, \mathrm{d}x \mathrm{d}y \\ &= \int_x x f_X (x) \, \mathrm{d}x + \int_y y f_Y (y) \, \mathrm{d}y \\ &= E(X) + E(Y) \end{align}As an example, suppose we have a random variable $Z$ which is the sum of two other random variables $X$ and $Y$. The first has mean $E(X) = 17$ and the second has mean $E(Y) = 24$. Since $Z = X + Y$, then the mean of $Z$ is $E(Z) = 24+17 = 41$. The actual shape of each distribution is irrelevant.
For any two random variables $X$ and $Y$, the variance of the sum of those variables is equal to the sum of the variances plus twice the covariance.
| $Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X,Y)$ |
The proof of this statement is similar to the proof of the expected value of a sum of random variables, but since variance is involved, there are a few more details that need attention.
\begin{align} &Var(X + Y) \\ &= \sum\limits_x \sum\limits_y (x+y)^2 P_{XY}(x,y) - (E(X+Y))^2 \\ &= \sum\limits_x \sum\limits_y x^2 P_{XY}(x,y) + \sum\limits_x \sum\limits_y 2xy P_{XY}(x,y) + \sum\limits_y \sum\limits_x y^2 P_{XY}(x,y) - (E(X))^2 - 2E(X)E(Y) - (E(Y))^2 \\ &= \sum\limits_x x^2 P_X (x) - (E(X))^2 + \sum\limits_y y^2 P_Y(y) - (E(Y))^2 + \sum\limits_x \sum\limits_y 2xy P_{XY}(x,y) - 2E(X)E(Y) \\ &= E(X^2) - (E(X))^2 + E(Y^2) - (E(Y))^2 + 2 (E(XY) - E(X)E(Y) ) \\ &= Var(X) + Var(Y) + 2 Cov(X,Y) \end{align}And now we do the continuous case.
\begin{align} &Var(X + Y) \\ &= \int_x \int_y (x+y)^2 f_{XY}(x,y) - (E(X+Y))^2 \\ &= \int_x \int_y x^2 f_{XY}(x,y) + \int_x \int_y 2xy f_{XY}(x,y) + \int_y \int_x y^2 f_{XY}(x,y) - (E(X))^2 - 2E(X)E(Y) - (E(Y))^2 \\ &= \int_x x^2 f_X (x) - (E(X))^2 + \int_y y^2 f_Y(y) - (E(Y))^2 + \int_x \int_y 2xy f_{XY}(x,y) - 2E(X)E(Y) \\ &= E(X^2) - (E(X))^2 + E(Y^2) - (E(Y))^2 + 2 (E(XY) - E(X)E(Y) ) \\ &= Var(X) + Var(Y) + 2 Cov(X,Y) \end{align}If the random variables are independent, then a simpler result occurs.
| If $X$ and $Y$ are independent, then $Var(X + Y) = Var(X) + Var(Y)$ $\sigma_{X+Y} = \sqrt{ \sigma_X^2 + \sigma_Y^2 }$ |
As an example, if two independent random variables have standard deviations of 7 and 11, then the standard deviation of the sum of the variables would be $\sqrt{7^2 + 11^2} = \sqrt{170} \approx 13.4$.
An immediate special case is the random variable $\bar{X}$, which is the sample mean of a collection of random variables $X_i$. When a sample is created by random selections of the data values, the random variables will be independent. Since the sample data all comes from the same population, the random variables will be identical. Then by the various expected value properties, we have:
| $E(\bar{X}) = E \left( \dfrac{ \sum\limits_{i=1}^n X_i }{n} \right) = \dfrac1n \sum\limits_{i=1}^n E(X) = \dfrac1n (n E(X)) = E(X)$ |
Also, by the variance properties, we have:
| $Var(\bar{X}) = Var \left( \dfrac{ \sum\limits_{i=1}^n X_i }{n} \right) = \dfrac{1}{n^2} \sum\limits_{i=1}^n Var(X) = \dfrac{1}{n^2} (n Var(X)) = \dfrac{Var(X)}{n}$ |
Therefore, we have the following results for the mean and standard deviation of a distribution of sample means. The standard deviation of the sample means is often called the standard error of the mean.
| $\mu_{\bar{x}} = \mu$ |
| $\sigma_{\bar{x}} = \dfrac{\sigma_x}{\sqrt{n}}$ |
We also consider the moment generating function of two independent random variables.
| If $X$ and $Y$ are independent, then $M_{X + Y} (t) = M_X (t) M_Y (t)$ |
Since the random variables are independent, we can also use the simpler form of the product rule for probabilities, $P(x,y) = P(x) P(y)$ in our proof of this statement.
| \begin{align} M_{X + Y}(t) &= E(e^{t(X+Y)}) \\ &= \sum\limits_x \sum\limits_y e^{t(x+y)} P_{XY} (x,y) \\ &= \sum\limits_x e^{tx} P_X (x) \sum\limits_y e^{ty} P_Y (y) \\ &= E(e^{tx}) E(e^{ty}) \\ &= M_X (t) M_Y (t) \end{align} | \begin{align} M_{X + Y}(t) &= E(e^{t(X+Y)}) \\ &= \int_x \int_y e^{t(x+y)} f_{XY} (x,y) \\ &= \int_x e^{tx} f_X (x) \int_y e^{ty} f_Y (y) \\ &= E(e^{tx}) E(e^{ty}) \\ &= M_X (t) M_Y (t) \end{align} |
Let $X$ and $Y$ be two random variables, and let the random variable $Z$ be their sum, so that $Z = X+Y$. Then, $F_Z(z)$, the CDF of the variable $Z$, would give the probabilities associated with that random variable. But by the definition of a CDF, $F_Z(z) = P(Z \le z)$, and we know that $z = x+y$. Therefore, we need to consider those $x$ and $y$ values whose sum is less than or equal to $z$, or alternatively, when $y \le z - x$.
Now $X$ and $Y$ will have a joint PDF, which we shall call $f_{XY}(x,y)$. Probabilities for the joint function are found by integrating the PDF, and we are interested in those probabilities for which $y \le z - x$. In other words, we have
\begin{equation} F_Z(z) = P(Z \le z) = \int_{-\infty}^{\infty} \int_{-\infty}^{z-x} f_{XY}(x,y) \,\mathrm{d}y \,\mathrm{d}x \end{equation}By using the substitution $y = v - x$, we can then interchange the order of integration.
\begin{align} F_Z(z) &= \int_{-\infty}^{\infty} \int_{-\infty}^{z} f_{XY}(x,v-x) \,\mathrm{d}v \,\mathrm{d}x \\ &= \int_{-\infty}^z \int_{-\infty}^{\infty} f_{XY}(x, v-x) \,\mathrm{d}x \,\mathrm{d}v \end{align}Taking the derivative of the CDF with respect to $z$, we obtain the following PDF.
| $f_{X+Y}(z) = \int_{-\infty}^{\infty} f_{XY}(x, z-x) \,\mathrm{d}x$ |
When the random variables $X$ and $Y$ are independent, then the joint PDF may be written as a product of the PDFs, and we obtain the following result.
| If $X$ and $Y$ are independent, then $f_{X+Y}(z) = \int_{-\infty}^{\infty} f_X(x) f_Y(z-x) \,\mathrm{d}x$ |
The operation which combines the two functions $f_X$ and $f_Y$ in this fashion is called convolution. In other words, the PDF of the sum of two independent random variables is the convolution of their two PDFs.