32 Probability

32.1 Elementary Probability Concepts

32.1.1 Definition of Probability

Let $$S$$ be a sample space associated with an experiment. For every event $$A \in S$$ (ie, $$A$$ is a subset of $$S$$), we assign a number, $$P(A)$$ - called the of $$A$$ - such that the following three axioms hold:

• Axiom 1: $$P(A) \geq 0$$.
• Axiom 2: $$P(S) = 1$$.
• Axiom 3: If $$A_1, A_2, A_3, ...$$ form a sequence of pairwise mutually exclusive events in $$S$$ (that is, $$A_i \cap A_j = \emptyset$$ if $$i \neq j$$), then
$$P(A_1 \cup A_2 \cup A_3 \cup ...) = \sum\limits_{i=1}^\infty P(A_i)$$.

32.1.2 Definition: Conditional Probability

The conditional probability of an event $$A$$, given that an event $$B$$ has occured and $$P(B) > 0$$ is equal to

$P(A|B) = \frac{P(A\cap B)}{P(B)}$

32.1.3 Definition: Independence

Events $$A$$ and $$B$$ are said to be independent if any of the following holds

$P(A|B) = P(A)$ $P(B|A) = P(B)$ $P(A\cap B) = P(A)\cdot P(B)$

32.1.4 Theorem: Multiplicative Law of Probability

The probability of the intersection of two events $$A$$ and $$B$$ is

$P(A\cap B) = P(A)\cdot P(B|A) = P(B)\cdot P(A|B)$

Proof:

By the definition of conditional probability

\begin{aligned} P(A|B) &= \frac{P(A\cap B)}{P(B)} \\ \Rightarrow P(A|B) \cdot P(B) &= P(A \cap B) \end{aligned}

Likewise

\begin{aligned} P(B|A) &= \frac{P(B\cap A)}{P(A)} \\ \Rightarrow P(B|A) \cdot P(A) &= P(B \cap A) \end{aligned}

Since $$P(A \cap B) = P(B \cap A)$$

\begin{aligned} P(A|B) \cdot P(B) &= P(A \cap B) \\ &= P(B \cap A) \\ &= P(B|A) \cdot P(A) \\ \Rightarrow P(A \cap B) &= P(A|B) \cdot P(B) \\ &= P(B|A) \cdot P(A) \end{aligned}

32.1.5 Corollary

If $$A$$ and $$B$$ are independent, then

$P(A \cap B) = P(A) \cdot P(B)$

Proof:

When $$A$$ and $$B$$ are independent, by the definition of independence,

\begin{aligned} P(A \cap B) & = P(A|B) \cdot P(B) = P(B|A) \cdot P(A) \\ \Rightarrow &= P(A) \cdot P(B) = P(B) \cdot P(A) \end{aligned}

The probability of the union of two events is $$P(A \cup B) = P(A) + P(B) - P(A \cap B)$$.

Proof:

$$A \cup B = A \cup (A^c \cap B)$$ where $$A$$ and $$(A^c \cap B)$$ are mutually exclusive. $$\Rightarrow P(A \cup B) = P(A) + P(A^c \cap B)$$

$$B = (A^c \cap B) \cup (A \cap B)$$ where $$(A^c \cap B)$$ and $$(A \cap B)$$ are mutually exclusive. $$\Rightarrow P(B) = P(A^c \cap B) + P(A \cap B)\\ \Rightarrow P(A^c \cap B) = P(B) - P(A \cap B)$$

$$P(A \cup B) = P(A) + P(A^c \cap B)\\ \Rightarrow P(A \cup B) = P(A) + P(B) - P(A \cap B)$$

32.1.7 Corollary

If $$A$$ and $$B$$ are mutually exclusive events, then $$P(A \cup B) = P(A) + P(B)$$.

Proof:

When $$A$$ and $$B$$ are mutually exclusive, $$(A \cap B) = \emptyset$$ and $$P(A \cap B) = 0$$. By Theorem ,

\begin{aligned} P(A \cup B) &= P(A) + P(B) - P(A \cap B) \\ \Rightarrow P(A \cup B) &= P(A) + P(B) - 0 \\ \Rightarrow P(A \cup B) &= P(A) + P(B) \end{aligned}

32.1.8 Theorem: Law of Complements

If $$A$$ is an event, then $$P(A) = 1 - P(A^c)$$.

Proof:

Let $$S$$ be the sample space.

\begin{aligned} S &= A \cup A^c \\ \Rightarrow P(S) &= P(A \cup A^c) \\ \Rightarrow P(S) &= P(A) + P(A^c) - P(A \cap A^c) \\ \Rightarrow P(S) &= P(A) + P(A^c) - 0 \\ \Rightarrow P(S) &= P(A) + P(A^c) \\ \Rightarrow 1 &= P(A) + P(A^c) \\ \Rightarrow 1 - P(A^c) &= P(A) \\ \Rightarrow P(A) &= 1 - P(A^c) \end{aligned}

32.1.9 Definition: Partition of a Sample Space

For some positive integer $$k$$, let the sets $$B_1, B_2, \ldots, B_k$$ be such that

• $$S = B_1 \cup B_2 \cup \ldots \cup B_k$$.
• $$B_i \cap B_j = \emptyset$$ for $$i \neq j$$.

Then the collection of sets $${B_1, B_2, \ldots, B_k}$$ is said to be a partition of $$S$$.

32.1.10 Definition: Decomposition

If $$A$$ is any subset of $$S$$ and $${B_1, B_2, \ldots, B_k}$$ is a partition of $$S$$, $$A$$ can be decomposed as follows:

$$A = (A \cap B_1) \cup (A \cap B_2) \cup \cdots \cup (A \cap B_k)$$

32.1.11 Theorem: Total Law of Probability

If $${B_1, B_2, \ldots, B_k}$$ is a partition of $$S$$ such that $$P(B_i) > 0$$, for $$i = 1, 2, \ldots, k$$, then for any event $$A$$

$P(A) = \sum\limits_{i=1}^k P(A|B_i)P(B_i)$

Proof:

Any subset $$A$$ of $$S$$ can be written as

$A = A \cap S = A \cap (B_1 \cup B_2 \cup \cdots \cup B_k) = (A \cap B_1) \cup (A \cap B_2) \cup \cdots \cup (A \cap B_k)$

Since $${B_1, B_2, \ldots, B_k}$$ is a partition of $$S$$, if $$i \neq j$$,

$$(A \cap B_i) \cup (A \cap B_j) = A \cap (B_i \cap B_j) = A \cap \emptyset = \emptyset$$. That is, $$(A \cap B_i)$$ and $$(A \cap B_j)$$ are mutually exclusive events. Thus,

\begin{aligned} P(A) &= P(A \cap B_1) + P(A \cap B_2) + \cdots + P(A \cap B_k) \\ ^{[1]} \Rightarrow &= P(A|B_1)P(B_1) + P(A|B_2)P(B_2) + \cdots + P(A|B_k)P(B_k) \\ \Rightarrow &= \sum\limits_{i=1}^k P(A|B_i)P(B_i) \end{aligned}

1. Theorem : Multiplicative Law of Probability

32.1.12 Theorem: Bayes’ Rule

If $${B_1, B_2, \ldots, B_k}$$ is a partition of $$S$$ such that $$P(B_i) > 0$$, for $$i = 1, 2, \ldots, k$$, then

$P(B_j|A) = \frac{P(A|B_j)P(B_j)}{\sum\limits_{i=1}^k P(A|B_i)P(B_i)}$

Proof:

\begin{aligned} \ ^{[1]} P(B_j|A) &= \frac{P(A \cap B)}{P(A)} \\ ^{[2]} \Rightarrow &= \frac{P(A|B_j)P(B_j)}{P(A)} \\ ^{[3]} \Rightarrow &= \frac{P(A|B_j)P(B_j)}{\sum\limits_{i=1}^k P(A|B_i)P(B_i)} \end{aligned}

1. Definition , Conditional Probability
2. Definition , Conditional Probability
3. Theorem , Law of Total Probability