You Can’t Always Get What You Want Some Considerations Regarding Conditional Probabilities

Abstract

The standard treatment of conditional probability leaves conditional probability undefined when the conditioning proposition has zero probability. Nonetheless, some find the option of extending the scope of conditional probability to include zero-probability conditions attractive or even compelling. This article reviews some of the pitfalls associated with this move, and concludes that, for the most part, probabilities conditional on zero-probability propositions are more trouble than they are worth.

Appendices

Appendix 1 Terminology

1.1 Probability Spaces

For any set $S$, an algebra of subsets of $S$ is a set of subsets of $S$ that contains $S$ and is closed under complementation and unions. A $\sigma$ -algebra of subsets of $S$ is an algebra that is closed under countable unions. For the real line $\mathbb {R}$, we define the Borel sets as the smallest $\sigma$-algebra containing all open intervals.

If $\mathcal {A}$ is an algebra of subsets of $S$, a function $P: \mathcal {A} \rightarrow \mathbb {R}$ is additive iff, for any disjoint $A, B \in \mathcal {A}$,

$$\begin{aligned} P(A \cup B) = P(A) + P(B). \end{aligned}$$

If $\mathcal {A}$ is a $\sigma$-algebra of subsets of $S$, a function $P: \mathcal {A} \rightarrow \mathbb {R}$ is countably additive iff, for any sequence $\{A_i\}$ of disjoint sets in $\mathcal {A}$,

$$\begin{aligned} P\left( \bigcup _{i=1}^\infty A_i\right) = \sum _{i = 1}^\infty P(A_i). \end{aligned}$$

A probability space is a triple $\langle S, \mathcal {A}, P \rangle$, where $S$ is a set, to be thought of as the set of elementary events, $\mathcal {A}$ is an algebra of subsets of $S$, which are the sets of events (propositions) to which probabilities will be ascribed, and $P: \mathcal {A} \rightarrow \mathbb {R}$ is a probability function, that is, a positive, additive set function with $P(S) = 1$. Since we will have reasons to consider probability functions that are not countably additive, we depart from tradition in not assuming countable additivity unless explicitly stated. If we require countable additivity, then $\mathcal {A}$ is required to be a $\sigma$-algebra, and we will refer to $P$ as a probability measure.

If $\langle S, \mathcal {A}, P \rangle$ is a probability space, a random variable is a measurable function $X: S \rightarrow \mathbb {R}$, that is, a function such that, for any Borel set $B$, the set

$$\begin{aligned} X^{-1}(B) = \{ \omega \in S \, | \, X(\omega ) \in B \} \end{aligned}$$

is in $\mathcal {A}$. A random variable $X$ generates a subalgebra of $\mathcal {A}$, called $\sigma (X)$, which is the set of all $X^{-1}(B)$, as $B$ ranges over Borel subsets of the real line.

1.2 Conditional Probability Spaces

Following Rényi (1955, 2007b),¹⁹ we define a conditional probability space as a quadruplet $\langle S, \mathcal {A}, \mathcal {B}, P \rangle$, where $S$ is a set of events, $\mathcal {A}$ an algebra of subsets of $S,\,\mathcal {B}$ a subset of $\mathcal {A}$, to be thought of as the set of events on which we may conditionalize, and $P$ is a function that takes $B \in \mathcal {B}$ to a function $P_B: \mathcal {A}_B \rightarrow \mathbb {R}$, where, for each $B,\,\mathcal {A}_B$ is a subalgebra of $\mathcal {A}$, and

• (i) For each $B \in \mathcal {B}$

  • (a) $P_B(A) \ge 0$ for all $A \in \mathcal {A}_B$.

  • (b) For all $A \in \mathcal {A}$, if $B \subseteq A$, then $A \in \mathcal {A}_B$ and $P_B(A) = 1$.

  • (c) For disjoint $A, A' \in \mathcal {A}_B,\,P_B(A \cup A') = P_B(A) + P_B(A')$.

• (ii) For all $B, C \in \mathcal {B}$ and $A, B \in \mathcal {A}_C$, if $BC \in \mathcal {B}$ then

  $$\begin{aligned} P_C(AB) = P_{BC}(A) \, P_C(B), \end{aligned}$$

  provided that $B \in \mathcal {A}_C$ and $A \in \mathcal {A}_{BC}$.

A conditional probability space can be thought of as a family of probability spaces $\{ \langle S, \mathcal {A}_B, P_B \rangle \; | \; B \in \mathcal {B} \}$, required to mesh with each other via (ii).

It is an immediate consequence of (ii) that, for any $C \in \mathcal {B}$ and $B \subseteq C$, if $B \in \mathcal {A}_C$ and $P_C(B) > 0$, then, for all $A \in \mathcal {A}_C$,

$$\begin{aligned} P_{B}(A) = \frac{P_C(AB)}{P_C(B)} \end{aligned}$$

(51)

provided that $B \in \mathcal {B}$ and $A \in \mathcal {A}_{B}$. This allows us to define probabilities conditional on $B$, provided they don’t clash with those yielded by some other $D \in \mathcal {B}$ such that $B \subseteq D,\,B \in \mathcal {A}_D$, and $P_D(B) > 0$. For this reason, we will usually assume the further condition,

• (iii) For all $C \in \mathcal {B}$ and $B \subseteq C$, if $B \in \mathcal {A}_C$ and $P_C(B) > 0$, then $B \in \mathcal {B}$ and $\mathcal {A}_C \subseteq \mathcal {A}_{B}$.

Given a probability space $\langle S, \mathcal {A}, P \rangle$, let $\mathcal {A}^*$ be the subset of $\mathcal {A}$ consisting of sets $B$ with $P(B) > 0$. Let $P^*$ be the function that maps $B \in \mathcal {A}^*$ to the probability function $P_B: \mathcal {A} \rightarrow [0, 1]$, given by

$$\begin{aligned} P_B(A) = \frac{P(AB)}{P(B)}. \end{aligned}$$

(52)

Then $\langle S, \mathcal {A}, \mathcal {A}^*, P^* \rangle$ is a conditional probability space, corresponding to the standard choice of having conditional probability defined only when the condition has nonzero probability.

We will say that a probability space $\langle S, \mathcal {A}, P \rangle$ is invariant under a bijection $\mathsf {T}: S \rightarrow S$ if and only if $\mathsf {T}$ leaves the set $\mathcal {A}$ of measurable sets invariant, and, for all $A \in \mathcal {A},\,P(\mathsf {T}(A)) = P(A)$, where $\mathsf {T}(A)$ is $\{\mathsf {T}(x) \, | \, x \in A \}$. Similarly, a conditional probability space $\langle S, \mathcal {A}, \mathcal {B}, P \rangle$ is invariant under $\mathsf {T}$ if and only if $\mathsf {T}(\mathcal {A}) = \mathcal {A},\,\mathsf {T}(\mathcal {B}) = \mathcal {B}$, and, for all $B \in \mathcal {B},\,\mathcal {A}_{\mathsf {T}(B)} = \mathsf {T}(\mathcal {A}_B)$ and $P_{\mathsf {T}(B)}(\mathsf {T}(A)) = P_B(A)$ for all $A \in \mathcal {A}_B$.

1.3 Lebesgue Measure

Consider the unit interval $I = (0, 1]$. Let $\mathcal {B}$ be the the smallest $\sigma$-algebra that contains all intervals $(a, b]$. These are the Borel sets. Extension to the entire real line, or to $\mathbb {R}^n$, is straightforward; the Borel subsets of $\mathbb {R}^n$ are the elements of the smallest $\sigma$-algebra containing all rectangles of the form

$$\begin{aligned} \{ (x_1, \ldots , x_n) \; | \; x_i \in (a_i, b_i], i = 1, \ldots , n \}. \end{aligned}$$

The uniform measure, or Lebesgue measure, on $\langle I, \mathcal {B} \rangle$ is the unique countably additive measure that assigns measure $b - a$ to each interval $(a, b]$. Call this measure $\lambda$. Though the Borel sets include all sets of interest for most purposes, we can extend our measure to a wider $\sigma$-algebra, called the Lebesgue measurable sets. We define an outer measure $P^*$, defined for any subset $A$ of the unit interval, by

$$\begin{aligned} P^*(A) = \inf \sum _i \lambda (A_i), \end{aligned}$$

(53)

where the infimum is taken over all countable collections $\{ A_i \}$ of intervals such that $A \subseteq \bigcup _i A _i$. The set $\mathcal {L}$ of Lebesgue measurable sets consists of all sets $A \subseteq I$ such that

$$\begin{aligned} P^*(AE) + P^*(A^c E) = P^*(E) \end{aligned}$$

(54)

for all $E \subseteq I$, where $A^c$ is the complement of $A$ in $I$. It can be shown that this is a $\sigma$-algebra, and that $\lambda$ has a unique countably additive extension to $\mathcal {L}$; this extension is also called Lebesgue measure. Again, extension of this concept to $\mathbb {R}^n$ is straightforward.

It can be shown that, if $A$ is a Lebesgue-measurable subset of $\mathbb {R}^n$ with Lebesgue measure zero, then every subset of $A$ is Lebesgue measurable (and, of course, also has Lebesgue measure zero).

Appendix 2 A Rotationally Invariant Conditional Probability Space

On the $n$-sphere $S^n$ (that is, the $n$-dimensional space of points at unit distance from a fixed point in $n+1$-dimensional Euclidean space, which is a circle for $n=1$, and the surface of a sphere for $n=2$), we construct a uniform spherical measure $\sigma _n$. One way to characterize these measures is in terms of Lebesgue measure on the ambient $(n+1)$-dimensional space. A subset of $A$ of $S_n$ is $\sigma _n$-measurable if and only the wedge subtended by $A$—that is, the set of points on straight lines between points of $A$ and the origin—is Lebesgue measurable, and we take the measure of $A$ to be proportional to the Lebesgue measure of the wedge it subtends. Let $\mathcal {L}_n$ be the set of all $\sigma _n$-measurable subsets of $S_n$.

Let $S$ be a $2$-sphere–that is, the surface of a sphere in 3D space—and let $\mathcal {L}_S$ be the set of all $\sigma _2$-measurable subsets of $S$, and let $\sigma _S$ be $\sigma _2$ measure on $\langle S, \mathcal {L}_S \rangle$. Let $\mathcal {C}$ be the set of all circles on the sphere $S$. For each circle $C \in \mathcal {C}$, let $\mathcal {L}_C$ be the set of $\sigma _1$-measurable subsets of $C$, and let $\sigma _C$ be $\sigma _1$ measure on $\langle C, \mathcal {L}_C \rangle$.

We can construct standard conditional probability spaces $\langle S, \mathcal {L}_S, \mathcal {L}_S^*, P_S^* \rangle$ and $\langle C, \mathcal {L}_C, \mathcal {L}_C^*, P_{C}^* \rangle$, where, as in 1.2, $\mathcal {L}_S^*$ is the set of sets in $\mathcal {L}_S$ with positive probability. We want to extend $\langle S, \mathcal {L}_S, \mathcal {L}_S^*, P^*_S \rangle$ to encompass, in our set of conditions, other sets, including, at minimum, all circles, in such a way that conditionalizing on any circle yields uniform probabilities on that circle

Let us take $\mathcal {B}$ to be

$$\begin{aligned} \mathcal {B} = \mathcal {L}_S^* \cup \bigcup _{C \in \mathcal {C}} \mathcal {L}^*_C. \end{aligned}$$

(55)

Note that each element of $\mathcal {B}$ is either in $\mathcal {L}_S^*$ or is a subset of a unique circle $C$ (this is because the members of $\mathcal {L}^*_C$ have positive measure, and hence contain infinitely many points, and no two circles share more than two points). Take $\mathcal {A}_B$ to be $\mathcal {L}_S$ for $B \in \mathcal {L}_S^*$. For $B \in \mathcal {L}^*_C$, take $A \in \mathcal {A}_B$ iff $AB \in \mathcal {L}_C$. Define

$$\begin{aligned} P_B(A) = \left\{ \begin{array}{lll} \frac{\lambda _S(AB)}{\lambda _S(B)}, &{} &{} B \in \mathcal {L}_S^*; \\ \\ \frac{\lambda _C(AB)}{\lambda _C(B)}, &{} &{} B \in \mathcal {L}_C^*. \end{array} \right. \end{aligned}$$

(56)

We have constructed a conditional probability space that is invariant under all rigid rotations, and includes conditionalization on circles and some subsets of circles.

For any $A \in \mathcal {L}_S$ with $\lambda _S(A) = 0$, every subset of $A$ is a measurable set, and is assigned measure 0. Since each circle $C$ has $\lambda _S(C) = 0$, this means that every subset of $C$ is in $\mathcal {L}_S$. Since not every subset of $C$ is in $\mathcal {L}_C,\,P_C(A)$ is not defined for arbitrary $A \in \mathcal {L}_S$. We might want to extend $P_C$ so that it is defined on all $A \in \mathcal {L}_S$. But, as already mentioned (see footnote 2), we can do so, and preserve rotational invariance, only at the price of sacrificing countable additivity. We can’t get all that we want.

Appendix 3 The Eternal Coin: Proof of Dorr’s Theorem

We will speak in general terms, but readers should think of the example at hand, that of the Eternal Coin. We assume Axiom (iii) of Appendix 1.2. In this appendix, we will find ourselves conditionalizing on complex propositions, and it will be convenient to shift from the subscript notation $Pr_C$ for probabilities conditional on conditional probabilities to the notation $Pr( \; \cdot \; | C)$.

Suppose there is a proposition $P$, and a transformation $\mathsf {T}$, such that $\mathsf {T}(P) \models P$. If $Pr(P) > 0$, then

$$\begin{aligned} Pr(\mathsf {T}(P) \, | \, P) = \frac{Pr(\mathsf {T}(P))}{Pr(P)}, \end{aligned}$$

(57)

and so $\mathsf {T}$-invariance would entail that $Pr(\mathsf {T}(P) \,|\, P) = 1$. Furthermore, if there exists a proposition $Z$ such that $\mathsf {T}(Z) = Z,\,P \models Z$, and $Pr(P | Z) > 0$, then

$$\begin{aligned} Pr(\mathsf {T}(P) \, | \, P) = \frac{Pr(\mathsf {T}(P) \, | \, Z)}{Pr(P \, | \, Z)}, \end{aligned}$$

(58)

and so, once again, $\mathsf {T}$-invariance would entail that $P(\mathsf {T}(P) \, |\,P) = 1$.

But$\mathsf {T}$-invariant propositions of the right sort may be hard to come by, and there may be no such $Z$. Suppose, however, that there exist propositions $X,\,Z$, such that $P \models X \models Z$, and $\mathsf {T}(P) \models \mathsf {T}(X) \models Z$. Then, if $P,\,\mathsf {T}(P),\,X$, and $\mathsf {T}(X)$ are all in $\mathcal {A}_Z$, and $Pr(P|Z) > 0$, and if $Pr(\mathsf {T}(P) \, | \mathsf {T}(X))$ is defined, we have

$$\begin{aligned} \begin{array}{l} Pr(P \,| \,Z) = Pr(P \,| \,X) \, Pr(X \,| \,Z); \\ \\ Pr(\mathsf {T}(P)\,|\, Z) = Pr(\mathsf {T}(P) \, | \mathsf {T}(X)) \, Pr(\mathsf {T}(X) \, |\, Z), \end{array} \end{aligned}$$

(59)

and so, still assuming that $\mathsf {T}(P) \models P$, we have

$$\begin{aligned} Pr(\mathsf {T}(P) \,| \,P) = \frac{Pr(\mathsf {T}(P) \, | \, Z)}{Pr(P \, | \, Z)} = \frac{Pr(\mathsf {T}(P)\,|\,\mathsf {T}(X))}{Pr(P\,|\,X)} \; \frac{Pr(\mathsf {T}(X)\,|\,Z)}{Pr(X\,|\,Z)}. \end{aligned}$$

(60)

Now suppose that there is also a proposition $F$ such that $\mathsf {T}^{-1}(F) \models F$, with $Pr(F|Z) > 0$. Suppose, also, that $\mathsf {T}^{-1}(F) \models X$. From this it follows that $F \models \mathsf {T}(X)$, and we have

$$\begin{aligned} Pr(\mathsf {T}^{-1}(F)\,| \,F) = \frac{Pr(\mathsf {T}^{-1}(F) \,| \, X)}{Pr(F \,| \, \mathsf {T}(X))} \; \frac{Pr(X\, | \,Z)}{Pr(\mathsf {T}(X) \,| \,Z)}. \end{aligned}$$

(61)

Multiplying (60) and (61) gives us,

$$\begin{aligned} Pr(\mathsf {T}(P) \, | \,P) \, Pr(\mathsf {T}^{-1}(F)\,| \,F) = \frac{Pr(\mathsf {T}(P) \,| \, \mathsf {T}(X))}{Pr(P \,| \,X)} \; \frac{Pr(\mathsf {T}^{-1}(F) \,| \,X)}{Pr(F \,| \, \mathsf {T}(X)}. \end{aligned}$$

(62)

So far, we haven’t invoked any symmetry assumptions. If we impose $\mathsf {T}$-invariance, we have

$$\begin{aligned} \begin{array}{l} Pr(\mathsf {T}(P) \,| \, \mathsf {T} (X)) = Pr(P \,| \, X); \\ Pr(\mathsf {T}^{-1}(F) \, | \,X) = Pr(F \,| \, \mathsf {T}(X)), \end{array} \end{aligned}$$

(63)

and (62) becomes

$$\begin{aligned} Pr(\mathsf {T}(P) \, | \,P) \; Pr(\mathsf {T}^{-1}(F) \,| \,F) = 1, \end{aligned}$$

(64)

from which it follows that

$$\begin{aligned} Pr(\mathsf {T}(P) \, | \, P) = Pr(\mathsf {T}^{-1}(F) \,| \,F) = 1. \end{aligned}$$

(65)

Now, since we have assumed that $\mathsf {T}(P) \models P$ and $\mathsf {T}^{-1}(F) \models F$, there always do exist $Z, X$ satisfying the conditions stipulated. Take $Z$ to be $P \vee F$, and take $X$ to be $P \vee \mathsf {T}^{-1}(F)$. Then $\mathsf {T}(X)$ is $\mathsf {T}(P) \vee F$.

To sum up: we have established

Proposition 1

Let $\langle \Omega , \mathcal {A}, \mathcal {B}, Pr \rangle$ be a conditional probability space that satisfies condition (iii) and is invariant under a transformation $\mathsf {T}$. Suppose there are propositions $P, F$, such that $Z = P \vee F \in \mathcal {B}$ and $P, F \in \mathcal {A}_{Z}$, such that

• (i)

  • (a) $\mathsf {T}(P) \models P$;

  • (b) $\mathsf {T}^{-1}(F) \models F$;

• (ii)

  • (a) $Pr(P|Z) > 0$;

  • (b) $Pr(F|Z) > 0$.

Then

$$\begin{aligned} Pr(\mathsf {T}(P) \,| \,P) = Pr(\mathsf {T}^{-1}(F) \,| \,F) = 1. \end{aligned}$$

Applied to the Eternal Coin, let $\mathsf {T}$ be $\mathsf {S}_1$, which shifts everything forward 1 day. $P$, as before, is the proposition that the coin handed Heads every day in the past, and $F$, the proposition that the coin will land Heads every day in the future. Let $H$ be the proposition that the coin lands Heads today. Then $\mathsf {S}_1(P)$ is $HP$, and $\mathsf {S}_1^{-1}(F)$ is $HF$. Clearly, $HP \models P$ and $HF \models F$. If

$$\begin{aligned} \begin{array}{l} Pr(P | P \vee F) > 0; \\ Pr(F | P \vee F) > 0; \end{array} \end{aligned}$$

(66)

and if

$$\begin{aligned} \begin{array}{l} Pr(HP | HP \vee F) = Pr( P | P \vee HF); \\ Pr(F | HP \vee F) = Pr( HF | P \vee HF), \end{array} \end{aligned}$$

(67)

then

$$\begin{aligned} Pr(H|P) = Pr(H|F) = 1. \end{aligned}$$

(68)

We can run the same argument with $\mathsf {S}_k$, for any positive $k$, yielding the conclusion that, for every $n \ge 0$, the probability conditional on $P$ that the coin lands Heads today and $n$ days into the future is 1, as is the probability, conditional on $F$, that the coin lands Heads today and $n$ days into the past.