Mathematics

For background on probability systems, see my previous post. A random variable is really a function, called a measure function, from the sample space to the state space of a probability system. Like , a state space is a -algebra, typically the space of real numbers . In this case a random variable is defined as , for a probability space . The notation will be used as shorthand for the more explicit, well defined .

Any real-valued random variable (e.g. ) can be fully characterized by its cumulative distribution function. The cumulative distribution function, or CDF, is defined for a random variable as . If the CDF is a continuous function, then the random variable is said to be a continuous random variable. This type of CDF is illustrated in Figure 1. For a continuous random variable the probability of taking on any particular value is infinitesimally small (i.e. ), so we talk about the probability of the random variable taking on a value in a particular range instead, . The probability of a random variable taking on a value in a particular range is called the probability density, and is represented by the probability density function, , illustrated in Figure 2. Note that can be estimated by using a sufficiently small interval,

continuous cdf — Figure 1. Continuous CDF

For discrete random variables, the variable takes on particular values over a discrete interval. We can define a function that represents the probability that is exactly equal so to some value , or . The CDF is a step function since increases in discrete steps as shown in Figure 3. The probability that takes on a particular value is given by the probability mass function, illustrated in Figure 4. Notice that the pmf is not continuous, .

Figure 2. Discrete CDF — Figure 3. Discrete CDF

In this post I will explain the basics of the mathematical formulation of probability theory. Firstly, a probability space ties together the ideas of events and the probability of these events occurring. The exact definition and methods of determining these probabilities are disputed (i.e. Frequentist vs. Bayesian), however we will take these as given. A probability space is given by the triple: , where is the sample space, is an -algebra on subsets of , and is a measure which maps the elements of to the interval .

The sample space is the set of all possible outcomes for the system. For example, if we are examining the probability of rolling a die, the sample space is . The subsets of the sample space that contain only one element are said to be the atomic or elementary events. The -algebra over the set is the collection of subsets of with the following properties:

, .
(closed under complementation).
where (closed under the union of countably many sets).

When discussing discrete probability spaces and finite sample spaces (which will usually be the case here), can be expressed generally as , the power set of . Any element of is called an event, which could also be thought of as the union of some elementary events.

Let be a function that defines the probability of each elementary event. In a discrete probability space we can define , the probability of an event is defined as .

Let us examine the example of the probability space of one roll of a six-sided die. Using the following definitions

If we are interested in the event that we roll an even number we can see:

Random Variables and Distributions

Fundamentals of Probability Theory