Elements of information theory

URL

Notes

draw a sequence of n independent and identically distributed (i.i.d.) random variables, we will show that the probability of a “typical” sequence is about 2−nH(X) and that there are about 2nH(X) such typical sequences (32)

Entropy is the uncertainty of a single random variable (32)

The reduction in uncertainty due to another random variable is called the mutual information (32)

Entropy H(X) - Conditional Entropy H(X|Y)

mutual information I (X; Y ) is a measure of the dependence between the two random variables (33)

equal to zero if and only if X and Y are independent (33)

X be a discrete random variable (39)

alphabet X (39)

probability mass function p(x) = Pr{X = x}, x ∈ X (39)

entropy H (X) of a discrete random variable X (40)

H (p) for the above quantity (40)

log is to the base 2 (40)

entropy is expressed in bits (40)

base of the logarithm is b, we denote the entropy as Hb(X) (40)

base of the logarithm is e, the entropy is measured in nats (40)

entropy is a functional of the distribution of X (40)