Math 480, Fall 2016, Assignment 4

We admit, in geometry, not only infinite magnitudes, that is to say, magnitudes greater than any assignable magnitude, but infinite magnitudes infinitely greater, the one than the other. This astonishes our dimension of brains, which is only about six inches long, five broad, and six in depth, in the largest heads.

- Voltaire

Carefully define the following terms, then give one example and one non-example of each:

1. Entropy (of a system of events).
2. Joint entropy (of two systems of events).
3. Conditional entropy (of two systems of events).

Carefully state the following theorems (you need not prove them):

1. Theorem concerning the sign of mutual information.
2. Theorem characterizing when the mutual information is zero.
3. Theorem putting upper and lower bounds on entropy, and characterizing when those bounds are achieved.
4. Identities and inequalities for entropy (all the theorems and corollaries in section 2.4.)

Solve the following problems:

1. Section 2.3, problems 1, 2, 3, and 6.
2. Section 2.4, problems 1, 2, 3, 4, 5, and 6.

--------------------End of assignment--------------------