Characterizing Messages

Channel Capacity

Note: in order to run the simulation referred to in this slide, go here, to the Java Applet version. You will be directed to download the latest version of the Java plug-in.

Channel capacity is a measure of how quickly information moves across a channel. Information is related to how many possible input messages we have to choose from. In the case of the simple binary channel, we have a choice of two symbols in every interval. This means that we can choose 2n different messages of length n. Shannon calls this a capacity of one bit per time period.

Notice that transmitting two bits in this channel corresponds to a choice of four different messages in two time periods, three bits is a choice of eight messages in three time periods, and so on. Because the number of choices increases exponentially, we might come to the conclusion that capacity should depend on the log of the number of message choices. In fact, Shannon arrives at the following expression for channel capacity:

,

where N is the number of possible messages of length T. In our example, N(T) is simply 2T, so the term inside the limit is always 1. If the base of the logarithm is taken to be two, the resulting capacity is said to be measured in bits. Thus the binary channel has a capacity of 1 bit per interval, as our intuition dictates.

You might wonder if the limit is necessary in the above expression. This is because different symbols may take different times to transmit. Consider the Morse Code. A dot might be represented by a 1 followed by a 0, while a dash may be three 1's followed by a 0. The agents to the left will transmit a simple message using this system. Press "Go" repeatedly to make this happen.

Notice that transmitting a dash takes twice as long as transmitting a dot. In this case, N(T) / T is not constant in T, so we cannot discern the capacity from short sequences. There is only one possible sequence of length 2 and none of length 1. For any meaning, we must consider how many messages are possible as the sequences grow long. The above limit can be calculated using the method of differences to be 0.34 bits per period

Previous Slide                                                           Next Slide