Luigi Paura » 2.Data Compression

Summary

• Information source model
• Measure of the information: entropy of a discrete random variable
• Entropy of an information source
• Conditional Entropy
• Source Coding Theorem (I Shannon Theorem)
• Huffman Coding
• Lempel-Ziv Coding

Reference : Proakis Salehi (II ed.) Cap.4

Information Source Model

Examples of information sources:

Audio-broadcasting System : speech signal

Video-broadcasting System : video signal

Fax transmission system: monochromatic image

Communication System among computers : ASCII symbol
Sequence or sequence of binary symbols

Information Source Model

Information Source Model (next)

Information Source Model (next)

Example of source coding

Example of source coding (next)

Example of source coding (next)

How much memory amount can we save at most?
I Shannon Theorem gives us the answer
The lowest number of symbols to represent “without distortion” (say, distortion-less or noiseless) every source symbol is given by the average amount of information carried by any source symbol.

Measure of the information

Measure of the information (next)

Measure of the information of a discrete random variable

Measure of the information of a discrete random variable  (next)

Measure of the information of a discrete random variable (next)

Measure of the information of a discrete random variable (next)

Measure of the information of a discrete random variable (next)

Entropy of a pair of RV

Entropy of a pair of RV (next)

Entropy of a pair of RV (next)

Entropy of a vector of RV’s

Conditional Entropy

Conditional Entropy (next)

Conditional Entropy (next)

Conditional Entropy (next)

Conditional Entropy (next)

Proprieties of the Entropy

Information rate of a discrete source

Information rate of a discrete memoryless source (DMS)

Information rate of a discrete memoryless source (DMS)

Measure of the informationof a DMS

Measure of the informationof a DMS  (next)