ECE 534: Elements of Information Theory
University of Illinois at Chicago, ECE
Fall 2009

Instructor: Natasha Devroye, devroye@ece.uic.edu
Course coordinates: Monday, Wednesday from 3:30-4:45pm in LH 103 (Lincoln Hall).
Office hours: Wednesday from 5-6:30pmm in SEO 1039, or by appointment

Welcome to ECE 534! This course is a graduate-level introduction to information theory, an extremely elegant mathematical theory with direct and significant impact on our life in the ``information age.'' Specifically, infomation theory allows us to quantify the fundamental limits of information communication and compression (to what size can we compress that image with and without distortion, how fast can we communicate ``reliably'' over a noisy channel?). Information theory has found its applications in many areas including statistics, computer science, biology, economics, as well as electrical engineering. We will touch upon many aspects of information theory, but the focus of this course will be on the direct applications of information theory in digital communications. Nonetheless, information theory teaches one a new and intuitive way of reasoning about problems, rendering it useful even to non-communication-engineering majors.

Course Textbook: Elements of Information Theory, by T. Cover and J. Thomas, Wiley 2006 (2nd edition!)
Other useful references:
Shannon's original 1948 paper, A Mathematical Theory of Communication.
A great list of papers related to network information theory are here.
The IEEE Information Theory Society web-page.
Recent (often just submitted for publication) results are found on arXiv.
Bob Gray's free online book called Entropy and Information Theory.
David MacKay's free online book called Information Theory, Pattern Recognition and Neural Networks.
A nice list or free online math/applied math textbooks.
Costello and Forney's excellent survey ``Channel Coding: The Road to Channel Capacity''

Notes: I will follow the course textbook fairly closely, using a mixture of slides (highlighting the main points and with nice illustrations) and more in-depth blackboard derivations/proofs in class. I will post a pdf version of the slides as they become ready here, but the derivations will be given in class only.
Introduction Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 7 Chapter 8 Chapter 9 Chapter 10 Chapter 15 Stefano Rini's BC lecture
Comments on the Broadcast Channel
Topics: Entropy and differential entropy, mutual information; data compression; channel capacity, the Gaussian channel; rate-distortion; universal-source-coding; network information theory. Contemporary examples and research topics.

Grading: Weekly homeworks (15%), Exam 1 = max(Exam1, Exam 2, Final) (20%), Exam 2 = max(Exam 2, Final) (20%), Project (15%), Final exam (30%).

Homework: Will be handed out each Monday, due the next Monday (1 week). All assignments from HW2 onwards MUST be submitted electronically as a latex file, and a printed pdf copy handed in during class. I will create the solutions from the best solutions I receive (with credit to the authors!). Submit the latex file via the "Course tools" then "Digital Dropbox" link on the Blackboard site (and choose "Send file", NOT "Add file"). Please make sure the "Title" you enter is of the form "HW#_Student_Name" (e.g. HW4_Natasha_Devroye). I will give a brief tutorial on how to use tex/latex during the 3rd lecture. You can find latex resources for Windows and for MAC by googling around. Finally, here is a template for you to use for homework submissions.
HW1: out 08/24, due 08/31. Textbook problems 2.6, 2.12, 2.14, 2.18, 2.29   Solutions
HW2: out 08/31, due 09/09. Textbook problems 2.32, 2.38, 2.48, 3.7, 3.9   Solutions
HW3: out 09/07, due 09/14. Textbook problems 3.2, 4.2, 4.13, 4.23, 4.27   Solutions
HW4: out 09/14, due 09/21. Textbook problems 5.4, 5.20, 5.30, 5.32, 5.33   Solutions
HW5: out 09/21, due 09/28. Textbook problems 5.44, 7.2, 7.4, 7.7, 7.20   Solutions
HW6: out 10/05, due 10/12. Textbook problems 7.21, 7.23, 7.24, 7.34, 7.37   Solutions
HW7: out 10/12, due 10/21. Textbook problems 8.1, 8.3, 8.8, 8.11   Solutions and alternative solutions.
HW8: out 10/19, due 10/26. Textbook problems 9.2, 9.5, 9.8, 9.9, 9.12   Solutions
HW9: out 10/26, due 11/02. Textbook problems 9.16, 9.18, 9.21, 10.1, 10.2   Solutions
HW10: BONUS homework due 11/09. Textbook problem 10.16   Solutions
HW11: out 11/09 due 11/16. Textbook problems 10.14, 10.15, 15.1, 15.2, 15.3   Solutions
HW12: out 11/16 due 11/23. Textbook problems 15.6, 15.11, 15.12, 15.16
HW13: out 11/23 due 11/30. Textbook problems 15.22, 15.31, 15.34, 15.35
Exams: For midterm 1, you may have one 8.5x11 double-sided sheet which you can fill with anything you like. No other books, notes or calculators. For midterm 2, you may have 2 of these crib sheets and for the final exam you may have 3 such crib sheets.
Past practice midterm 1: Spring 2006, Spring 2007, Fall 2009, Fall 2009 midterm 2

Project: The project, to be done individually, will consist of a short, professional, well- written report (8-10 pages, single spaced, latex 11-12pt). The goal will be to explore contemporary research topics in the area of information theory that are not covered in class. Pick (or suggest) a topic of interest to you and provide a comprehensive treatment of it: introduce the problem/topic, survey what has been done by whom on the topic (we expect many citations to relevant journal and conference papers. The most relevant journal is the IEEE Transactions on Information Theory, while some relevant conferences are the International Symposium on Information Theory (ISIT, held once a year), Allerton (held in UIUC), Information Theory Workshop (held twice a year).) After summarizing past work on the topic in your own words (plagiarism will NOT be tolerated), state some open problems in the area. Depending on the final number of students in the class, oral presentations may accompany the written reports. Some guidelines. Some help with bibtex and more. Here's a template which you can use if you like. It's in the standard IEEE format for journal articles. You'll need the IEEEtran document class (IEEEtrans.cls needs to be in the directory in which your file is), which you can download and read more about here.