מבוא לעיבוד מידע ולמידה

Introduction to Information Processing and Learning (76915)

Fall Semester 2011/12

This course is a mandatory course for the multidisciplinary Ph.D. program in computation and information processing in the brain (Neural Computation). The course is also open for CS 3rd year and graduate students. It assumes knowledge of probability theory (80420 or 52114/5), as well as basic programming skills. This page contains material that will help you in the course of the course and will be updated constantly. Lecture notes (in Hebrew) are available on-line.

Course goals and plan

Acquiring the basic concepts of computation, statistical information processing, and computational learning, required for modeling biological and cognitive systems and for designing machines that can learn.

 

List of Topics

  1. Statistical decision theory - Bayesian approaches and Hypothesis testing

 

  1. Parameter Estimation - Bayesian and maximum likelihood approaches. Bias variance tradeoffs. Minimum variance unbiased estimators. Cramer Rao inequality. Conjugate priors. The Expectation Maximization (EM) algorithm.

 

  1. Information theory - Source and channel coding theorems. Mutual information and information processing principles.

 

  1. Theory of classification algorithms - The PAC framework, VC dimension, generalization bounds, model selection.

 

  1. (time permitting) Graphical models - Directed and undirected models. HMMs.

 

NOTES

 

Teacher

Amir Globerson                     gamir at cs.huji.ac.il

 

Ross 210. Reception hours: by appointment.

Assistant

Ben Engelhard                      ben.engelhard at mail.huji.ac.il

 

Faculty of Medicine, 5-41. Phone:  6757071 (87071).

Reception hours: by appointment.

Exercise Checker

Rea Mitelman                      ream at alice.nc.huji.ac.il

 

Formalities

Lecture: Monday 12:00-13:45 , ICNC
Exercise: Thursday 12:00-13:45 , ICNC

 

Requirements:  Submit and pass all exercises (published in a weekly or bi-weekly basis) + written exam. Exercises will be weighted as 25% of the grade.

 

Further guidelines regarding the exercises:

Exercises should be submitted to the “ibud meida” drawer in the ICNC up to the date and time due (usually Thursdays at 12:00 noon). The only acceptable reasons for deadline deferral are miluim and trips abroad. All other reasons will not be accepted. There will be a five point reduction for every day of late submission. Those who submit late need inform the exercise checker by email the date they are submitting; submissions on weekends will be counted as submission on the following Sunday. Of the exercises, three or four will be computational exercises which need to be programmed in Matlab. Students with little familiarity with Matlab are encouraged to  improve their skills in the beginning of the semester; some links for help with Matlab are provided on this webpage in the Bibliography section. Computational exercises may be submitted in hard copy or by email to the exercise checker up to the due date and time. In any case, all required answers and graphs must be in a single, orderly document. Additionally, all code used must be submitted. All work must be done individually. The 2 lowest non-computational exercises’ score will not be considered for the grade.

 

Lecture Notes

Online version of the course book

The book will be handed to the course participants, so there is no need to print it.

These notes were written by Gal Chechik, Lidror Tryoanski and Naftali Tishby.
The on-line lectures notes are available in Hebrew PDF formats, to view PDF files, use Adobe acrobat reader.

Recommended Bibliography

  1. M.H. Degroot, "Probability and Statistics", Addison-Wesley Pub Co.
    Check availability in JMC library

 

  1. A.V. Duda and Hart, “Pattern Recognition and Scene Analysis:, (Wiley 1973), Ch. 2-6.
    Check availability in JMC library

 

  1. T. Hastie, R. Tibshirani and J. H. Friedman, “The Elements of Statistical Learning”. Available Online.

 

  1. T. Cover and J. Thomas, “Elements of Information Theory”, (Wiley & Sons 1991), from Ch. 2-5,8,11,12.
    Check availability in JMC library
    Shannon's 1948 paper

 

  1. Bishop, Christopher M. “Pattern recognition and machine learning”, (New York : Springer, 2006.)

      Check availability in JMC library 

 

  1. M.J. Kearns and U.V. Vazirani,  “Introduction to Computational Learning Theory”, (MIT Press 1994)
    Check availability in JMC library

  2. V. Vapnik, “Estimation of Dependences Based on Empirical Data”, (Springer Verlag 1982), from Ch. 1,2,6,7.

   

  1. Kay S.M. “Fundamentals of Statistical Signal Processing”, Prentice Hall Signal Processing Series. 1993
     
  2. E. Samuel, "TEORYA STATISTIT". (in Hebrew) Part A. Ch. IV. Part B. Ch. II, III.
     
  3. L.R. Rabiner, “Fundamentals of Speech Recognition”, Ch. 6.
    A Tutorial on Hidden Markov Models
     
  4. J. Pearl, “Probabilistic Reasoning in Intelligent Systems”, (Morgan Kaufmann, 1988)
     
  5. M. Anthony and N. Biggs, “Computational Learning Theory, An Introduction”, (Cambridge U. Press 1992)
     
  6. J. O. Berger, “Statistical Decision Theory and Bayesian Analysis”, (Springer Verlag 1985), from Ch. 1-4.
    Check availability in JMC library
     
  7. D.J.C. MacKay, A short course in Information theory
     
  8. Matlab tutorials and FAQ's: MatLab help, MatLab FAQ, MatLab Primer,
     

Exercises

 

Videos

Lectures Video Playlist (youtube)

Grades

Grades File (Excel)