Instructor: Rich Sutton, (sutton@cs.ualberta.ca) (http://www.cs.ualberta.ca/~sutton)
Office: Athabasca 3-13 Office hours: Tues 5-6, Thurs 1-2, and by appointment
Class Times: Monday and Wednesday, 4:00–5:20
Class Room: CSC B-41
Lab Time: Fridays at 3 in B-41
Teaching assistants: Patrick Pilarsky, Joseph Modayil, and Rupam Mahmood will help
Description: This course will provide a comprehensive introduction to reinforcement learning as an approach to artificial intelligence, emphasizing the design of complete agents interacting with stochastic, incompletely known environments. Reinforcement learning has adapted key ideas from machine learning, operations research, psychology, and neuroscience to produce some strikingly successful engineering applications. The focus is on algorithms for learning what actions to take, and when to take them, so as to optimize long-term performance. This may involve sacrificing immediate reward to obtain greater reward in the long-term or just to obtain more information about the environment. The course will cover Markov decision processes, dynamic programming, temporal-difference learning, Monte Carlo reinforcement learning methods, eligibility traces, the role of function approximation, and the integration of learning and planning. The course will emphasize the development of intuition relating the mathematical theory of reinforcement learning to the design of human-level artificial intelligence.
Textbook: Reinforcement Learning: An Introduction, by Richard S. Sutton and Andrew G. Barto. Although a version of the textbook is available online, and a scanned version can be obtained through the library, students are strongly encouraged to get their hands on the physical textbook. Much of the readings and questions will come directly from the book. The textbook is available in the bookstore or through internet sellers (e.g., see booksprice.com).
Prerequisites: Interest in learning approaches to artificial intelligence; basic probability theory; computer programming ability. You should be comfortable with statistical ideas such as probability distributions and expected values. Familiarity with linear algebra would be helpful but is not required.
Written Exercises: There will be a set of exercises for most chapters. These will be due at the beginning of the second day on which the chapter is covered in class. All exercises will be marked and returned to you. Answer sheets for each week’s exercises will be made available at the class on the day on which the exercises are due, so your exercises must be turned in on time.
Grading will be on the basis of (with relative weighting):
6 Written exercises
4 Mid-term
10 Term projects