Downloaded 506 times
Uploaded byOmar Enayet
Reinforcement Learning : A Beginners Tutorial
This document provides an overview of reinforcement learning concepts including: 1) It defines the key components of a Markov Decision Process (MDP) including states, actions, transitions, rewards, and discount rate. 2) It describes value functions which estimate the expected return for following a particular policy from each state or state-action pair. 3) It discusses several elementary solution methods for reinforcement learning problems including dynamic programming, Monte Carlo methods, temporal-difference learning, and actor-critic methods.
Reinforcement Learning : A Beginners Tutorial
- 1. Reinforcement LearningA Beginner’sTutorialBy: Omar Enayet(Presentation Version)
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8. Markov Decision ProcessAnMDP is defined by < S, A, p, r, >S - set of states of the environmentA(s)– set of actions possible in state s - probability of transition from s- expected reward when executing ain s - discount rate for expected rewardAssumption: discrete timet = 0, 1, 2, . . .rrrt +2t +3. . .s. . .t +1ssst+3t+1t+2taaaatt+1t +2t +3
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23. Advantages over DPNoModel
- 24. Simulation OR partof Model
- 25. Focus on smallsubset of states
- 26. Less Harmed byviolations of Markov PropertyFirst Visit VS Every-Visit
- 27.
- 28. Action-value instead ofState-value
- 29.
- 30. Advantages of TDLearning
- 31.
- 32.
- 34.
- 35. R-Learning (Off-Policy)>>Average Expectedreward per time-step
- 36.
- 39. ReferencesRichard S. Suttonand Andrew G. Barto. Reinforcement Learning, Bradford Books, 1998.Richard Crouch, Peter Bennett, Stephen Bridges, Nick Piper and Robert Oates - Monte Carlo - 2003Slides for reading with :Omar Enayet – Reinforcement Learning : A Beginner’s Tutorial- 2009
Editor's Notes
- #5 By a model of the environment we mean anything that an agent can use to predict how the environment will respond to its actions. Given a state and an action, a model produces a prediction of the resultant next state and next reward. If the model is stochastic, then there are several possible next states and next rewards, each with some probability of occurring. Some models produce a description of all possibilities and their probabilities; these we call distribution models. Other models produce just one of the possibilities, sampled according to the probabilities; these we call sample models. For example, consider modeling the sum of a dozen dice. A distribution model would produce all possible sums and their probabilities of occurring, whereas a sample model would produce an individual sum drawn according to this probability distribution.
- #8 Credit assignment problem: How do you distribute credit for success among the many decisions that may have been involved in producing it?
- #14 One of the challenges that arise in reinforcement learning and not in other kinds of learning is the trade-off between exploration and exploitation. To obtain a lot of reward, a reinforcement learning agent must prefer actions that it has tried in the past and found to be effective in producing reward. But to discover such actions, it has to try actions that it has not selected before. The agent has to exploit what it already knows in order to obtain reward, but it also has to explore in order to make better action selections in the future. The dilemma is that neither exploration nor exploitation can be pursued exclusively without failing at the task. The agent must try a variety of actions andprogressively favor those that appear to be best. On a stochastic task, each action must be tried many times to gain a reliable estimate its expected reward. The exploration-exploitation dilemma has been intensively studied by mathematicians for many decades