From 1197198d2b973c091791cb867dd07b9ad05525db Mon Sep 17 00:00:00 2001 From: simoninithomas Date: Mon, 2 Jan 2023 14:43:34 +0100 Subject: [PATCH] Added policy gradient section --- units/en/unit4/advantages-disadvantages.mdx | 30 +++---- units/en/unit4/introduction.mdx | 12 +-- units/en/unit4/policy-gradient.mdx | 88 +++++++++---------- .../unit4/what-are-policy-based-methods.mdx | 35 ++++---- 4 files changed, 79 insertions(+), 86 deletions(-) diff --git a/units/en/unit4/advantages-disadvantages.mdx b/units/en/unit4/advantages-disadvantages.mdx index 836d36d..e1f6d65 100644 --- a/units/en/unit4/advantages-disadvantages.mdx +++ b/units/en/unit4/advantages-disadvantages.mdx @@ -1,10 +1,10 @@ -# The advantages and disadvantages of Policy-based methods +# The advantages and disadvantages of policy-gradient methods -At this point, you might ask "But Deep Q-Learning is excellent! Why use policy gradient methods?", let's study then the advantages and disadvantages of Policy-gradient methods +At this point, you might ask "But Deep Q-Learning is excellent! Why use policy-gradient methods?", let's study then the **advantages and disadvantages of policy-gradient methods**. ## Advantages -There are multiple advantages over Value-based methods. Let's see some of them: +There are multiple advantages over value-based methods. Let's see some of them: ### The simplicity of integration @@ -12,7 +12,7 @@ Indeed, **we can estimate the policy directly without storing additional data (a ### Policy-gradient methods can learn a stochastic policy -Policy gradient methods can **learn a stochastic policy while value functions can't**. +Policy-gradient methods can **learn a stochastic policy while value functions can't**. This has two consequences: @@ -28,7 +28,7 @@ Let's take an example: we have an intelligent vacuum cleaner whose goal is to su Our vacuum cleaner can only perceive where the walls are. -The problem is that the two red cases are aliased states because the agent perceives an upper and lower wall for each. +The problem is that the **two rose cases are aliased states because the agent perceives an upper and lower wall for each**.
Hamster 1 @@ -36,9 +36,9 @@ The problem is that the two red cases are aliased states because the agent perce Under a deterministic policy, the policy will either move right when in a red state or move left. **Either case will cause our agent to get stuck and never suck the dust**. -Under a value-based RL algorithm, we learn a quasi-deterministic policy ("greedy epsilon strategy"). Consequently, our agent can spend a lot of time before finding the dust. +Under a value-based Reinforcement learning algorithm, we learn a **quasi-deterministic policy** ("greedy epsilon strategy"). Consequently, our agent can **spend a lot of time before finding the dust**. -On the other hand, an optimal stochastic policy will randomly move left or right in grey states. Consequently, **it will not be stuck and will reach the goal state with a high probability**. +On the other hand, an optimal stochastic policy **will randomly move left or right in rose states**. Consequently, **it will not be stuck and will reach the goal state with a high probability**.
Hamster 1 @@ -52,21 +52,21 @@ But what if we have an infinite possibility of actions? For instance, with a self-driving car, at each state, you can have a (near) infinite choice of actions (turning the wheel at 15°, 17.2°, 19,4°, honking, etc.). We'll need to output a Q-value for each possible action! And taking the max action of a continuous output is an optimization problem itself! -Instead, with a policy gradient, we output a **probability distribution over actions.** +Instead, with a policy-gradient methods, we output a **probability distribution over actions.** ### Policy-gradient methods have better convergence properties -In Value-based methods, we use an aggressive operator to **change the value function: we take the maximum over Q-estimates**. +In value-based methods, we use an aggressive operator to **change the value function: we take the maximum over Q-estimates**. Consequently, the action probabilities may change dramatically for an arbitrary small change in the estimated action-values if that change results in a different action having the maximal value. For instance if during the training the best action was left (with Q-value of 0.22) and the training step after it's right (since the right Q-value become 0.23) we dramatically changed the policy since now the policy will take most of the time right instead of left. -On the other hand, in Policy-based methods, stochastic policy action preferences (probability of taking action) **changes smoothly over time**. +On the other hand, in policy-gradient methods, stochastic policy action preferences (probability of taking action) **changes smoothly over time**. ## Disadvantages -Naturally, Policy Gradient methods have also some disadvantages: +Naturally, policy-gradient methods have also some disadvantages: -- **Policy gradients converge a lot of time on a local maximum instead of a global optimum.** -- Policy gradient goes less faster, **step by step: it can take longer to train (inefficient).** -- Policy gradient can have high variance, we'll see in Actor Critic unit why and how we can solve this problem. +- **Policy-gradient converge a lot of time on a local maximum instead of a global optimum.** +- Policy-gradient goes less faster, **step by step: it can take longer to train (inefficient).** +- Policy-gradient can have high variance, we'll see in Actor Critic unit why and how we can solve this problem. -👉 If you want to go deeper on the advantages and disadvantages of Policy Gradients methods, [you can check this video](https://youtu.be/y3oqOjHilio). +👉 If you want to go deeper on the advantages and disadvantages of policy-gradient methods, [you can check this video](https://youtu.be/y3oqOjHilio). diff --git a/units/en/unit4/introduction.mdx b/units/en/unit4/introduction.mdx index 902d164..5495901 100644 --- a/units/en/unit4/introduction.mdx +++ b/units/en/unit4/introduction.mdx @@ -1,6 +1,6 @@ # Introduction [[introduction]] -In the last unit, we learned about Deep Q-Learning. In this value-based Deep Reinforcement Learning algorithm, we **used a deep neural network to approximate the different Q-values for each possible action at a state.** +In the last unit, we learned about Deep Q-Learning. In this value-based deep reinforcement learning algorithm, we **used a deep neural network to approximate the different Q-values for each possible action at a state.** Indeed, since the beginning of the course, we only studied value-based methods, **where we estimate a value function as an intermediate step towards finding an optimal policy.** @@ -10,7 +10,7 @@ Because, in value-based, ** \\(π\\) exists only because of the action value es But, with policy-based methods, we want to optimize the policy directly **without having an intermediate step of learning a value function.** -So today, **we'll learn about Policy based methods, and we'll study a subset of these methods called Policy Gradients**. Then we'll implement our first Policy Gradient algorithm called Monte Carlo **Reinforce** from scratch using PyTorch. Before testing its robustness using CartPole-v1, PixelCopter, and Pong. +So today, **we'll learn about policy-based methods, and we'll study a subset of these methods called Policy Gradients**. Then we'll implement our first policy gradient algorithm called Monte Carlo **Reinforce** from scratch using PyTorch. Before testing its robustness using CartPole-v1, PixelCopter, and Pong.
Environments @@ -18,12 +18,6 @@ So today, **we'll learn about Policy based methods, and we'll study a subset of Let's get started, -- [What are Policy-Gradient Methods?](#what-are-policy-gradient-methods) - - [An Overview of Policy Gradients](#an-overview-of-policy-gradients) - - [The Advantages of Policy-Gradient Methods](#the-advantages-of-policy-gradient-methods) - - [The Disadvantages of Policy-Gradient Methods](#the-disadvantages-of-policy-gradient-methods) -- [Reinforce (Monte Carlo Policy Gradient)](#reinforce-monte-carlo-policy-gradient) - @@ -58,7 +52,7 @@ Don't hesitate to train your agent in other environments. The **best way to lea We published additional readings in the syllabus if you want to go deeper 👉 **[https://github.com/huggingface/deep-rl-class/blob/main/unit5/README.md](https://github.com/huggingface/deep-rl-class/blob/main/unit5/README.md)** -In the next unit, we’re going to learn about a combination of Policy-Based and Value-based methods called Actor Critic Methods. +In the next unit, we’re going to learn about a combination of Policy-Based and value-based methods called Actor Critic Methods. And don't forget to share with your friends who want to learn 🤗! diff --git a/units/en/unit4/policy-gradient.mdx b/units/en/unit4/policy-gradient.mdx index 71bdfe2..532bbee 100644 --- a/units/en/unit4/policy-gradient.mdx +++ b/units/en/unit4/policy-gradient.mdx @@ -1,47 +1,40 @@ -# Diving deeper into Policy-gradient methods +# Diving deeper into policy-gradient methods ## Getting the big picture -We just learned that the goal of Policy-gradient methods is to find parameters //(/theta //) that maximize the expected return. +We just learned that the goal of policy-gradient methods is to find parameters //(/theta //) that maximize the expected return. -The idea is that we have a parameterized stochastic policy. In our case, a neural network that output a probability distribution over actions. The probability of taking each action is also called action preference. +The idea is that we have a *parameterized stochastic policy*. In our case, a neural network that output a probability distribution over actions. The probability of taking each action is also called *action preference*. If we take the example of CartPole-v1: - As input, we have a state. - As output, we have a probability distribution over actions at that state. -TODO: IMAGE NEURAL NETWORK CARTPOLE +Policy based - -Our goal with Policy-Gradients is to control the probability distribution of actions by tuning the policy such that **good actions (that maximize the return) are sampled more frequently in the future.** +Our goal with policy-gradient is to **control the probability distribution of actions** by tuning the policy such that **good actions (that maximize the return) are sampled more frequently in the future.** Each time the agent interacts with the environment, we tweak the parameters such that good actions will be sampled more likely in the future. But how we're going to optimize the weights using the expected return? -The idea, is that we're going to let the agent interact during an episode. And if we win the episode, we consider that each action taken were good and must be more sampled in the future +The idea, is that we're going to **let the agent interact during an episode**. And if we win the episode, we consider that each action taken were good and must be more sampled in the future since they lead to win. So for each state, action pair, we want to increase the //(P(a|s)//): probability of taking that action at that state. Or decrease if we lost. -TODO: IMAGE -Collect an episode -Change the weights of the policy network +The Policy-gradient algorithm (simplified) looks like this: +
+ Policy Gradient Big Picture +
- If Good return: increase the P(a|s) each state action combination taken - If Bad: decrease the P(a|s) +Now that we got the big picture, let's dive deeper into policy-gradient. - The Policy Gradient algorithm (simplified) looks like this: -
- Policy Gradient Big Picture -
+## Diving deeper into policy-gradient - -## Diving deeper into Policy-gradient - -We have our policy \\(\pi\\) which has a parameter \\(\theta\\). This \\(\pi\\), given a state, **outputs a probability distribution of actions**. +We have our stochastic policy \\(\pi\\) which has a parameter \\(\theta\\). This \\(\pi\\), given a state, **outputs a probability distribution of actions**.
- Policy + Policy
Where \\(\pi_\theta(a_t|s_t)\\) is the probability of the agent selecting action at from state st, given our policy. @@ -50,36 +43,50 @@ Where \\(\pi_\theta(a_t|s_t)\\) is the probability of the agent selecting action ### The Objective Function -The Objective function gives us the **performance of the agent** given a trajectory, it outputs the *expected cumulative reward*. +The Objective function gives us the **performance of the agent** given a trajectory (state action sequence without taking into account reward (contrary to an episode)), it outputs the *expected cumulative reward*. -TODO: Illustration Expected reward +Return -Our objective then is to maximize the expected cumulative rewards by finding \\(\theta \\) that will output the best action probability distributions. +Let's detail a little bit more this formula: +- The *expected return* (also called expected cumulative reward), is the weighted average (where the weights are given by \\(P(\tau;\theta)\\) of all possible values that the return \\(R(\tau)\\) can take. +Return +- \\(R(\tau)\\) : Return from an arbitrary trajectory. To take this quantity and use it to calculate the expected return, we need to multiply it by the probability of each possible trajectory. +- \\(P(\tau;\theta)\\) : Probability of each possible trajectory \\(tau\\) (that probability depends on (\\theta\\) since it defines the policy that it uses to select the actions of the trajectory which as an impact of the states visited). +- \\(J(\theta)\\) : Expected return, we calculate it by summing for all trajectories, the probability of taking that trajectory given $\theta$ and the return of this trajectory. + +Our objective then is to maximize the expected cumulative rewards by finding \\(\theta \\) that will output the best action probability distributions: -## Gradient Ascent -Since we want to find the values of \\(\theta\\) that maximize our objective function \\(J(\theta)\\), we need to use **gradient-ascent**. It's the inverse of *gradient-descent* since it gives the direction of the steepest increase of \\(J(\theta)\\). +Max objective + + +## Gradient Ascent and the Policy-gradient Theorem +Policy-gradient is an optimization problem: we want to find the values of \\(\theta\\) that maximize our objective function \\(J(\theta)\\), we need to use **gradient-ascent**. It's the inverse of *gradient-descent* since it gives the direction of the steepest increase of \\(J(\theta)\\). + +Gradient Ascent Our update step for gradient-ascent is: -\\ \theta \leftarrow \theta + \alpha * \nabla_\theta U(\theta) \\) +\\ \theta \leftarrow \theta + \alpha * \nabla_\theta J(\theta) \\) -But to +We can repeatidly apply this update state in the hope that \\(\theta)\\ converges to the value that maximize \\J(\theta)\\). + +However, we have two problems to derivate \\(J(\theta)\\): +1. We can't calculate the true gradient of the objective function, since it would imply to calculate the probability of each possible trajectory which is computatially super expensive. +We want then to **calculate an estimation of the gradient with a sample-based estimate (collect some trajectories)**. + +2. We have another problem, that I detail in the optional next section. That is to differentiate this objective function we need to differentiate the state distribution (attached to the environment it gives us the probability of the environment goes into next state given the current state and the action taken) but we might not know about it. + +Fortunately we're going to use a solution called the Policy Gradient Theorem that will help us to reformulate the objective function into a differentiable function that does not involve the differentiation of the state distribution. +Policy Gradient +## The policy-gradient algorithm -The score function J is the expected return: -
- Return -
- -Remember that policy gradient can be seen as an optimization problem. So we must find the best parameters (θ) to maximize the score function, J(θ). - -To do that we’re going to use the [Policy Gradient Theorem](https://www.youtube.com/watch?v=AKbX1Zvo7r8). I’m not going to dive on the mathematical details but if you’re interested check [this video](https://www.youtube.com/watch?v=AKbX1Zvo7r8) The Reinforce algorithm works like this: Loop: @@ -98,12 +105,3 @@ The interpretation we can make is this one: - \\(R(\tau)\\): is the scoring function: - If the return is high, it will push up the probabilities of the (state, action) combinations. - Else, if the return is low, it will push down the probabilities of the (state, action) combinations. - - - - -## The Policy Gradient Theorem - - - -## diff --git a/units/en/unit4/what-are-policy-based-methods.mdx b/units/en/unit4/what-are-policy-based-methods.mdx index 4456c44..b8d8299 100644 --- a/units/en/unit4/what-are-policy-based-methods.mdx +++ b/units/en/unit4/what-are-policy-based-methods.mdx @@ -1,40 +1,41 @@ -# What are the Policy Based methods? +# What are the policy-based methods? -The main goal of Reinforcement learning is to find the optimal policy \\(\pi*\\) that will maximize the expected cumulative reward. +The main goal of Reinforcement learning is to **find the optimal policy \\(\pi*\\) that will maximize the expected cumulative reward**. Because Reinforcement Learning is based on the *reward hypothesis* that is **all goals can be described as the maximization of the expected cumulative reward.** For instance, in a soccer game (that you're going to train the agents in two units), the goal is to win the game. We can describe this goal in reinforcement learning as maximizing the number of goal scored (when the ball cross the goal line) into your opponent soccer goals. And minimize the number of goals into yours soccer goals. -TODO ADD IMAGE SOCCER +Soccer -## Value-based, Policy-based and Actor-Critic methods +## Value-based, Policy-based and Actor-critic methods We studied in the Unit 1, that we had two methods to find (most of the time approximate) this optimal policy \\(\pi*\\). -- In *Value-Based methods*, we learn a value function. +- In *value-Based methods*, we learn a value function. - The idea then is that an optimal value function leads to an optimal policy \\(\pi*\\). - Our objective is to **minimize the loss between, predicted and target value to match the true action-value function. - We have a policy, but it's implicit since it **was generated directly from the Value function**. For instance, in Q-Learning we defined an epsilon-greedy policy. -- On the other hand, in *Policy-based methods*, we directly learn to approximate \\(\pi*\\) without having to learn a value function. - - The idea then is to parametize policy, for instance using a neural network \\(\pi_\theta\\), this policy will output a probability distribution over actions (stochastic policy). - - Our objective then is *to maximize the performance of the parametized policy using gradient ascent*. +- On the other hand, in *policy-based methods*, we directly learn to approximate \\(\pi*\\) without having to learn a value function. + - The idea then is to parameterize policy, for instance using a neural network \\(\pi_\theta\\), this policy will output a probability distribution over actions (stochastic policy). + - Our objective then is *to maximize the performance of the parameterized policy using gradient ascent*. - To do that we control the parameter \\(\theta\\) that will affect the distribution of actions over a state. -- Finally, we'll study next time *Actor-Critic* which is a combination of value-based and policy based methods. +- Finally, we'll study next time *actor-critic* which is a combination of value-based and policy-based methods. -Consequently, thanks to Policy-based methods, we can directly optimize our policy \\(\pi_theta\\) to output a probability distribution over actions \\(\pi_\theta(a|s)\\) that leads to the best cumulative return. +Policy based + +Consequently, thanks to policy-based methods, we can directly optimize our policy \\(\pi_theta\\) to output a probability distribution over actions \\(\pi_\theta(a|s)\\) that leads to the best cumulative return. To do that we define an objective function \\(J(\theta)\\), that is the expected cumulative reward and we **want to find \\(\theta\\) that maximize this objective function**. -## The difference between Policy-based and Policy-gradient methods +## The difference between policy-based and policy-gradient methods + +Policy-gradient methods, what we're going to study in this unit, is a subclass of policy-based methods. In policy-based methods, the optimization is most of the time *on-policy* since for each update we only use data (trajectories) collected **by our most recent version of** \\(\pi_\theta\\). -Policy-gradient methods, what we're going to study in this unit, is a subclass of Policy-based methods. The difference between these two methods **lies on how we optimize the parameter** //(/theta//): -- In *Policy-Based methods*, we search directly for the optimal policy. We can optimize the parameter //(/theta//) **indirectly** by maximize the local approximation of the objective function with techniques like hill climbing, simulated annealing or Evolution Strategies. -- In *Policy-Gradient methods*, because we're a subclass of the Policy-based methods, we search directly for the optimal policy. And we optimize the parameter //(/theta//) **directly** by performing the gradient ascent on the performance of the objective function \\(J(\theta)\\). +- In *policy-based methods*, we search directly for the optimal policy. We can optimize the parameter //(/theta//) **indirectly** by maximize the local approximation of the objective function with techniques like hill climbing, simulated annealing or evolution strategies. +- In *policy-gradient methods*, because we're a subclass of the policy-based methods, we search directly for the optimal policy. But we optimize the parameter //(/theta//) **directly** by performing the gradient ascent on the performance of the objective function \\(J(\theta)\\). -In Policy-based methods, the optimization is most of the time *on-policy* since for each update we only use data (trajectories) collected **by our most recent version of** \\(\pi_\theta\\). - -Before diving more into how works Policy-gradient methods (the objective function, policy gradient theorem, gradient ascent etc.) let's study the advantages and disadvantages of policy based methods. +Before diving more into how works policy-gradient methods (the objective function, policy gradient theorem, gradient ascent etc.) let's study the advantages and disadvantages of policy-based methods.