diff --git a/unit2/README.md b/unit2/README.md index b02f1af..4fc6892 100644 --- a/unit2/README.md +++ b/unit2/README.md @@ -44,9 +44,11 @@ Are you new to Discord? Check our **discord 101 to get the best practices** πŸ‘‰ 3️⃣ πŸ“– **Read An [Introduction to Q-Learning Part 1](https://huggingface.co/blog/deep-rl-q-part1)**. -4️⃣ πŸ“– **Read An [Introduction to Q-Learning Part 2](https://huggingface.co/blog/deep-rl-q-part2)**. +4️⃣ πŸ“ Take a piece of paper and **check your knowledge with this series of questions** ❔ πŸ‘‰ https://github.com/huggingface/deep-rl-class/blob/main/unit2/quiz1.md -5️⃣ πŸ‘©β€πŸ’» Then dive on the hands-on, where **you’ll implement our first RL agent from scratch**, a Q-Learning agent, and will train it in two environments: +5️⃣ πŸ“– **Read An [Introduction to Q-Learning Part 2](https://huggingface.co/blog/deep-rl-q-part2)**. + +6️⃣ πŸ‘©β€πŸ’» Then dive on the hands-on, where **you’ll implement our first RL agent from scratch**, a Q-Learning agent, and will train it in two environments: 1. Frozen Lake v1 ❄️: where our agent will need toΒ **go from the starting state (S) to the goal state (G)**Β by walking only on frozen tiles (F) and avoiding holes (H). 2. An autonomous taxi πŸš•: where the agent will needΒ **to learn to navigate**Β a city toΒ **transport its passengers from point A to point B.** @@ -58,7 +60,7 @@ The leaderboard πŸ‘‰ https://huggingface.co/spaces/chrisjay/Deep-Reinforcement-L You can work directly **with the colab notebook, which allows you not to have to install everything on your machine (and it’s free)**. -6️⃣ The best way to learn **is to try things on your own**. That’s why we have a challenges section in the colab where we give you some ideas on how you can go further: using another environment, using another model etc. +7️⃣ The best way to learn **is to try things on your own**. That’s why we have a challenges section in the colab where we give you some ideas on how you can go further: using another environment, using another model etc. ## Additional readings πŸ“š - [Reinforcement Learning: An Introduction, Richard Sutton and Andrew G. Barto Chapter 5, 6 and 7](http://incompleteideas.net/book/RLbook2020.pdf) diff --git a/unit2/assets/img/MC-3.jpg b/unit2/assets/img/MC-3.jpg new file mode 100644 index 0000000..dc74d5b Binary files /dev/null and b/unit2/assets/img/MC-3.jpg differ diff --git a/unit2/assets/img/TD-1.jpg b/unit2/assets/img/TD-1.jpg new file mode 100644 index 0000000..6d59d50 Binary files /dev/null and b/unit2/assets/img/TD-1.jpg differ diff --git a/unit2/assets/img/bellman4-quiz.jpg b/unit2/assets/img/bellman4-quiz.jpg new file mode 100644 index 0000000..f2d9585 Binary files /dev/null and b/unit2/assets/img/bellman4-quiz.jpg differ diff --git a/unit2/assets/img/bellman4.jpg b/unit2/assets/img/bellman4.jpg new file mode 100644 index 0000000..867bf67 Binary files /dev/null and b/unit2/assets/img/bellman4.jpg differ diff --git a/unit2/assets/img/mc-ex.jpg b/unit2/assets/img/mc-ex.jpg new file mode 100644 index 0000000..e7437a2 Binary files /dev/null and b/unit2/assets/img/mc-ex.jpg differ diff --git a/unit2/assets/img/monte-carlo-approach.jpg b/unit2/assets/img/monte-carlo-approach.jpg new file mode 100644 index 0000000..09012bf Binary files /dev/null and b/unit2/assets/img/monte-carlo-approach.jpg differ diff --git a/unit2/assets/img/summary-learning-mtds.jpg b/unit2/assets/img/summary-learning-mtds.jpg new file mode 100644 index 0000000..6f55e69 Binary files /dev/null and b/unit2/assets/img/summary-learning-mtds.jpg differ diff --git a/unit2/assets/img/td-ex.jpg b/unit2/assets/img/td-ex.jpg new file mode 100644 index 0000000..11d30e7 Binary files /dev/null and b/unit2/assets/img/td-ex.jpg differ diff --git a/unit2/assets/img/two-approaches.jpg b/unit2/assets/img/two-approaches.jpg new file mode 100644 index 0000000..b8f6add Binary files /dev/null and b/unit2/assets/img/two-approaches.jpg differ diff --git a/unit2/quiz1.md b/unit2/quiz1.md new file mode 100644 index 0000000..2075759 --- /dev/null +++ b/unit2/quiz1.md @@ -0,0 +1,99 @@ +# Knowledge Check βœ”οΈ + +The best way to learn and [avoid the illusion of competence](https://fr.coursera.org/lecture/learning-how-to-learn/illusions-of-competence-BuFzf) **is to test yourself.** This will help you to find **where you need to reinforce your knowledge**. + +πŸ“ Take a piece of paper and try to answer by writing, **then check the solutions**. + +### Q1: What are the two main approaches to find optimal policy? + +
+Solution + +The two main approaches are: +- *Policy-based methods*: **Train the policy directly** to learn which action to take given a state. +- *Value-based methods* : Train a value function to **learn which state is more valuable and use this value function to take the action that leads to it**. + +Two approaches of Deep RL + +πŸ“– If you don't remember, check πŸ‘‰ https://huggingface.co/blog/deep-rl-q-part1#what-is-rl-a-short-recap + +
+ + +### Q2: What is the Bellman Equation? + +
+Solution + +**The Bellman equation is a recursive equation** that works like this: instead of starting for each state from the beginning and calculating the return, we can consider the value of any state as: + +$R_{t+1} + ( gamma * V(S_{t+1}))$ + +The immediate reward + the discounted value of the state that follows + +πŸ“– If you don't remember, check πŸ‘‰ https://huggingface.co/blog/deep-rl-q-part1#the-bellman-equation-simplify-our-value-estimation + +
+ + +### Q3: Define each part of the Bellman Equation + +Bellman equation quiz + + +
+Solution + +Bellman equation solution + +πŸ“– If you don't remember, check πŸ‘‰ https://huggingface.co/blog/deep-rl-q-part1#the-bellman-equation-simplify-our-value-estimation + +
+ +### Q4: What is the difference between Monte Carlo and Temporal Difference learning methods? + +
+Solution + +There are two types of methods to learn a policy or a value function: +- With the *Monte Carlo method*, we update the value function **from a complete episode**, and so we use the actual accurate discounted return of this episode. +- With the *TD Learning method*, we update the value function **from a step, so we replace Gt that we don't have with an estimated return called TD target**. + +summary-learning-mtds + +πŸ“– If you don't remember, check πŸ‘‰ https://huggingface.co/blog/deep-rl-q-part1#monte-carlo-vs-temporal-difference-learning + +
+ +### Q5: Define each part of Temporal Difference learning formula + +TD Learning exercise + +
+Solution + +TD Exercise + + +πŸ“– If you don't remember, check πŸ‘‰ https://huggingface.co/blog/deep-rl-q-part1#temporal-difference-learning-learning-at-each-step +
+ + +### Q6: Define each part of Monte Carlo learning formula + +MC Learning exercise + +
+Solution + +MC Exercise + +πŸ“– If you don't remember, check πŸ‘‰ https://huggingface.co/blog/deep-rl-q-part1#monte-carlo-learning-at-the-end-of-the-episode +
+ + +--- + +Congrats on **finishing this Quiz** πŸ₯³, if you missed some elements, take time to [read again the chapter](https://huggingface.co/blog/deep-rl-q-part1) to reinforce (😏) your knowledge. + +**Keep Learning, Stay Awesome**