diff --git a/units/en/_toctree.yml b/units/en/_toctree.yml index d7772a7..21d4d59 100644 --- a/units/en/_toctree.yml +++ b/units/en/_toctree.yml @@ -92,6 +92,8 @@ title: The Deep Q-Network (DQN) - local: unit3/deep-q-algorithm title: The Deep Q Algorithm + - local: unit3/glossary + title: Glossary - local: unit3/hands-on title: Hands-on - local: unit3/quiz diff --git a/units/en/unit3/glossary.mdx b/units/en/unit3/glossary.mdx new file mode 100644 index 0000000..f0adaa9 --- /dev/null +++ b/units/en/unit3/glossary.mdx @@ -0,0 +1,39 @@ +# Glossary + +This is a community-created glossary. Contributions are welcomed! + +- **Tabular Method:** type of problem in which the state and action spaces are small enough to approximate value functions to be represented as arrays and tables. +**Q-learning** is an example of tabular method since a table is used to represent the value for different state-action pairs. + +- **Deep Q-Learning:** method that trains a neural network to approximate, given a state, the different **Q-values** for each possible action at that state. +Is used to solve problems when observational space is too big to apply a tabular Q-Learning approach. + +- **Temporal Limitation:** is a difficulty presented when the environment state is represented by frames. A frame by itself does not provide temporal information. +In order to obtain temporal information, we need to **stack** a number of frames together. + +- **Phases of Deep Q-Learning:** + - **Sampling:** actions are performed, and observed experience tuples are stored in a **replay memory**. + - **Training:** batches of tuples are selected randomly and the neural network updates its weights using gradient descent. + +- **Solutions to stabilize Deep Q-Learning:** + - **Experience Replay:** a replay memory is created to save experiences samples that can be reused during training. + This allows the agent to learn from the same experiences multiple times. Also, it makes the agent avoid to forget previous experiences as it get new ones. + **Random sampling** from replay buffer allows to remove correlation in the observation sequences and prevents action values from oscillating or diverging + catastrophically. + + - **Fixed Q-Target:** In order to calculate the **Q-Target** we need to estimate the discounted optimal **Q-value** of the next state by using Bellman equation. The problem + is that the same network weigths are used to calculate the **Q-Target** and the **Q-value**. This means that everytime we are modifying the **Q-value**, the **Q-Target** also moves with it. + To avoid this issue, a separate network with fixed parameters is used for estimating the Temporal Difference Target. The target network is updated by copying parameters from + our Deep Q-Network after certain **C steps**. + + - **Double DQN:** method to handle **overstimation** of **Q-Values**. This solution uses two networks to decouple the action selection from the target **-Value generation**: + -**DQN Network** to select the best action to take for the next state (the action with the highest **Q-Value**) + -**Target Network** to calculate the target **Q-Value** of taking that action at the next state. + This approach reduce the **Q-Values** overstimation, it helps to train faster and have more stable learning. + + +If you want to improve the course, you can [open a Pull Request.](https://github.com/huggingface/deep-rl-class/pulls) + +This glossary was made possible thanks to: + +- [Dario Paez](https://github.com/dario248)