First, we need to understand the key nomenclature in reinforcement learning, which includes: Agent, Environment, Reward, Action, State, Policy Function, and Value Function.
Terms
- Agent It takes action; e.g., if you are training a drone, then the drone is the agent.
- Environment Where the agent exists and moves.
- Actions What the agent performs on the environment; a finite set of actions (e.g., moving forward or backward).
- Action Space The complete set of all possible actions.
- Observations How the environment responds for each action taken by the agent; States.
- State A specific situation in the environment.
- Reward A measure of the success or failure of the action taken by the agent.
- Total Reward The sum of all rewards: \( R_t = \sum_{i=t}^\infty r_i \).
- Discounted Total Reward Discounts some of the future rewards: $ R_t = \sum_{i=t}^\infty \gamma^{i-t} r_i $ (where \(\gamma\) is the discount factor, \(0 < \gamma < 1\)).
-
Reinforcement Learning (RL)
-
Model-Free RL
-
Value-Based Learns the value of states/actions. Picks the highest value. Q-Learning, DQN, SARSA
-
Policy-Based Directly optimizes the behavior strategy (policy). REINFORCE, Policy Gradients
-
-
Actor-Critic RL
-
Hybrid Architecture Actor chooses actions, Critic evaluates them. PPO, A3C, SAC
-
-
Model-Based RL
-
Learn the Model Agent figures out environment rules from scratch to plan. Dreamer, Dyna-Q
-
Given the Model Rules are known (like Chess); agent simulates ahead. AlphaZero
-
-
Specialized Domains
-
Multi-Agent (MARL) Multiple agents cooperating or competing. MADDPG, MAPPO
-
Offline RL Learning purely from a static, pre-recorded dataset. CQL, Behavioral Cloning
-
Inverse RL Learning the reward function by watching experts.
-
-
Resources:
- Emmanouil Tzorakoleftherakis, “What Is Reinforcement Learning?,” MathWorks, 2026. https://www.mathworks.com/discovery/reinforcement-learning.html
- Zhang, H., Yu, T. (2020). Taxonomy of Reinforcement Learning Algorithms. In: Dong, H., Ding, Z., Zhang, S. (eds) Deep Reinforcement Learning. Springer, Singapore. https://doi.org/10.1007/978-981-15-4095-0_3
- “Chap 3. Taxonomy of Reinforcement Learning,” Deep Reinforcement Learning Book, 2020. https://deepreinforcementlearningbook.org/docs/Chap%203.%20Taxonomy%20of%20Reinforcement%20Learning/
- Jacob Murel Ph.D and E. Kavlakoglu, “What is reinforcement learning? | IBM,” Ibm.com, Nov. 22, 2024. https://www.ibm.com/think/topics/reinforcement-learning
- Steve Brunton, “Reinforcement Learning Series: Overview of Methods,” Jan. 03, 2022. https://www.youtube.com/watch?v=i7q8bISGwMQ
- Alexander Amini, “MIT 6.S191: Reinforcement Learning,” Apr. 27, 2026. https://www.youtube.com/watch?v=1ij3dweHu-0
- T. Wang, 2018. [Online]. Available: https://tingwuwang.github.io/papers_slides/REINFORCE.pdf
- “5.3. Model-Free Learning,” Introduction to Artificial Intelligence, 2026. https://inst.eecs.berkeley.edu/~cs188/textbook/rl/mfl.html
- D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “DREAM TO CONTROL: LEARNING BEHAVIORS BY LATENT IMAGINATION,” in Proceedings of the International Conference on Learning Representations (ICLR), Mar. 2020. doi: arXiv:1912.01603v3.
- Neuromatch, “Tutorial 4: Model-Based Reinforcement Learning — Neuromatch Academy: Computational Neuroscience,” Neuromatch.io. https://compneuro.neuromatch.io/tutorials/W3D4_ReinforcementLearning/student/W3D4_Tutorial4.html
- “Multi-Agent Deep Deterministic Policy Gradient (MADDPG) – AgileRL Documentation,” Agilerl.com, 2023. https://docs.agilerl.com/en/latest/api/algorithms/maddpg.html
- “Multi-Agent Reinforcement Learning (PPO) with TorchRL Tutorial — torchrl 0.4 documentation,” Pytorch.org, 2022. https://docs.pytorch.org/rl/0.4/tutorials/multiagent_ppo.html
- A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative Q-Learning for Offline Reinforcement Learning,” June 08, 2020. https://arxiv.org/abs/2006.04779v3
- [1]
- Pieter Abbeel, “Inverse Reinforcement Learning,” 2010, https://people.eecs.berkeley.edu/~pabbeel/cs287-fa12/slides/inverseRL.pdf.