This book is licensed for the personal use of the buyer. Please do not share or resell the file. Short quotations with attribution are welcome.
The information in this book is provided for education. It is not legal, financial or other professional advice. Product and company names are used only to describe them and belong to their owners.
This is a free sample: the introduction and first chapter.
IntroductionIntroduction
Most machine learning learns from examples with correct answers. Reinforcement learning (RL) learns from experience: an agent tries actions, receives rewards or penalties, and gradually discovers which behaviour pays off. It is how computers learned to play board games and video games at superhuman level, how robots learn to walk in simulation, and part of how language models are shaped to be helpful.
This book explains reinforcement learning clearly and concretely. It starts with the simplest setting, choosing between slot machines, and builds up through Markov decision processes, value functions, dynamic programming, Monte Carlo and temporal-difference learning, Q-learning, deep reinforcement learning and policy gradients. Later chapters cover reinforcement learning from human feedback, safety problems such as reward hacking, real-world applications and when RL is the wrong tool.
Small grid-world examples make every algorithm concrete, and all calculations are shown.
How the book is organised
Chapters 1 to 8: the RL problem, bandits, exploration and rewards.
Chapters 9 to 17: Markov decision processes, returns, values, Bellman equations and dynamic programming.
Chapters 18 to 27: learning from experience: Monte Carlo, temporal difference, SARSA, Q-learning and function approximation.
Chapters 28 to 36: deep RL, policy gradients, actor–critic and modern methods.
Chapters 37 to 45: RL from human feedback, reward hacking, safe exploration, applications, practice, worked examples, code and evaluation.
Chapters 46 to 54: value iteration by hand, reward shaping, partial observability, hierarchies, multiple agents, curiosity, verifiable rewards, hyperparameters and debugging.
Chapters 55 and 56: long-term recommendations and the ethics of optimising behaviour.
Chapters 57 to 59: case studies and a checklist.
Chapter 1Learning From Rewards
An agent interacts with an environment. At each step it observes a state, takes an action, receives a reward and moves to a new state. Its goal is to maximise total reward over time.
Examples:
Agent
Actions
Reward
Game player
Moves
Points, winning
Robot
Motor commands
Progress towards a goal, minus energy use
Recommender
Items to show
Engagement or satisfaction
Thermostat
Heating levels
Comfort minus energy cost
You have reached the end of the free sample. The full book continues with 58 more chapters, plus the appendices.