Reinforcement Learning Basics

What you will learn in this lesson:

• Why reinforcement learning is a fundamentally different problem setup than supervised learning
• The agent, environment, state, action, reward, and policy vocabulary
• The exploration-exploitation tradeoff and why it has no equivalent in supervised learning
• The Q-learning update rule, explained term by term
• A small runnable Q-learning example that visibly improves over many episodes

A Different Kind of Problem

Every algorithm so far in this course learns from a fixed dataset where the correct answer is either given (supervised learning) or doesn't exist at all (unsupervised learning). Reinforcement learning is neither. There is no dataset of correct answers to learn from. Instead, an agent takes actions inside an environment, and the only feedback it gets is a reward signal, often delayed, that tells it how well things are going, not what it should have done differently.

Consider teaching a program to play a board game. There is no labeled dataset saying "in this exact position, the correct move is X." The program only finds out it won or lost at the very end of the game, dozens of moves after the decisions that actually mattered. Figuring out which of those many moves deserves credit for the eventual win is called the credit assignment problem, and it has no equivalent in supervised learning, where every training example already comes with its answer attached.

The Core Vocabulary

The agent's goal is to learn a policy that maximizes its expected return over time, not just the immediate reward from the next single action. A policy that grabs a small reward right now but walks into a much larger penalty two steps later is a bad policy, even though its very next reward looked good.

Exploration vs. Exploitation

At any point, the agent can exploit what it currently believes is the best action, or explore by trying something else to find out if an even better option exists. Pure exploitation risks getting stuck on a mediocre strategy the agent found early and never improving further. Pure exploration never settles down enough to actually use what it has learned.

A common, simple strategy is epsilon-greedy: with probability epsilon (a small number like 0.1), take a random action instead of the currently-best-known one. Epsilon is often decreased over time, so the agent explores heavily early on, when it knows little, and exploits more as its estimates become more reliable. This tradeoff has no real parallel in supervised learning, where the training data is simply fixed and given to you. It only shows up when the agent's own choices determine what data it gets to learn from next.

Q-Learning: Learning the Value of Actions

Q-learning learns a function Q(s, a), the expected total future return from taking action a in state s and acting optimally afterward. Once you know Q(s, a) for every state-action pair, the optimal policy is simple: in any state, take whichever action has the highest Q-value.

Q-values are learned through repeated experience using this update rule, applied after every action the agent takes:

Q(s, a) ← Q(s, a) + α · [ r + γ · max Q(s', a') − Q(s, a) ]

Reading this term by term: r is the reward just received. γ (gamma) is a discount factor between 0 and 1 that controls how much the agent values future reward versus immediate reward. max Q(s', a') is the best Q-value available from the new state s' the agent just landed in. α (alpha) is a learning rate, the same role it plays in gradient descent, controlling how big each update step is.

The bracketed term, r + γ · max Q(s', a') − Q(s, a), is the gap between what the agent expected (Q(s, a)) and a better estimate built from the reward it actually got plus the best value available from where it ended up. Q-learning nudges its estimate toward that better target a little bit at a time, every single step, across many episodes.

A Small Worked Example

Consider a tiny 1D world: 5 positions in a row, numbered 0 to 4. The agent starts at position 0 and can move left or right. Position 4 is the goal, worth a reward of +10. Every other move costs −1, which encourages the agent to reach the goal quickly rather than wander indefinitely.

import numpy as np

np.random.seed(0)
n_states = 5
n_actions = 2  # 0 = left, 1 = right
goal_state = 4

Q = np.zeros((n_states, n_actions))
alpha = 0.5
gamma = 0.9
epsilon = 0.2
n_episodes = 200

steps_per_episode = []

for episode in range(n_episodes):
    state = 0
    steps = 0
    while state != goal_state and steps < 50:
        if np.random.rand() < epsilon:
            action = np.random.randint(n_actions)
        else:
            action = np.argmax(Q[state])

        next_state = max(0, state - 1) if action == 0 else min(n_states - 1, state + 1)
        reward = 10 if next_state == goal_state else -1

        # The Q-learning update rule
        best_next_q = np.max(Q[next_state])
        Q[state, action] += alpha * (reward + gamma * best_next_q - Q[state, action])

        state = next_state
        steps += 1

    steps_per_episode.append(steps)

print("Average steps to reach goal, first 10 episodes:", np.mean(steps_per_episode[:10]))
print("Average steps to reach goal, last 10 episodes: ", np.mean(steps_per_episode[-10:]))
print("\nLearned Q-table (rows=position, columns=[left, right]):")
print(np.round(Q, 2))

Run this and compare the first 10 episodes' average steps to the last 10. The agent starts out wandering almost randomly and takes many steps to stumble onto the goal. After enough episodes, the Q-table converges toward consistently preferring "right" from every position, since that is in fact the shortest path to the reward, and the average steps per episode should drop sharply.

How This Connects to the Rest of the Course

Q-learning is still fundamentally about minimizing an error, the gap inside the brackets of the update rule, which is the same spirit as every loss function you've minimized so far in this course. What is genuinely new is that the agent's own actions determine which states and rewards it even gets to see next, unlike a fixed training set that exists before you ever touch it. That single difference is why exploration, delayed reward, and credit assignment are uniquely reinforcement learning's problems to solve.