增强学习（Reinforcement Learning）简介

增强学习（Reinforcement Learning）简介
增强学习是机器学习的一种范式，其核心目标是让智能体（Agent）通过与环境的交互，基于试错机制和延迟奖励反馈，学习如何选择最优动作以最大化长期累积回报。其核心要素包括：
• 状态（State）：描述环境的当前信息（如棋盘布局、机器人传感器数据）。

• 动作（Action）：智能体在特定状态下可执行的操作（如移动、下棋）。

• 奖励（Reward）：环境对动作的即时反馈信号（如得分增加或惩罚）。

• 策略（Policy）：从状态到动作的映射规则（如基于Q值选择动作）。

• 价值函数（Value Function）：预测某状态或动作的长期回报（如Q-Learning中的Q表）。

与监督学习不同，增强学习无需标注数据，而是通过探索-利用权衡（Exploration vs Exploitation）自主学习。

使用PyTorch实现深度Q网络（DQN）演示
以下以CartPole-v0（平衡杆环境）为例，展示完整代码及解释：

环境与依赖库

import gym
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
from collections import deque
import random# 初始化环境
env = gym.make('CartPole-v1')
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n

定义DQN网络

class DQN(nn.Module):def __init__(self, state_dim, action_dim):super(DQN, self).__init__()self.fc = nn.Sequential(nn.Linear(state_dim, 128),nn.ReLU(),nn.Linear(128, 64),nn.ReLU(),nn.Linear(64, action_dim))def forward(self, x):return self.fc(x)

经验回放缓冲区（Replay Buffer）

class ReplayBuffer:def __init__(self, capacity):self.buffer = deque(maxlen=capacity)def push(self, state, action, reward, next_state, done):self.buffer.append((state, action, reward, next_state, done))def sample(self, batch_size):batch = random.sample(self.buffer, batch_size)states, actions, rewards, next_states, dones = zip(*batch)return (torch.FloatTensor(states),torch.LongTensor(actions),torch.FloatTensor(rewards),torch.FloatTensor(next_states),torch.FloatTensor(dones))

训练参数与初始化

# 超参数
batch_size = 64
gamma = 0.99        # 折扣因子
epsilon_start = 1.0
epsilon_decay = 0.995
epsilon_min = 0.01
target_update = 10  # 目标网络更新频率# 初始化网络与优化器
policy_net = DQN(state_dim, action_dim)
target_net = DQN(state_dim, action_dim)
target_net.load_state_dict(policy_net.state_dict())
optimizer = optim.Adam(policy_net.parameters(), lr=0.001)
buffer = ReplayBuffer(10000)
epsilon = epsilon_start

训练循环

num_episodes = 500
for episode in range(num_episodes):state = env.reset()total_reward = 0while True:# ε-贪婪策略选择动作if random.random() < epsilon:action = env.action_space.sample()else:with torch.no_grad():q_values = policy_net(torch.FloatTensor(state))action = q_values.argmax().item()# 执行动作并存储经验next_state, reward, done, _ = env.step(action)buffer.push(state, action, reward, next_state, done)state = next_statetotal_reward += reward# 经验回放与网络更新if len(buffer.buffer) >= batch_size:states, actions, rewards, next_states, dones = buffer.sample(batch_size)# 计算目标Q值with torch.no_grad():next_q = target_net(next_states).max(1)[0]target_q = rewards + gamma * next_q * (1 - dones)# 计算当前Q值current_q = policy_net(states).gather(1, actions.unsqueeze(1))# 均方误差损失loss = nn.MSELoss()(current_q, target_q.unsqueeze(1))# 反向传播optimizer.zero_grad()loss.backward()optimizer.step()if done:break# 更新目标网络与εif episode % target_update == 0:target_net.load_state_dict(policy_net.state_dict())epsilon = max(epsilon_min, epsilon * epsilon_decay)print(f"Episode {episode}, Reward: {total_reward}, Epsilon: {epsilon:.2f}")