强化学习算法：Actor-Critic方法

1. 技术分析

1.1 Actor-Critic概述

Actor-Critic结合策略和价值函数：

Actor-Critic架构
    Actor: 策略网络，输出动作概率
    Critic: 价值网络，评估状态价值
    
    优势:
        降低方差
        提高样本效率
        在线学习

1.2 Actor-Critic组成

组件	作用	更新目标
Actor	选择动作	最大化优势函数
Critic	评估价值	最小化TD误差

1.3 Actor-Critic变体

Actor-Critic变体
    A2C: 同步优势Actor-Critic
    A3C: 异步优势Actor-Critic
    DDPG: 深度确定性策略梯度
    TD3: 双延迟深度确定性策略梯度

2. 核心功能实现

2.1 Actor-Critic算法

import numpy as np

class ActorCritic:
    def __init__(self, actor, critic, actor_optimizer, critic_optimizer, gamma=0.99):
        self.actor = actor
        self.critic = critic
        self.actor_optimizer = actor_optimizer
        self.critic_optimizer = critic_optimizer
        self.gamma = gamma
    
    def compute_advantage(self, state, reward, next_state, done):
        value = self.critic(state)
        
        if done:
            target = reward
        else:
            target = reward + self.gamma * self.critic(next_state)
        
        advantage = target - value
        
        return advantage, target
    
    def train_step(self, state, action, reward, next_state, done):
        advantage, target = self.compute_advantage(state, reward, next_state, done)
        
        actor_loss = -np.log(self.actor(state)[action]) * advantage
        critic_loss = (target - self.critic(state)) ** 2
        
        self.actor_optimizer.step(actor_loss)
        self.critic_optimizer.step(critic_loss)
        
        return actor_loss, critic_loss
    
    def train(self, env, episodes=1000):
        for episode in range(episodes):
            state = env.reset()
            done = False
            
            while not done:
                action_probs = self.actor(state)
                action = np.random.choice(len(action_probs), p=action_probs)
                
                next_state, reward, done = env.step(action)
                
                self.train_step(state, action, reward, next_state, done)
                
                state = next_state

2.2 A2C算法

class A2C(ActorCritic):
    def __init__(self, actor, critic, actor_optimizer, critic_optimizer, gamma=0.99, num_workers=4):
        super().__init__(actor, critic, actor_optimizer, critic_optimizer, gamma)
        self.num_workers = num_workers
    
    def train(self, env, episodes=1000):
        for episode in range(episodes):
            states = []
            actions = []
            rewards = []
            next_states = []
            dones = []
            
            for _ in range(self.num_workers):
                state = env.reset()
                done = False
                
                while not done:
                    action_probs = self.actor(state)
                    action = np.random.choice(len(action_probs), p=action_probs)
                    
                    next_state, reward, done = env.step(action)
                    
                    states.append(state)
                    actions.append(action)
                    rewards.append(reward)
                    next_states.append(next_state)
                    dones.append(done)
                    
                    state = next_state
            
            total_actor_loss = 0
            total_critic_loss = 0
            
            for i in range(len(states)):
                advantage, target = self.compute_advantage(
                    states[i], rewards[i], next_states[i], dones[i]
                )
                
                actor_loss = -np.log(self.actor(states[i])[actions[i]]) * advantage
                critic_loss = (target - self.critic(states[i])) ** 2
                
                total_actor_loss += actor_loss
                total_critic_loss += critic_loss
            
            self.actor_optimizer.step(total_actor_loss / len(states))
            self.critic_optimizer.step(total_critic_loss / len(states))

2.3 Actor和Critic网络

class ActorNetwork:
    def __init__(self, state_dim, action_dim, hidden_dim=64):
        self.W1 = np.random.randn(state_dim, hidden_dim) * 0.01
        self.b1 = np.zeros(hidden_dim)
        self.W2 = np.random.randn(hidden_dim, action_dim) * 0.01
        self.b2 = np.zeros(action_dim)
    
    def forward(self, state):
        h = np.maximum(0, state @ self.W1 + self.b1)
        logits = h @ self.W2 + self.b2
        
        exp_logits = np.exp(logits - np.max(logits))
        probs = exp_logits / np.sum(exp_logits)
        
        return probs

class CriticNetwork:
    def __init__(self, state_dim, hidden_dim=64):
        self.W1 = np.random.randn(state_dim, hidden_dim) * 0.01
        self.b1 = np.zeros(hidden_dim)
        self.W2 = np.random.randn(hidden_dim, 1) * 0.01
        self.b2 = np.zeros(1)
    
    def forward(self, state):
        h = np.maximum(0, state @ self.W1 + self.b1)
        value = h @ self.W2 + self.b2
        
        return value[0]

3. 性能对比

3.1 Actor-Critic变体对比

方法	并行性	稳定性	样本效率
A2C	同步	高	中
A3C	异步	中	高
DDPG	同步	中	高

3.2 Actor-Critic vs 策略梯度

指标	REINFORCE	Actor-Critic
方差	高	低
偏差	无偏	有偏
样本效率	低	高

3.3 网络规模影响

隐藏层大小	性能	训练时间	过拟合风险
32	低	快	低
64	中	中	中
128	高	慢	高

4. 最佳实践

4.1 Actor-Critic选择

def choose_actor_critic_method(environment_type):
    if environment_type == 'continuous':
        return 'DDPG'
    elif environment_type == 'discrete':
        return 'A2C'
    else:
        return 'A2C'

class ActorCriticSelector:
    @staticmethod
    def select(config):
        methods = {
            'a2c': A2C,
            'a3c': A3C,
            'ddpg': DDPG
        }
        
        return methods[config['method']](**config.get('params', {}))

4.2 训练技巧

class ActorCriticTrainingTips:
    @staticmethod
    def separate_learning_rates(actor_lr=0.001, critic_lr=0.005):
        return {'actor_lr': actor_lr, 'critic_lr': critic_lr}
    
    @staticmethod
    def target_networks():
        return {'use_target': True, 'tau': 0.001}
    
    @staticmethod
    def gradient_clipping(max_norm=1.0):
        return {'max_norm': max_norm}

5. 总结

Actor-Critic是强化学习的主流方法：

1. Actor：策略网络，选择动作
2. Critic：价值网络，评估价值
3. A2C：同步训练，稳定可靠
4. DDPG：连续动作的标准方法

对比数据如下：

• A2C比REINFORCE样本效率更高
• DDPG是连续控制的首选
• 推荐使用分离的学习率
• 目标网络可以提高稳定性