​
优先级经验回放（Prioritized Experience Replay, PER）是深度强化学习（DRL）中的一种优化技术，通过优先选择对学习更有价值的经验（通常基于 TD 误差）来提高训练效率和性能。相比随机采样的传统经验回放，PER 利用概率分布优先选择高误差的经验，结合重要性采样（Importance Sampling）修正偏差。在动态车间调度（Job Shop Scheduling, JSS）的 DQN（Deep Q-Network）场景中，PER 可以加速收敛并优化调度决策（最小化 Makespan）。

本文将基于之前的 C# 代码（使用 TensorFlow.NET），添加优先级经验回放，优化动态车间调度问题，处理新作业到达（时间t=5t=5t=5新增作业 J4）。代码将包含详细的注释，说明 PER 的实现、概率视角和数学推导，并提供图解。以下内容涵盖 PER 的理论、实现和与微积分/概率的结合。

1. 优先级经验回放（PER）理论

1.1 核心概念
是一个小正数（避免优先级为 0）。
控制优先级的强度（为均匀采样，为完全基于优先级）。

：控制修正强度，逐渐增加到 1。

经验池：[(s1, a1, r1, s1’, δ1), (s2, a2, r2, s2’, δ2), …]
优先级：p_i = |δ_i| + ε
采样概率：P(i) = p_i^α / Σ p_j^α
权重：w_i = (1/N / P(i))^β
[图：经验池，优先级分布，采样概率]
1.2 PER 在 DQN 中的作用重要性采样权重修正非均匀采样的偏差。
1.3 数学推导
对参数的梯度：链式法则分解，如：
Q(s, a; θ) -> TD 误差 δ -> 优先级 p_i -> 采样概率 P(i)
损失 L = Σ w_i * δ_i²
梯度：∂L/∂θ = Σ w_i * 2 * δ_i * ∂Q/∂θ
[图：PER 采样与梯度流]
2. C# 代码：添加优先级经验回放
以下代码基于 TensorFlow.NET 实现 DQN，优化动态车间调度问题，新增 PER 功能。代码包含详细注释，说明 PER 的实现和数学背景。

using System;
using System.Collections.Generic;
using System.Linq;
using Tensorflow;
using NumSharp;
using static Tensorflow.Binding;

namespace DynamicDQNScheduler
{
    /// <summary>
    /// 工序类，表示作业的一个工序。
    /// </summary>
    public class Operation
    {
        public int JobId { get; set; } // 作业ID
        public int MachineId { get; set; } // 机器ID
        public int Duration { get; set; } // 加工时间
    }

    /// <summary>
    /// 作业类，表示一个作业。
    /// </summary>
    public class Job
    {
        public int Id { get; set; } // 作业ID
        public List<Operation> Operations { get; set; } = new List<Operation>(); // 工序列表
        public int ArrivalTime { get; set; } // 到达时间
    }

    /// <summary>
    /// 经验类，包含优先级（基于 TD 误差）。
    /// </summary>
    public class Experience
    {
        public float[] State { get; set; } // 当前状态
        public int Action { get; set; } // 动作索引
        public float Reward { get; set; } // 奖励
        public float[] NextState { get; set; } // 下一状态
        public bool Done { get; set; } // 是否完成
        public float Priority { get; set; } // 优先级（|TD 误差| + ε）
    }

    /// <summary>
    /// 动态车间调度器，使用 DQN 和优先级经验回放优化调度。
    /// 数学背景：
    /// - Q 值：Q(s, a; θ) = W2 * ReLU(W1 * s + b1) + b2
    /// - 损失：L(θ) = Σ w_i * (r + γ max Q(s', a'; θ') - Q(s, a; θ))²
    /// - 优先级：p_i = |δ_i| + ε，采样概率：P(i) = p_i^α / Σ p_j^α
    /// - 重要性采样权重：w_i = (1/N / P(i))^β
    /// </summary>
    public class DynamicDQNScheduler
    {
        private readonly List<Job> jobs; // 作业列表
        private readonly int machineCount; // 机器数量
        private List<int> machineAvailableTime; // 机器可用时间
        private List<int> jobNextOperation; // 作业当前工序索引
        private List<int> jobLastEndTime; // 作业最后完工时间
        private readonly Random rand = new Random(); // 随机数生成器
        private readonly List<Experience> replayBuffer = new List<Experience>(); // 经验回放缓冲区
        private float epsilon = 0.5f; // ε-贪婪探索率
        private readonly float epsilonDecay = 0.995f; // ε衰减率
        private readonly float gamma = 0.9f; // 折扣因子
        private readonly float alpha = 0.6f; // 优先级指数
        private readonly float beta = 0.4f; // 重要性采样指数
        private readonly float betaIncrement = 0.001f; // β 增量
        private readonly float epsilonPriority = 0.01f; // 优先级小正数
        private readonly Graph graph = new Graph().as_default(); // TensorFlow 计算图
        private readonly Session session; // TensorFlow 会话
        private Operation trainOp; // 训练操作
        private Tensor stateInput; // 状态输入
        private Tensor qValues; // 主网络 Q 值
        private Tensor targetQ; // 目标 Q 值
        private Tensor weights; // 重要性采样权重
        private readonly int maxActions = 10; // 最大动作数量
        private readonly int targetUpdateFreq = 10; // 目标网络更新频率

        /// <summary>
        /// 构造函数，初始化调度器和神经网络。
        /// </summary>
        /// <param name="jobs">作业列表</param>
        /// <param name="machineCount">机器数量</param>
        public DynamicDQNScheduler(List<Job> jobs, int machineCount)
        {
            this.jobs = jobs;
            this.machineCount = machineCount;
            this.machineAvailableTime = new List<int>(new int[machineCount]);
            this.jobNextOperation = new List<int>(new int[jobs.Count]);
            this.jobLastEndTime = new List<int>(new int[jobs.Count]);
            InitializeNeuralNetwork();
            session = tf.Session();
            session.run(tf.global_variables_initializer());
        }

        /// <summary>
        /// 初始化主网络和目标网络。
        /// 数学背景：
        /// - 主网络：Q(s, a; θ) = W2 * ReLU(W1 * s + b1) + b2
        /// - 目标网络：Q(s', a'; θ') = tw2 * ReLU(tw1 * s' + tb1) + tb2
        /// - 损失：L(θ) = Σ w_i * (r + γ max Q(s', a'; θ') - Q(s, a; θ))²
        /// </summary>
        private void InitializeNeuralNetwork()
        {
            tf_with(graph, g =>
            {
                int stateSize = 1 + machineCount + jobs.Count;
                stateInput = tf.placeholder(tf.float32, shape: new Shape(-1, stateSize), name: "state");

                // 主网络
                var w1 = tf.get_variable("w1", shape: new Shape(stateSize, 64), initializer: tf.random_normal_initializer());
                var b1 = tf.get_variable("b1", shape: new Shape(64), initializer: tf.zeros_initializer());
                var h1 = tf.nn.relu(tf.matmul(stateInput, w1) + b1);
                var w2 = tf.get_variable("w2", shape: new Shape(64, maxActions), initializer: tf.random_normal_initializer());
                var b2 = tf.get_variable("b2", shape: new Shape(maxActions), initializer: tf.zeros_initializer());
                qValues = tf.matmul(h1, w2) + b2;

                // 目标网络
                var tw1 = tf.get_variable("tw1", shape: new Shape(stateSize, 64), initializer: tf.random_normal_initializer());
                var tb1 = tf.get_variable("tb1", shape: new Shape(64), initializer: tf.zeros_initializer());
                var th1 = tf.nn.relu(tf.matmul(stateInput, tw1) + tb1);
                var tw2 = tf.get_variable("tw2", shape: new Shape(64, maxActions), initializer: tf.random_normal_initializer());
                var tb2 = tf.get_variable("tb2", shape: new Shape(maxActions), initializer: tf.zeros_initializer());
                targetQValues = tf.matmul(th1, tw2) + tb2;

                // 损失函数（带权重）
                targetQ = tf.placeholder(tf.float32, shape: new Shape(-1, maxActions), name: "targetQ");
                weights = tf.placeholder(tf.float32, shape: new Shape(-1), name: "weights");
                var loss = tf.reduce_mean(weights * tf.square(targetQ - qValues));
                trainOp = tf.train.AdamOptimizer(0.001f).minimize(loss);
            });
        }

        /// <summary>
        /// 更新目标网络参数：θ' ← θ。
        /// </summary>
        private void UpdateTargetNetwork()
        {
            session.run(new[]
            {
                tf.get_variable("tw1").assign(tf.get_variable("w1")),
                tf.get_variable("tb1").assign(tf.get_variable("b1")),
                tf.get_variable("tw2").assign(tf.get_variable("w2")),
                tf.get_variable("tb2").assign(tf.get_variable("b2"))
            });
        }

        /// <summary>
        /// 添加新作业，动态扩展状态空间。
        /// </summary>
        /// <param name="newJob">新作业</param>
        public void AddJob(Job newJob)
        {
            jobs.Add(newJob);
            jobNextOperation.Add(0);
            jobLastEndTime.Add(0);
            Console.WriteLine($"时间 {newJob.ArrivalTime}：新作业 J{newJob.Id} 到达");
        }

        /// <summary>
        /// 获取状态向量：s = [时间, 机器可用时间, 作业进度]。
        /// </summary>
        /// <param name="currentTime">当前时间</param>
        /// <returns>状态向量</returns>
        private float[] GetState(int currentTime)
        {
            var state = new float[1 + machineCount + jobs.Count];
            state[0] = currentTime;
            for (int i = 0; i < machineCount; i++) state[i + 1] = machineAvailableTime[i];
            for (int i = 0; i < jobs.Count; i++) state[i + 1 + machineCount] = jobNextOperation[i];
            return state;
        }

        /// <summary>
        /// 获取可用动作（可调度的工序）。
        /// </summary>
        /// <param name="currentTime">当前时间</param>
        /// <returns>动作列表（作业、工序、索引）</returns>
        private List<(Job, Operation, int)> GetAvailableActions(int currentTime)
        {
            var actions = new List<(Job, Operation, int)>();
            int actionIndex = 0;
            for (int i = 0; i < jobs.Count; i++)
            {
                if (jobs[i].ArrivalTime <= currentTime && jobNextOperation[i] < jobs[i].Operations.Count)
                {
                    actions.Add((jobs[i], jobs[i].Operations[jobNextOperation[i]], actionIndex++));
                }
            }
            return actions;
        }

        /// <summary>
        /// 使用ε-贪婪策略选择动作，基于 Softmax 概率分布。
        /// 数学背景：P(a|s) = exp(Q(s, a)) / Σ exp(Q(s, a'))
        /// </summary>
        /// <param name="state">当前状态</param>
        /// <param name="actions">可用动作</param>
        /// <returns>动作索引</returns>
        private int SelectAction(float[] state, List<(Job, Operation, int)> actions)
        {
            if (rand.NextDouble() < epsilon)
                return actions.Count > 0 ? actions[rand.Next(actions.Count)].actionIndex : -1;

            var stateTensor = np.array(state).reshape(1, -1);
            var qVals = session.run(qValues, new FeedItem(stateInput, stateTensor))[0];
            var probs = Softmax(qVals);
            float maxProb = float.MinValue;
            int bestAction = -1;

            foreach (var action in actions)
            {
                if (probs[action.actionIndex] > maxProb)
                {
                    maxProb = probs[action.actionIndex];
                    bestAction = action.actionIndex;
                }
            }

            return bestAction;
        }

        /// <summary>
        /// 计算 Softmax 概率分布。
        /// 数学公式：P(a_i|s) = exp(Q_i) / Σ exp(Q_j)
        /// </summary>
        /// <param name="qValues">Q 值数组</param>
        /// <returns>概率分布</returns>
        private float[] Softmax(float[] qValues)
        {
            var expQ = qValues.Select(x => (float)Math.Exp(x)).ToArray();
            var sumExpQ = expQ.Sum();
            return expQ.Select(x => x / sumExpQ).ToArray();
        }

        /// <summary>
        /// 计算经验的优先级：p_i = |δ_i| + ε。
        /// </summary>
        /// <param name="experience">经验</param>
        /// <returns>优先级</returns>
        private float CalculatePriority(Experience experience)
        {
            var stateTensor = np.array(experience.State).reshape(1, -1);
            var nextStateTensor = np.array(experience.NextState).reshape(1, -1);
            var qVal = session.run(qValues, new FeedItem(stateInput, stateTensor))[0][experience.Action];
            var nextQ = session.run(targetQValues, new FeedItem(stateInput, nextStateTensor))[0].Max();
            var target = experience.Reward + (experience.Done ? 0 : gamma * nextQ);
            var tdError = Math.Abs(target - qVal);
            return (float)tdError + epsilonPriority;
        }

        /// <summary>
        /// 训练 Q 网络，使用优先级经验回放。
        /// 数学背景：
        /// - 优先级：p_i = |δ_i| + ε
        /// - 采样概率：P(i) = p_i^α / Σ p_j^α
        /// - 权重：w_i = (1/N / P(i))^β
        /// - 损失：L = Σ w_i * δ_i²
        /// </summary>
        private void TrainNetwork()
        {
            if (replayBuffer.Count < 32) return;

            // 计算采样概率
            var priorities = replayBuffer.Select(e => Math.Pow(e.Priority, alpha)).ToArray();
            var sumPriorities = priorities.Sum();
            var probabilities = priorities.Select(p => p / sumPriorities).ToArray();

            // 优先级采样
            var batchIndices = new List<int>();
            for (int i = 0; i < 32; i++)
            {
                double r = rand.NextDouble();
                double cumulative = 0;
                for (int j = 0; j < probabilities.Length; j++)
                {
                    cumulative += probabilities[j];
                    if (r <= cumulative)
                    {
                        batchIndices.Add(j);
                        break;
                    }
                }
            }

            var batch = batchIndices.Select(i => replayBuffer[i]).ToList();
            var states = batch.Select(e => e.State).ToArray();
            var actions = batch.Select(e => e.Action).ToList();
            var weightsBatch = batchIndices.Select(i => (float)Math.Pow(1.0 / replayBuffer.Count / probabilities[i], beta)).ToArray();
            var targets = new float[batch.Count, maxActions];

            // 计算目标 Q 值并更新优先级
            for (int i = 0; i < batch.Count; i++)
            {
                var experience = batch[i];
                var nextStateTensor = np.array(experience.NextState).reshape(1, -1);
                var nextQ = session.run(targetQValues, new FeedItem(stateInput, nextStateTensor))[0];
                float maxNextQ = nextQ.Max();
                targets[i, experience.Action] = experience.Reward + (experience.Done ? 0 : gamma * maxNextQ);
                experience.Priority = CalculatePriority(experience); // 更新优先级
            }

            var stateBatch = np.array(states);
            session.run(trainOp, new FeedItem(stateInput, stateBatch), new FeedItem(targetQ, targets), new FeedItem(weights, weightsBatch));
        }

        /// <summary>
        /// 运行 DQN 调度，优化 Makespan。
        /// </summary>
        /// <param name="maxSimulationTime">最大模拟时间</param>
        /// <returns>最佳 Makespan</returns>
        public int Run(int maxSimulationTime)
        {
            int episodeCount = 0;
            int bestMakespan = int.MaxValue;
            float currentBeta = beta;

            while (episodeCount < 50)
            {
                machineAvailableTime = new List<int>(new int[machineCount]);
                jobNextOperation = new List<int>(new int[jobs.Count]);
                jobLastEndTime = new List<int>(new int[jobs.Count]);
                int currentTime = 0;
                int stepCount = 0;

                while (currentTime < maxSimulationTime || jobs.Any((j, i) => jobNextOperation[i] < j.Operations.Count))
                {
                    // 动态事件：时间 t=5 添加新作业
                    if (currentTime == 5 && episodeCount == 0)
                    {
                        var newJob = new Job
                        {
                            Id = jobs.Count,
                            ArrivalTime = currentTime,
                            Operations = new List<Operation>
                            {
                                new Operation { JobId = jobs.Count, MachineId = 0, Duration = 2 },
                                new Operation { JobId = jobs.Count, MachineId = 1, Duration = 3 },
                                new Operation { JobId = jobs.Count, MachineId = 2, Duration = 1 }
                            }
                        };
                        AddJob(newJob);
                    }

                    var state = GetState(currentTime);
                    var actions = GetAvailableActions(currentTime);
                    if (actions.Count == 0)
                    {
                        currentTime++;
                        continue;
                    }

                    int actionIndex = SelectAction(state, actions);
                    if (actionIndex == -1) continue;

                    var (job, operation, _) = actions.Find(a => a.actionIndex == actionIndex);
                    int jobIndex = jobs.IndexOf(job);
                    int machineId = operation.MachineId;
                    int duration = operation.Duration;

                    int oldMakespan = machineAvailableTime.Max();
                    int startTime = Math.Max(machineAvailableTime[machineId], jobLastEndTime[jobIndex]);
                    startTime = Math.Max(startTime, currentTime);
                    int endTime = startTime + duration;

                    machineAvailableTime[machineId] = endTime;
                    jobLastEndTime[jobIndex] = endTime;
                    jobNextOperation[jobIndex]++;
                    currentTime = startTime;

                    int newMakespan = machineAvailableTime.Max();
                    float reward = -(newMakespan - oldMakespan);
                    var nextState = GetState(currentTime);
                    bool done = !jobs.Any((j, i) => jobNextOperation[i] < j.Operations.Count);

                    var experience = new Experience
                    {
                        State = state,
                        Action = actionIndex,
                        Reward = reward,
                        NextState = nextState,
                        Done = done,
                        Priority = 1.0f // 初始优先级
                    };
                    experience.Priority = CalculatePriority(experience);
                    replayBuffer.Add(experience);

                    TrainNetwork();
                    if (stepCount % targetUpdateFreq == 0) UpdateTargetNetwork();

                    Console.WriteLine($"时间 {startTime}：调度 J{job.Id}-工序{jobNextOperation[jobIndex]} 到 M{machineId}，加工时间：{duration}，完成时间：{endTime}");

                    epsilon = Math.Max(0.1f, epsilon * epsilonDecay);
                    currentBeta = Math.Min(1.0f, currentBeta + betaIncrement);
                    stepCount++;
                }

                int makespan = machineAvailableTime.Max();
                bestMakespan = Math.Min(bestMakespan, makespan);
                episodeCount++;
                Console.WriteLine($"第 {episodeCount} 次训练，Makespan：{makespan}");
            }

            return bestMakespan;
        }

        /// <summary>
        /// 测试程序，初始化作业和机器，运行调度器。
        /// </summary>
        public static void Main()
        {
            var jobs = new List<Job>
            {
                new Job
                {
                    Id = 0,
                    ArrivalTime = 0,
                    Operations = new List<Operation>
                    {
                        new Operation { JobId = 0, MachineId = 0, Duration = 3 },
                        new Operation { JobId = 0, MachineId = 1, Duration = 2 },
                        new Operation { JobId = 0, MachineId = 2, Duration = 5 }
                    }
                },
                new Job
                {
                    Id = 1,
                    ArrivalTime = 0,
                    Operations = new List<Operation>
                    {
                        new Operation { JobId = 1, MachineId = 1, Duration = 4 },
                        new Operation { JobId = 1, MachineId = 2, Duration = 1 },
                        new Operation { JobId = 1, MachineId = 0, Duration = 2 }
                    }
                },
                new Job
                {
                    Id = 2,
                    ArrivalTime = 0,
                    Operations = new List<Operation>
                    {
                        new Operation { JobId = 2, MachineId = 2, Duration = 2 },
                        new Operation { JobId = 2, MachineId = 0, Duration = 4 },
                        new Operation { JobId = 2, MachineId = 1, Duration = 3 }
                    }
                }
            };

            var scheduler = new DynamicDQNScheduler(jobs, 3);
            int makespan = scheduler.Run(20);
            Console.WriteLine($"最终 Makespan：{makespan}");
        }
    }
}

3. 代码说明

3.1 优先级经验回放实现

添加 Priority 属性，存储优先级
优先级：
逐渐从 0.4 增加到 1.0，平衡偏差。
3.2 概率与微积分
3.3 图解
经验池：[(s1, a1, r1, s1’, p1), (s2, a2, r2, s2’, p2), …]
优先级：p_i = |δ_i| + ε
采样概率：P(i) = p_i^α / Σ p_j^α
权重：w_i = (1/N / P(i))^β
[图：优先级分布，采样概率曲线]
Q(s, a; θ) -> δ -> p_i -> P(i) -> w_i
损失：L = Σ w_i * δ_i²
梯度：∂L/∂θ = Σ w_i * 2 * δ_i * ∂Q/∂θ
[图：Q 网络与梯度流]
3.4 与原代码的差异加入重要性采样权重。
增加和增量。
4. 安装与运行
dotnet add package TensorFlow.NET
dotnet add package NumSharp
dotnet run
5. 总结
数学：
Softmax 动作选择：
链式法则计算梯度：

如果您需要更深入的 PER 优化（如 SumTree 实现）、更详细的数学推导（如权重对梯度的影响）、或可视化结果（如甘特图），请告诉我，我可以进一步扩展内容！

​