注：对于含有rnn的模型怎样设置参数不回传

在模型蒸馏的常规训练中，大模型的参数都是设置不更新回传，使用

self.model_T.eval()

就可以达到固定大模型参数的效果；但是当模型网络结构中包含有rnn，例：

class BidirectionalLSTM(nn.Module):

    def __init__(self, nIn, nHidden, nOut):
        super(BidirectionalLSTM, self).__init__()

        self.rnn = nn.LSTM(nIn, nHidden, bidirectional=True)
        self.embedding = nn.Linear(nHidden * 2, nOut)

    def forward(self, input):
        recurrent, _ = self.rnn(input)
        T, b, h = recurrent.size()
        t_rec = recurrent.view(T * b, h)

        output = self.embedding(t_rec)  # [T * b, nOut]
        output = output.view(T, b, -1)

        return output #, recurrent

使用eval模式就会出现如下报错：

RuntimeError: cudnn RNN backward can only be called in training mode

当前的解决办法就是只能使用train模式，然后将所有网络参数固定。到底该如何操作？当前采用了如下方法：

for k,v in self.model_T.named_parameters():
    v.requires_grad=False#固定参数
    for name, module in self.model_T.named_modules():
        if isinstance(module, nn.BatchNorm2d):
            module.training = False
        elif isinstance(module, nn.BatchNorm1d):
            module.training = False

在训练过程中，为了监测大模型参数是否发生变化，可以将最初的大模型和中间存储的模型参数，读取出来进行，参数比较，查看大模型的参数是否发生改变：

if os.path.exists(modelpath_ori) and os.path.exists(modelpath_dis):
    print('Load model from "%s" and "%s"...', modelpath_ori, modelpath_dis)

    backbone_dict = model.state_dict()
    pretrained_dict_ori = torch.load(modelpath_ori, map_location=torch.device('cpu'))
    pretrained_dict_dis = torch.load(modelpath_dis, map_location=torch.device('cpu'))

    pretrained_dict_backbone_ = {}
    for (k_ori, v_ori), (k_dis, v_dis) in zip(pretrained_dict_ori.items(),pretrained_dict_dis.items()):
        k_ = k_ori.replace("module.", "")

        if k_ in backbone_dict:  # and k not in ['rnn.1.embedding.weight', 'rnn.1.embedding.bias']:
             diff = v_ori - v_dis
             zero_value = torch.zeros(diff.size()).type(diff.type())
             # diff.gt(0.0)
             if not torch.equal(diff, zero_value):
                 print(k_)
             else:
                 print('not k', k_)

参考链接：

1.https://zhuanlan.zhihu.com/p/258906996

2.https://www.jianshu.com/p/fcafcfb3d887