聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现

聊一聊计算机视觉中常用的注意力机制以及Pytorch代码实现

注意力机制(Attention)是深度学习中常用的tricks,可以在模型原有的基础上直接插入,进一步增强你模型的性能。注意力机制起初是作为自然语言处理中的工作Attention Is All You Need被大家所熟知,从而也引发了一系列的XX is All You Need的论文命题,SENET-Squeeze-and-Excitation Networks是注意力机制在计算机视觉中应用的早期工作之一,并获得了2017年imagenet, 同时也是最后一届Imagenet比赛的冠军,后面就又出现了各种各样的注意力机制,应用在计算机视觉的任务中,今天我们就来一起聊一聊计算机视觉中常用的注意力机制以及他们对应的Pytorch代码实现,另外我还使用这些注意力机制做了一些目标检测的实验,实验效果我也一并放在博客中,大家可以一起对自己感兴趣的部分讨论讨论。

新出的手把手教程,感兴趣的兄弟们快去自己动手试试看!

手把手教你使用YOLOV5训练自己的目标检测模型-口罩检测-视频教程_dejahu的博客-CSDN博客

这里是我数据集的基本情况,这里我使用的是交通标志检测的数据集

CocoDataset Train dataset with number of images 2226, and instance counts: 
+------------+-------+-----------+-------+-----------+-------+-----------------------------+-------+---------------------+-------+
| category   | count | category  | count | category  | count | category                    | count | category            | count |
+------------+-------+-----------+-------+-----------+-------+-----------------------------+-------+---------------------+-------+
| 0 [red_tl] | 1465  | 1 [arr_s] | 1133  | 2 [arr_l] | 638   | 3 [no_driving_mark_allsort] | 622   | 4 [no_parking_mark] | 1142  |
+------------+-------+-----------+-------+-----------+-------+-----------------------------+-------+---------------------+-------+

baseline选择的是fasterrcnn,实验的结果如下:

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.341
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.502
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.400
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.115
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.473
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.655
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.417
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.417
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.417
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.156
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.570
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.726

如果大家遇到论文下载比较慢

推荐使用中科院的 arxiv 镜像: http://xxx.itp.ac.cn, 国内网络能流畅访问
简单直接的方法是, 把要访问 arxiv 链接中的域名从 https://arxiv.org 换成 http://xxx.itp.ac.cn , 比如:

https://arxiv.org/abs/1901.07249 改为 http://xxx.itp.ac.cn/abs/1901.07249

1. SeNet: Squeeze-and-Excitation Attention

论文地址:https://arxiv.org/abs/1709.01507

  • 网络结构

    对通道做注意力机制,通过全连接层对每个通道进行加权。

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第1张图片

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第2张图片

  • Pytorch代码

    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init
    
    
    class SEAttention(nn.Module):
    
        def __init__(self, channel=512, reduction=16):
            super().__init__()
            self.avg_pool = nn.AdaptiveAvgPool2d(1)
            self.fc = nn.Sequential(
                nn.Linear(channel, channel // reduction, bias=False),
                nn.ReLU(inplace=True),
                nn.Linear(channel // reduction, channel, bias=False),
                nn.Sigmoid()
            )
    
        def init_weights(self):
            for m in self.modules():
                if isinstance(m, nn.Conv2d):
                    init.kaiming_normal_(m.weight, mode='fan_out')
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
                elif isinstance(m, nn.BatchNorm2d):
                    init.constant_(m.weight, 1)
                    init.constant_(m.bias, 0)
                elif isinstance(m, nn.Linear):
                    init.normal_(m.weight, std=0.001)
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
    
        def forward(self, x):
            b, c, _, _ = x.size()
            y = self.avg_pool(x).view(b, c)
            y = self.fc(y).view(b, c, 1, 1)
            return x * y.expand_as(x)
    
    
    if __name__ == '__main__':
        input = torch.randn(50, 512, 7, 7)
        se = SEAttention(channel=512, reduction=8)
        output = se(input)
        print(output.shape)
    
    
  • 实验结果

     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.338
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.511
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.375
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.126
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.458
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.696
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.411
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.411
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.411
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.163
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.551
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.758
    

2. (有用)CBAM: Convolutional Block Attention Module

论文地址:CBAM: Convolutional Block Attention Module

  • 网络结构

    对通道方向上做注意力机制之后再对空间方向上做注意力机制

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第3张图片

  • Pytorch代码

    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init
    
    
    class ChannelAttention(nn.Module):
        def __init__(self, channel, reduction=16):
            super().__init__()
            self.maxpool = nn.AdaptiveMaxPool2d(1)
            self.avgpool = nn.AdaptiveAvgPool2d(1)
            self.se = nn.Sequential(
                nn.Conv2d(channel, channel // reduction, 1, bias=False),
                nn.ReLU(),
                nn.Conv2d(channel // reduction, channel, 1, bias=False)
            )
            self.sigmoid = nn.Sigmoid()
    
        def forward(self, x):
            max_result = self.maxpool(x)
            avg_result = self.avgpool(x)
            max_out = self.se(max_result)
            avg_out = self.se(avg_result)
            output = self.sigmoid(max_out + avg_out)
            return output
    
    
    class SpatialAttention(nn.Module):
        def __init__(self, kernel_size=7):
            super().__init__()
            self.conv = nn.Conv2d(2, 1, kernel_size=kernel_size, padding=kernel_size // 2)
            self.sigmoid = nn.Sigmoid()
    
        def forward(self, x):
            max_result, _ = torch.max(x, dim=1, keepdim=True)
            avg_result = torch.mean(x, dim=1, keepdim=True)
            result = torch.cat([max_result, avg_result], 1)
            output = self.conv(result)
            output = self.sigmoid(output)
            return output
    
    
    class CBAMBlock(nn.Module):
    
        def __init__(self, channel=512, reduction=16, kernel_size=49):
            super().__init__()
            self.ca = ChannelAttention(channel=channel, reduction=reduction)
            self.sa = SpatialAttention(kernel_size=kernel_size)
    
        def init_weights(self):
            for m in self.modules():
                if isinstance(m, nn.Conv2d):
                    init.kaiming_normal_(m.weight, mode='fan_out')
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
                elif isinstance(m, nn.BatchNorm2d):
                    init.constant_(m.weight, 1)
                    init.constant_(m.bias, 0)
                elif isinstance(m, nn.Linear):
                    init.normal_(m.weight, std=0.001)
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
    
        def forward(self, x):
            b, c, _, _ = x.size()
            residual = x
            out = x * self.ca(x)
            out = out * self.sa(out)
            return out + residual
    
    
    if __name__ == '__main__':
        input = torch.randn(50, 512, 7, 7)
        kernel_size = input.shape[2]
        cbam = CBAMBlock(channel=512, reduction=16, kernel_size=kernel_size)
        output = cbam(input)
        print(output.shape)
    
    
  • 实验结果

     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.364
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.544
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.425
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.137
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.499
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.674
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.439
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.439
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.439
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.185
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.590
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.755
    

3. BAM: Bottleneck Attention Module

论文地址:https://arxiv.org/pdf/1807.06514.pdf

  • 网络结构

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第4张图片

  • Pytorch代码

    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init
    
    
    class Flatten(nn.Module):
        def forward(self, x):
            return x.view(x.shape[0], -1)
    
    
    class ChannelAttention(nn.Module):
        def __init__(self, channel, reduction=16, num_layers=3):
            super().__init__()
            self.avgpool = nn.AdaptiveAvgPool2d(1)
            gate_channels = [channel]
            gate_channels += [channel // reduction] * num_layers
            gate_channels += [channel]
    
            self.ca = nn.Sequential()
            self.ca.add_module('flatten', Flatten())
            for i in range(len(gate_channels) - 2):
                self.ca.add_module('fc%d' % i, nn.Linear(gate_channels[i], gate_channels[i + 1]))
                self.ca.add_module('bn%d' % i, nn.BatchNorm1d(gate_channels[i + 1]))
                self.ca.add_module('relu%d' % i, nn.ReLU())
            self.ca.add_module('last_fc', nn.Linear(gate_channels[-2], gate_channels[-1]))
    
        def forward(self, x):
            res = self.avgpool(x)
            res = self.ca(res)
            res = res.unsqueeze(-1).unsqueeze(-1).expand_as(x)
            return res
    
    
    class SpatialAttention(nn.Module):
        def __init__(self, channel, reduction=16, num_layers=3, dia_val=2):
            super().__init__()
            self.sa = nn.Sequential()
            self.sa.add_module('conv_reduce1',
                               nn.Conv2d(kernel_size=1, in_channels=channel, out_channels=channel // reduction))
            self.sa.add_module('bn_reduce1', nn.BatchNorm2d(channel // reduction))
            self.sa.add_module('relu_reduce1', nn.ReLU())
            for i in range(num_layers):
                self.sa.add_module('conv_%d' % i, nn.Conv2d(kernel_size=3, in_channels=channel // reduction,
                                                            out_channels=channel // reduction, padding=1, dilation=dia_val))
                self.sa.add_module('bn_%d' % i, nn.BatchNorm2d(channel // reduction))
                self.sa.add_module('relu_%d' % i, nn.ReLU())
            self.sa.add_module('last_conv', nn.Conv2d(channel // reduction, 1, kernel_size=1))
    
        def forward(self, x):
            res = self.sa(x)
            res = res.expand_as(x)
            return res
    
    
    class BAMBlock(nn.Module):
    
        def __init__(self, channel=512, reduction=16, dia_val=2):
            super().__init__()
            self.ca = ChannelAttention(channel=channel, reduction=reduction)
            self.sa = SpatialAttention(channel=channel, reduction=reduction, dia_val=dia_val)
            self.sigmoid = nn.Sigmoid()
    
        def init_weights(self):
            for m in self.modules():
                if isinstance(m, nn.Conv2d):
                    init.kaiming_normal_(m.weight, mode='fan_out')
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
                elif isinstance(m, nn.BatchNorm2d):
                    init.constant_(m.weight, 1)
                    init.constant_(m.bias, 0)
                elif isinstance(m, nn.Linear):
                    init.normal_(m.weight, std=0.001)
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
    
        def forward(self, x):
            b, c, _, _ = x.size()
            sa_out = self.sa(x)
            ca_out = self.ca(x)
            weight = self.sigmoid(sa_out + ca_out)
            out = (1 + weight) * x
            return out
    
    
    if __name__ == '__main__':
        input = torch.randn(50, 512, 7, 7)
        bam = BAMBlock(channel=512, reduction=16, dia_val=2)
        output = bam(input)
        print(output.shape)
    
  • 实验结果

4. (有用)ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

论文地址:https://arxiv.org/pdf/1910.03151.pdf

  • 网络结构

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第5张图片

  • Pytorch代码

    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init
    from collections import OrderedDict
    
    
    class ECAAttention(nn.Module):
    
        def __init__(self, kernel_size=3):
            super().__init__()
            self.gap = nn.AdaptiveAvgPool2d(1)
            self.conv = nn.Conv1d(1, 1, kernel_size=kernel_size, padding=(kernel_size - 1) // 2)
            self.sigmoid = nn.Sigmoid()
    
        def init_weights(self):
            for m in self.modules():
                if isinstance(m, nn.Conv2d):
                    init.kaiming_normal_(m.weight, mode='fan_out')
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
                elif isinstance(m, nn.BatchNorm2d):
                    init.constant_(m.weight, 1)
                    init.constant_(m.bias, 0)
                elif isinstance(m, nn.Linear):
                    init.normal_(m.weight, std=0.001)
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
    
        def forward(self, x):
            y = self.gap(x)  # bs,c,1,1
            y = y.squeeze(-1).permute(0, 2, 1)  # bs,1,c
            y = self.conv(y)  # bs,1,c
            y = self.sigmoid(y)  # bs,1,c
            y = y.permute(0, 2, 1).unsqueeze(-1)  # bs,c,1,1
            return x * y.expand_as(x)
    
    
    if __name__ == '__main__':
        input = torch.randn(50, 512, 7, 7)
        eca = ECAAttention(kernel_size=3)
        output = eca(input)
        print(output.shape)
    
  • 实验结果

    2021-12-17 12:18:08,911 - mmdet - INFO - 
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.360
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.545
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.414
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.141
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.489
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.676
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.432
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.432
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.432
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.184
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.576
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.748
    

5. SA-NET: SHUFFLE ATTENTION FOR DEEP CONVOLUTIONAL NEURAL NETWORKS

论文地址:https://arxiv.org/pdf/2102.00240.pdf

  • 网络结构

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第6张图片

  • Pytorch代码

    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init
    from torch.nn.parameter import Parameter
    
    
    class ShuffleAttention(nn.Module):
    
        def __init__(self, channel=512, reduction=16, G=8):
            super().__init__()
            self.G = G
            self.channel = channel
            self.avg_pool = nn.AdaptiveAvgPool2d(1)
            self.gn = nn.GroupNorm(channel // (2 * G), channel // (2 * G))
            self.cweight = Parameter(torch.zeros(1, channel // (2 * G), 1, 1))
            self.cbias = Parameter(torch.ones(1, channel // (2 * G), 1, 1))
            self.sweight = Parameter(torch.zeros(1, channel // (2 * G), 1, 1))
            self.sbias = Parameter(torch.ones(1, channel // (2 * G), 1, 1))
            self.sigmoid = nn.Sigmoid()
    
        def init_weights(self):
            for m in self.modules():
                if isinstance(m, nn.Conv2d):
                    init.kaiming_normal_(m.weight, mode='fan_out')
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
                elif isinstance(m, nn.BatchNorm2d):
                    init.constant_(m.weight, 1)
                    init.constant_(m.bias, 0)
                elif isinstance(m, nn.Linear):
                    init.normal_(m.weight, std=0.001)
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
    
        @staticmethod
        def channel_shuffle(x, groups):
            b, c, h, w = x.shape
            x = x.reshape(b, groups, -1, h, w)
            x = x.permute(0, 2, 1, 3, 4)
    
            # flatten
            x = x.reshape(b, -1, h, w)
    
            return x
    
        def forward(self, x):
            b, c, h, w = x.size()
            # group into subfeatures
            x = x.view(b * self.G, -1, h, w)  # bs*G,c//G,h,w
    
            # channel_split
            x_0, x_1 = x.chunk(2, dim=1)  # bs*G,c//(2*G),h,w
    
            # channel attention
            x_channel = self.avg_pool(x_0)  # bs*G,c//(2*G),1,1
            x_channel = self.cweight * x_channel + self.cbias  # bs*G,c//(2*G),1,1
            x_channel = x_0 * self.sigmoid(x_channel)
    
            # spatial attention
            x_spatial = self.gn(x_1)  # bs*G,c//(2*G),h,w
            x_spatial = self.sweight * x_spatial + self.sbias  # bs*G,c//(2*G),h,w
            x_spatial = x_1 * self.sigmoid(x_spatial)  # bs*G,c//(2*G),h,w
    
            # concatenate along channel axis
            out = torch.cat([x_channel, x_spatial], dim=1)  # bs*G,c//G,h,w
            out = out.contiguous().view(b, -1, h, w)
    
            # channel shuffle
            out = self.channel_shuffle(out, 2)
            return out
    
    
    if __name__ == '__main__':
        input = torch.randn(50, 512, 7, 7)
        se = ShuffleAttention(channel=512, G=8)
        output = se(input)
        print(output.shape)
    
    
  • 实验结果

     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.350
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.523
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.401
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.123
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.479
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.662
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.424
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.424
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.424
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.160
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.576
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.733
    

6. Polarized Self-Attention: Towards High-quality Pixel-wise Regression

论文地址:https://arxiv.org/abs/2107.00782

  • 网络结构

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第7张图片

  • Pytorch代码

    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init
    
    
    class ParallelPolarizedSelfAttention(nn.Module):
    
        def __init__(self, channel=512):
            super().__init__()
            self.ch_wv = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
            self.ch_wq = nn.Conv2d(channel, 1, kernel_size=(1, 1))
            self.softmax_channel = nn.Softmax(1)
            self.softmax_spatial = nn.Softmax(-1)
            self.ch_wz = nn.Conv2d(channel // 2, channel, kernel_size=(1, 1))
            self.ln = nn.LayerNorm(channel)
            self.sigmoid = nn.Sigmoid()
            self.sp_wv = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
            self.sp_wq = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
            self.agp = nn.AdaptiveAvgPool2d((1, 1))
    
        def forward(self, x):
            b, c, h, w = x.size()
    
            # Channel-only Self-Attention
            channel_wv = self.ch_wv(x)  # bs,c//2,h,w
            channel_wq = self.ch_wq(x)  # bs,1,h,w
            channel_wv = channel_wv.reshape(b, c // 2, -1)  # bs,c//2,h*w
            channel_wq = channel_wq.reshape(b, -1, 1)  # bs,h*w,1
            channel_wq = self.softmax_channel(channel_wq)
            channel_wz = torch.matmul(channel_wv, channel_wq).unsqueeze(-1)  # bs,c//2,1,1
            channel_weight = self.sigmoid(self.ln(self.ch_wz(channel_wz).reshape(b, c, 1).permute(0, 2, 1))).permute(0, 2,
                                                                                                                     1).reshape(
                b, c, 1, 1)  # bs,c,1,1
            channel_out = channel_weight * x
    
            # Spatial-only Self-Attention
            spatial_wv = self.sp_wv(x)  # bs,c//2,h,w
            spatial_wq = self.sp_wq(x)  # bs,c//2,h,w
            spatial_wq = self.agp(spatial_wq)  # bs,c//2,1,1
            spatial_wv = spatial_wv.reshape(b, c // 2, -1)  # bs,c//2,h*w
            spatial_wq = spatial_wq.permute(0, 2, 3, 1).reshape(b, 1, c // 2)  # bs,1,c//2
            spatial_wq = self.softmax_spatial(spatial_wq)
            spatial_wz = torch.matmul(spatial_wq, spatial_wv)  # bs,1,h*w
            spatial_weight = self.sigmoid(spatial_wz.reshape(b, 1, h, w))  # bs,1,h,w
            spatial_out = spatial_weight * x
            out = spatial_out + channel_out
            return out
    
    
    class SequentialPolarizedSelfAttention(nn.Module):
    
        def __init__(self, channel=512):
            super().__init__()
            self.ch_wv = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
            self.ch_wq = nn.Conv2d(channel, 1, kernel_size=(1, 1))
            self.softmax_channel = nn.Softmax(1)
            self.softmax_spatial = nn.Softmax(-1)
            self.ch_wz = nn.Conv2d(channel // 2, channel, kernel_size=(1, 1))
            self.ln = nn.LayerNorm(channel)
            self.sigmoid = nn.Sigmoid()
            self.sp_wv = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
            self.sp_wq = nn.Conv2d(channel, channel // 2, kernel_size=(1, 1))
            self.agp = nn.AdaptiveAvgPool2d((1, 1))
    
        def forward(self, x):
            b, c, h, w = x.size()
    
            # Channel-only Self-Attention
            channel_wv = self.ch_wv(x)  # bs,c//2,h,w
            channel_wq = self.ch_wq(x)  # bs,1,h,w
            channel_wv = channel_wv.reshape(b, c // 2, -1)  # bs,c//2,h*w
            channel_wq = channel_wq.reshape(b, -1, 1)  # bs,h*w,1
            channel_wq = self.softmax_channel(channel_wq)
            channel_wz = torch.matmul(channel_wv, channel_wq).unsqueeze(-1)  # bs,c//2,1,1
            channel_weight = self.sigmoid(self.ln(self.ch_wz(channel_wz).reshape(b, c, 1).permute(0, 2, 1))).permute(0, 2,
                                                                                                                     1).reshape(
                b, c, 1, 1)  # bs,c,1,1
            channel_out = channel_weight * x
    
            # Spatial-only Self-Attention
            spatial_wv = self.sp_wv(channel_out)  # bs,c//2,h,w
            spatial_wq = self.sp_wq(channel_out)  # bs,c//2,h,w
            spatial_wq = self.agp(spatial_wq)  # bs,c//2,1,1
            spatial_wv = spatial_wv.reshape(b, c // 2, -1)  # bs,c//2,h*w
            spatial_wq = spatial_wq.permute(0, 2, 3, 1).reshape(b, 1, c // 2)  # bs,1,c//2
            spatial_wq = self.softmax_spatial(spatial_wq)
            spatial_wz = torch.matmul(spatial_wq, spatial_wv)  # bs,1,h*w
            spatial_weight = self.sigmoid(spatial_wz.reshape(b, 1, h, w))  # bs,1,h,w
            spatial_out = spatial_weight * channel_out
            return spatial_out
    
    
    if __name__ == '__main__':
        input = torch.randn(1, 512, 7, 7)
        psa = SequentialPolarizedSelfAttention(channel=512)
        output = psa(input)
        print(output.shape)
    
    
  • 实验结果

    2021-12-16 20:30:36,981 - mmdet - INFO - 
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.346
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.522
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.385
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.123
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.474
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.676
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.422
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.422
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.422
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.170
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.570
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.743
    

7. Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks

论文地址:https://arxiv.org/pdf/1905.09646.pdf

  • 网络结构

    主要是用在语义分割上,所以在检测上的效果一般,没有带来多少提升

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第8张图片

  • Pytorch代码

    import numpy as np
    import torch
    from torch import nn
    from torch.nn import init
    
    
    class SpatialGroupEnhance(nn.Module):
    
        def __init__(self, groups):
            super().__init__()
            self.groups = groups
            self.avg_pool = nn.AdaptiveAvgPool2d(1)
            self.weight = nn.Parameter(torch.zeros(1, groups, 1, 1))
            self.bias = nn.Parameter(torch.zeros(1, groups, 1, 1))
            self.sig = nn.Sigmoid()
            self.init_weights()
    
        def init_weights(self):
            for m in self.modules():
                if isinstance(m, nn.Conv2d):
                    init.kaiming_normal_(m.weight, mode='fan_out')
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
                elif isinstance(m, nn.BatchNorm2d):
                    init.constant_(m.weight, 1)
                    init.constant_(m.bias, 0)
                elif isinstance(m, nn.Linear):
                    init.normal_(m.weight, std=0.001)
                    if m.bias is not None:
                        init.constant_(m.bias, 0)
    
        def forward(self, x):
            b, c, h, w = x.shape
            x = x.view(b * self.groups, -1, h, w)  # bs*g,dim//g,h,w
            xn = x * self.avg_pool(x)  # bs*g,dim//g,h,w
            xn = xn.sum(dim=1, keepdim=True)  # bs*g,1,h,w
            t = xn.view(b * self.groups, -1)  # bs*g,h*w
    
            t = t - t.mean(dim=1, keepdim=True)  # bs*g,h*w
            std = t.std(dim=1, keepdim=True) + 1e-5
            t = t / std  # bs*g,h*w
            t = t.view(b, self.groups, h, w)  # bs,g,h*w
    
            t = t * self.weight + self.bias  # bs,g,h*w
            t = t.view(b * self.groups, 1, h, w)  # bs*g,1,h*w
            x = x * self.sig(t)
            x = x.view(b, c, h, w)
    
            return x
    
    
    if __name__ == '__main__':
        input = torch.randn(50, 512, 7, 7)
        sge = SpatialGroupEnhance(groups=8)
        output = sge(input)
        print(output.shape)
    
  • 实验结果

    2021-12-16 21:39:42,785 - mmdet - INFO - 
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.342
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.516
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.381
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.117
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.474
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.652
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.415
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.415
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.415
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.155
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.565
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.718
    

8. Coordinate Attention for Efficient Mobile Network Design

论文地址:https://arxiv.org/abs/2103.02907

  • 网络结构

    主要应用在轻量级网络上,在resnet系列上效果不好。

    聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第9张图片

  • Pytorch代码

    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    
    
    class h_sigmoid(nn.Module):
        def __init__(self, inplace=True):
            super(h_sigmoid, self).__init__()
            self.relu = nn.ReLU6(inplace=inplace)
    
        def forward(self, x):
            return self.relu(x + 3) / 6
    
    
    class h_swish(nn.Module):
        def __init__(self, inplace=True):
            super(h_swish, self).__init__()
            self.sigmoid = h_sigmoid(inplace=inplace)
    
        def forward(self, x):
            return x * self.sigmoid(x)
    
    
    class CoordAtt(nn.Module):
        def __init__(self, inp, oup, reduction=32):
            super(CoordAtt, self).__init__()
            self.pool_h = nn.AdaptiveAvgPool2d((None, 1))
            self.pool_w = nn.AdaptiveAvgPool2d((1, None))
    
            mip = max(8, inp // reduction)
    
            self.conv1 = nn.Conv2d(inp, mip, kernel_size=1, stride=1, padding=0)
            self.bn1 = nn.BatchNorm2d(mip)
            self.act = h_swish()
    
            self.conv_h = nn.Conv2d(mip, oup, kernel_size=1, stride=1, padding=0)
            self.conv_w = nn.Conv2d(mip, oup, kernel_size=1, stride=1, padding=0)
    
        def forward(self, x):
            identity = x
    
            n, c, h, w = x.size()
            x_h = self.pool_h(x)
            x_w = self.pool_w(x).permute(0, 1, 3, 2)
    
            y = torch.cat([x_h, x_w], dim=2)
            y = self.conv1(y)
            y = self.bn1(y)
            y = self.act(y)
    
            x_h, x_w = torch.split(y, [h, w], dim=2)
            x_w = x_w.permute(0, 1, 3, 2)
    
            a_h = self.conv_h(x_h).sigmoid()
            a_w = self.conv_w(x_w).sigmoid()
    
            out = identity * a_w * a_h
    
            return out
    
    
  • 实验结果

    2021-12-16 19:04:16,776 - mmdet - INFO - 
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.340
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.516
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.386
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.127
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.457
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.632
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.408
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.408
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.408
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.162
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.546
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.716
    

9. Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions

论文地址: https://arxiv.org/abs/2112.05561

  • 网络结构

    计算量特别大,效果一般

  • Pytorch代码

    class GAM_Attention(nn.Module):
        def __init__(self, in_channels, out_channels, rate=4):
            super(GAM_Attention, self).__init__()
    
            self.channel_attention = nn.Sequential(
                nn.Linear(in_channels, int(in_channels / rate)),
                nn.ReLU(inplace=True),
                nn.Linear(int(in_channels / rate), in_channels)
            )
    
            self.spatial_attention = nn.Sequential(
                nn.Conv2d(in_channels, int(in_channels / rate), kernel_size=7, padding=3),
                nn.BatchNorm2d(int(in_channels / rate)),
                nn.ReLU(inplace=True),
                nn.Conv2d(int(in_channels / rate), out_channels, kernel_size=7, padding=3),
                nn.BatchNorm2d(out_channels)
            )
    
        def forward(self, x):
            # print(x)
            b, c, h, w = x.shape
            x_permute = x.permute(0, 2, 3, 1).view(b, -1, c)
            x_att_permute = self.channel_attention(x_permute).view(b, h, w, c)
            x_channel_att = x_att_permute.permute(0, 3, 1, 2)
    
            x = x * x_channel_att
    
            x_spatial_att = self.spatial_attention(x).sigmoid()
            out = x * x_spatial_att
            # print(out)
    
            return out
    
  • 实验结果

    2021-12-16 16:14:20,693 - mmdet - INFO - 
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.350
     Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.530
     Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.399
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.131
     Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.481
     Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.683
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.424
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.424
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.424
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.171
     Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.575
     Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.745
    

更多注意力

参考:https://github.com/xmu-xiaoma666/External-Attention-pytorch

另外还有一些用在语义分割上面的结构,这里就不测试了,大家可以自行下去测试

双路注意力机制-DANET

论文标题:Fu_Dual_Attention_Network_for_Scene_Segmentation

论文地址:https://openaccess.thecvf.com/content_CVPR_2019/papers/Fu_Dual_Attention_Network_for_Scene_Segmentation_CVPR_2019_paper.pdf

时间:2019

相当于之前是并行的结构,现在改成了串行的结构然后做特征的concat

聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第10张图片

聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第11张图片

位置注意力-CCNET

在上面的danet上改的,主要是解决计算量的问题, 通过十字交叉的结构来解决

论文标题:CCNet: Criss-Cross Attention for Semantic Segmentation

论文地址:https://openaccess.thecvf.com/content_ICCV_2019/papers/Huang_CCNet_Criss-Cross_Attention_for_Semantic_Segmentation_ICCV_2019_paper.pdf

时:2019

聊一聊计算机视觉中常用的注意力机制 附Pytorch代码实现_第12张图片

找到我

你可以通过这些方式来寻找我。

B站:肆十二-

CSDN:肆十二

知乎:肆十二

微博:肆十二-

现在关注以后就是老朋友喽!

image-20211212195912911

你可能感兴趣的