智能感知与计算系列讲座
Lecture Series in Intelligent Perception and Computing 

题    目 （TITLE）：Local Transformer Attention is Equivalent to Convolution

讲 座 人 （SPEAKER）: 王井东 首席研究员，微软亚洲研究院

主 持 人 (CHAIR)：张兆翔 研究员

时    间 (TIME)：2021年7月2日（周五），15：00

地    点 (VENUE)： 智能化大厦16层1610会议室

报告摘要（ABSTRACT）：

Local Vision Transformer has been attracting a lot of interest. The major component in Local Vision Transformer, local attention, performs the attention separately over small local windows. In this talk, I present the analysis of local attention from sparse connectivity, weight sharing and dynamic weight. I show that local attention is equivalent to inhomogeneous dynamic depth-wise convolution. Experimental results on ImageNet classification, COCO object detection, and ADE segmentation indicate that inhomogeneous dynamic depth-wise convolution using convolution for dynamic weight prediction outperforms local vision transformer, Swin Transformer, in the tiny model case, and performs almost the same for the base model case. In addition, I provide a relation graph to explain the relations between various networks, including recently-developed MLP-based models, ViT, and convolution-based models.

报告人简介（BIOGRAPHY）：

Jingdong Wang is a Senior Principal Research Manager with the Visual Computing Group at Microsoft Research Asia, Beijing, China. He received the B.Eng. and M.Eng. degrees from the Department of Automation at Tsinghua University in 2001 and 2004, respectively, and the PhD degree from the Department of Computer Science and Engineering, the Hong Kong University of Science and Technology, Hong Kong, in 2007. His areas of interest include neural network design, human pose estimation, large-scale indexing, and person re-identification. He is/was an Associate Editor of the IEEE TPAMI, the IEEE TMM and the IEEE TCSVT, and is an area chair of several leading Computer Vision and AI conferences, such as CVPR, ICCV, ECCV, ACM MM, IJCAI, and AAAI. He was elected as an IAPR Fellow, an ACM Distinguished Member, and an Industrial Distinguished Lecturer Program (iDLP) speaker of the IEEE Circuits and Systems Society.