• Home
  • About
    • James Bockman photo

      James Bockman

      A place to organise my projects and research

    • Learn More
    • Email
    • LinkedIn
    • Instagram
    • Github
    • Youtube
  • Posts
    • All Posts
    • All Tags
  • Projects

[Reading] DeepLab

27 Mar 2019

Reading time ~4 minutes

DeepLab

L. Chen et al. developed a network capable of high-performance semantic segmentation built using a deep fully convolution network and in early versions a conditional random field. The architecture has been refined over time with several iterations, each achieving state of the art results on benchmarks at the time. The network itself makes use of atrous convolution to allow fast inference times in comparison to networks of similar depth, while maintaining a large receptive field size. The fully connected conditional random field acted as a post processing step on the convolutional networks dense predictions to localise Labels to boundaries of objects in the original image. This step is discarded for versions 3 and 3+ of the network in favour of an atrous version of spatial pyramid pooling.

Atrous/Dilated Convolution

This form of convolution is analogous to standard convolution with holes at specific intervals in the filter the image is being convoled with. By allowing these holes the overall size of the convolution’s filter can remain wide, capturing long range patterns in the image, without needing to store and update as many parameters as the filter is learnt. As an example a traditional 3x3 kernel could have a dilation rate or input stride of 2 producing a receptive field that would span a 5x5 section of the original image, while still using the same 9 parameters of the 3x3. An illustration of this example is seen below. [1-2]

Visualisation of a 3x3 kernel with dilation rate/input stride of 2 convolving with an image. [1]

Visualisation of a 3x3 kernel with dilation rate/input stride of 2 convolving with an image. [1]

Conditional Random Fields

Intuitively the fully connected CRF can be viewed as a form of bilateral filtering, persevering edges and Gaussian filtering to apply smoothing combined. The parameters of the CRF are optimised using cross-validation over a set number of iterations and are therefore not learnt as the network trains. This post-processing step allowed for edge localisation and smoothing of output in DeepLab 1 and 2. [1-3]

Atrous Spatial Pyramid Pooling

Frist seen in version 2 of DeepLab, ASPP uses atrous convolutions of different input strides in parallel to capture objects of different sizes within the image. [4] DeepLab 3 refined the ASSP process by including batch normalisation parameters that are finetuned over the course of the networks training and an image pooling layer is added in parallel with the ASSP. These changes enabled version 3 to outperform its predecessor without using CRFs at all. The specific configuration of the ASSP used in version 3 are to use stride rates of 6, 12 and 18 in parallel with an image pooling layer and 1x1 convolution. The resulting tensors are then stacked and passed through another 1x1 convolution layer to reduce the dimensionality of the channels. [5]

DeepLabv3+

Building on the success of many encoder, decoder style fully convolutional networks DeepLabv3+ takes DeepLabv3's network and uses it as an encoder. A decoder is then added orthogonally receiving input from the encoder prior to and after the ASPP. These two inputs are concatenated, subjected to a dimensionality reducing 3x3 convolution and the up sampled to produce the final prediction.

A version of this network where the backbone network was changed from ResNet-101 to a modified version of the Aligned Xception network was demonstrated to have a significant performance advantage. The Xception network is made deeper, max pooling operations are replaced by atrous separable convolution, more batch normalisation and ReLU activations are added after every depth wise convolution prior to its use in DeepLabv3+.

Even though the decoder network appears somewhat small its inclusion allows for a specific part of the network to learn transforms that preserve object boundaries more effectively than simple up sampling techniques used in prior implementations. [6]

References

[1] Vincent Dumoulin, Francesco Visin - A guide to convolution arithmetic for deep learning, arXiv:1603.07285

[2] Chen, L. Papandreou, G. Kokkinos, I. Murphy, K. & Yuille, A. L. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs, In ICLR, 2015

[3] Krähenbühl, P. & Koltun, V. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials CoRR, 2012 , abs/1210.5644

[4] Chen, L. Papandreou, G. Kokkinos, I. Murphy, K. & Yuille, A. L. DeepLab: Semantic Segmentation with Deep Convolution Nets, Atrous Convolution and Fully Connected CRFs, CoRR, 2016, abs/1606.00915

[5] Chen, L. Papandreou, G. Schroff, F. & Adam, H. Rethinking Atrous Convolution for Semantic Image Segmentation CoRR, 2017 , abs/1706.05587

[6] Chen, L. Zhu, Y. Papandreou, G. Schroff, F. & Adam, H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation CoRR, 2018 , abs/1802.02611



Atrous Convolution, Dialated Convolution, Semantic Segmentation, Segmentation, DeepLab Share Tweet +1