Skip to content

rajatsaini0294/lambda-mobilenets

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Lambda Mobilenets

Mobilenets with Lambda layers

Lambda Networks proposed in LambdaNetworks: Modeling Long Range Interactions without Attention.

Install

We use the implementation done by lucidrains

pip install lambda-networks

Method

We replace some layers in MobileNet-v1 architecture with lambda layer. It significantly reduces the parameters with some performance gain on cifar100 dataset.

MobileNet-v1 architecture:

The following table shows which layer to replace and remove to get the performance boost:

C: Conv layer same as in original architecture

L: Lambda layer

Blank cell represents that the layer is removed (replaced with identity layer)

  Layer number and Type    
Id 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Params (M) Top-1 (%)
A1 C C C C C C C C C C C C C C FC 3.30 65.54
A2 C C C C C C C C         L C FC 1.84 69.48
A3 C C C C C C C C L C L C L C FC 2.53 65.25
A4 C C C C C C C C L       C   FC 1.24 68.22
A5 C C C C C C L           C   FC 0.80 69.91
A6 C C C   C   L           C   FC 0.71 66.38

The above table shows significant gain in A5 configuration as compared to original configuration A1.

Citations

@inproceedings{
    anonymous2021lambdanetworks,
    title={LambdaNetworks: Modeling long-range Interactions without Attention},
    author={Anonymous},
    booktitle={Submitted to International Conference on Learning Representations},
    year={2021},
    url={https://openreview.net/forum?id=xTJEN-ggl1b},
    note={under review}
}
@article{DBLP:journals/corr/HowardZCKWWAA17,
     author    = {Andrew G. Howard and
                  Menglong Zhu and
                  Bo Chen and
                  Dmitry Kalenichenko and
                  Weijun Wang and
                  Tobias Weyand and
                  Marco Andreetto and
                  Hartwig Adam},
     title     = {MobileNets: Efficient Convolutional Neural Networks for Mobile Vision
                  Applications},
     journal   = {CoRR},
     volume    = {abs/1704.04861},
     year      = {2017},
     url       = {http://arxiv.org/abs/1704.04861},
     archivePrefix = {arXiv},
     eprint    = {1704.04861},
     timestamp = {Mon, 13 Aug 2018 16:46:35 +0200},
     biburl    = {https://dblp.org/rec/journals/corr/HowardZCKWWAA17.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }