Added 2D Upscaling Methods - Bilinear (#857), Nearest Neighbor, Subpixel Convolution (Phase Shift) #877
Conversation
|
Apologies for the several commits- forgot about how Python 3 handles floating-point division. |
| if b > 1 or a > 1: | ||
| output_shape = self.get_output_shape_for(input.shape) | ||
| upscaled = T.zeros(shape=output_shape, dtype=input.dtype) | ||
| upscaled = T.set_subtensor(upscaled[:, :, ::a, ::b], input) | ||
| elif self.mode == 'bilinear2D': | ||
| upscaled = T.nnet.abstract_conv.bilinear_upsampling( | ||
| input, self.scale_factor[0], |
There was a problem hiding this comment.
I am not sure with the change in this line. Since Bilinear Upsampling doesn't use learnable filters, I think we should be having same number of filters as in the previous layer. Please correct me if I am wrong
There was a problem hiding this comment.
I got notified by github that i didn't "submit" my comment. Usually, comment(shift + enter) alone is enough. The interface got complicated, I will check it out. Apologise if it has occurred as a repost.
There was a problem hiding this comment.
I'm not sure I understand your requested change- Theano handles creation of bilinear upsample filters within bilinear_upsampling, it's not an argument passed into bilinear_upsampling by the change in the PR.
The second input to bilinear_upsampling is the scalar scale factor to upsample the tensor by, not the filters used for convolution as it would be with something like T.nnet.conv2d().
The output of bilinear_upsampling is a 4D tensor of dims [batchsize, num_filters, row * scale_factor, col*scale_factor] , where num_filters is the same as the number of filters of the previous layer- only the dims of the two trailing axes are affected by bilinear_upsampling.
Please do correct me if I'm wrong about any of the above, any feedback is welcome.
There was a problem hiding this comment.
The second input to bilinear_upsampling is the scalar scale factor to upsample the tensor by, not the filters used for convolution as it would be with something like T.nnet.conv2d().
Yeah, my bad! Had it been the case, it should have been an issue in Theano primarily.
|
Hey, sorry for the delay, and thanks for the PR!
👍
👍
Lasagne has transposed convolution, which subpixel convolution is a special case of (https://arxiv.org/abs/1609.07009, Figure 3). So I guess we can omit this? |
Ok- I've removed it from the PR, and squashed the 4 commits into 1- should be ready to merge. |
| if b > 1 or a > 1: | ||
| output_shape = self.get_output_shape_for(input.shape) | ||
| upscaled = T.zeros(shape=output_shape, dtype=input.dtype) | ||
| upscaled = T.set_subtensor(upscaled[:, :, ::a, ::b], input) | ||
| elif self.mode == 'bilinear2D': | ||
| upscaled = T.nnet.abstract_conv.bilinear_upsampling( | ||
| input, self.scale_factor[0], |
There was a problem hiding this comment.
The second input to bilinear_upsampling is the scalar scale factor to upsample the tensor by, not the filters used for convolution as it would be with something like T.nnet.conv2d().
Yeah, my bad! Had it been the case, it should have been an issue in Theano primarily.
This PR:
Merges in the added 2D bilinear upscaling from Add 2D bilinear upsampling layer #857 and adds
'bilinear1D'and'bilinear2D'as modes to the existingUpscale2Dclass.Adds in a mode for nearest-neighbor interpolation. Not exactly state of the art, but I've been using it to implement BEGAN, so it may be useful to others.
I'm aware of Checkerboard artifact free sub-pixel convolution #862 and may attempt to implement it at some point, but from my skimming of the referenced paper, it mentions that the initialization scheme is for subpixel convolution. As Lasagne appears to lack a subpixel convolution option for upscaling included as part of the library, I added in subpixel convolution phase shift based primarily on the second subpixel convolution method here + unit tests with the corresponding numpy implementation based on the algorithm in the paper.
Currently, Nearest neighbor and subpixel convolution are checked in numpy, bilinear convolution is only checked against shape as in #857.
All suggestions are welcome- It'd be good to see some more upscaling methods in Lasagne. If the documentation does not display correctly, I will try to fix it- I've been having a bit of trouble rendering it on my machine.