Problem with test routine in interpolation mode

[tportenier]

First of all, this is great work, amazing!

While playing around with your code I encountered an issue relating to the synthesis of interpolated samples.

First of all, s_neural_texture.py line 254: z_texture_interpolated = z_texture_interpolated[:, :-2] does not work, since it causes a dimension mismatch later on. I don't know that it is supposed to do so I commented it out.

Next, s_neural_texture.py line 68+69 only make sense in non-interpolation mode, I therefore prepended the line if z_encoding.shape[2]==1: to mitigate this.

Finally, transform_coord() in neural_texture_helper.py is also not handling interpolation mode properly. I replaced the lines 105-107 with the following:

inter = (t_coeff.shape[2] != 1)
    if inter:
        t_coeff = t_coeff.reshape(bs, octaves, dim, dim, h, w)
        t_coeff = t_coeff.permute(0, 1, 4, 5, 2, 3)
    else:
        t_coeff = t_coeff.reshape(bs, octaves, dim, dim).unsqueeze(2).unsqueeze(2)

        t_coeff = t_coeff.expand(bs, octaves, h, w, dim, dim)

An unrelated question: is it possible to somehow run interference on cpu? I am a tensorflow guy and not familiar with pytorch, but it seems that your custom noise sampler is not cpu capable. Is there a way to run it on the cpu?

Cheers,
tiziano

[henzler]

Hi,

thank you for your interest and for spotting those bugs (Those things were
some legacy lines that I must have missed when cleaning up the code)! Feel
free to send a pull request!

Unfortunately, the noise sampler is a Cuda only implementation. I can give you some legacy code where I implemented the noise in python / pytorch only. This should run on cpu, but you might have
to make some adjustments and permute / transpose the position and or seed
tensors.

here is the code and some insights: outer_noise is the "noise sampler" that calls
"middle_noise" and "inner_noise". Inner noise is the actual noise function
and the other two are just helpers to sample bilinear noise and reshape
tensor accordingly.

def outer_noise(coord, seed, octaves):

    channel_noise_stack = []
    for channel in range(3):
        noise_stack = []
        for octave in range(octaves):
            resolution = 2 ** (octave + 1)
            lattice_coord = coord[:, octave] * resolution

            noise_octave = middle_noise(lattice_coord, seed[:, channel, octave])
            noise_stack.append(noise_octave)
        channel_noise_stack.append(torch.stack(noise_stack, dim=1))

    return torch.stack(channel_noise_stack,dim=2)


def middle_noise(lattice_coord, seed):

    coord_integer = torch.floor(lattice_coord)

    bs = lattice_coord.shape[0]
    n_dim = lattice_coord.shape[1]

    list_nearest_neighbours = []
    list_bilinear_neighbours = []
    sub_texel_coord = lattice_coord - torch.floor(lattice_coord)

    weights = bilinear_weights(sub_texel_coord)

    for i in range(2 ** n_dim):
        string = format(i, '0{}b'.format(n_dim))
        offset = torch.tensor([0 if c == '0' else 1 for c in string], dtype=torch.uint8, device='cuda')
        offset_batch = offset.expand(bs, n_dim)
        offset_batch = offset_batch.view(bs, n_dim, 1, 1)

        weights_picked_list = []
        for idx, value in enumerate(offset):
            weights_picked_list.append(weights[:, idx, value.item()])

        weights_picked = torch.stack(weights_picked_list, dim=1)
        weights_prod = torch.prod(weights_picked, dim=1, keepdim=True)

        list_nearest_neighbours.append(inner_noise(coord_integer + offset_batch, seed))
        list_bilinear_neighbours.append(inner_noise(coord_integer + offset_batch, seed) * weights_prod)

    noise_nearest = torch.sum(torch.cat(list_nearest_neighbours, dim=1), dim=1, keepdim=True)
    noise_bilinear = torch.sum(torch.cat(list_bilinear_neighbours, dim=1), dim=1, keepdim=True)

    noise = torch.cat([noise_nearest, noise_bilinear], dim=1)

    return noise


def inner_noise(integer_coord, seed):
    bs, c, w, h = integer_coord.size()

    PHI = 1.61803398874989484820459 * 00000.1
    PI = 3.14159265358979323846264 * 00000.1
    THETA = (3.14159265358979323846264 / 4.0) * 00000.1
    SQ2 = 1.41421356237309504880169 * 10000.0

    a = torch.tensor([PHI, PI, THETA])
    b = torch.tensor(SQ2)

    a = a.expand(bs, 3).unsqueeze(-1).cuda()
    integer_coord = integer_coord.view(bs, c, -1)
    integer_coord = integer_coord * (seed + PHI)
    distance = torch.norm(integer_coord - a[:, :c], dim=1, keepdim=True)

    noise = torch.tan(distance) * b
    noise = noise - torch.floor(noise)
    noise = noise.view(bs, 1, w, h)

    return noise