feat : Adds ggml_upscale_ext

[balisujohn]

Closes issue #812

Adds ggml_upscale_ext alternate version of ggml_upscale which lets the user upscale a tensor to a specified shape using nearest neighbor interpolation.

Adds a CPU and cuda implementation, and test, though it's not clear to me either this or ggml_upscale has adequate tests.

Also, the reason I added this is because it is necessary to convert tortoise-tts to ggml.

This is unfinished since I still need to add the cuda implementation and possibly expand tests since I haven't verified it actually works yet. Will remove draft annotation when ready for review.

[balisujohn]

(I also realize there are some unnecessary params in some of the functions, so I will clean that up before review)

[balisujohn]

Ready for review! :^)

[ggerganov]

There is no need to add extra OP, we can reuse the existing GGML_OP_UPSCALE

[balisujohn]

fixed

[ggerganov]

We should generalize the existing upscale OP and kernels since the existing ggml_upscale functionality is a special case of the more general ggml_upscale_to_shape

[balisujohn]

We should generalize the existing upscale OP and kernels since the existing ggml_upscale functionality is a special case of the more general ggml_upscale_to_shape

I can do that. My thinking was to avoid breaking backwards compatibility, but I can just put the new behavior in the ggml_upscale op since it is a strict superset of ggml_upscales original behavior.

[ggerganov]

ggml_upscale can still do what it currently does in order to keep backwards compatibility. Take a look at ggml_soft_max and ggml_soft_max_ext as an example of what I have in mind

[balisujohn]

ready for review again :^)

[balisujohn]

very impressive response time!

[ggerganov]

This implementation ignore half-pixel effects - I'm not sure how the reference PyTorch operators behave, but it's something that needs a deeper look

I also wonder if we should take the time to make the upscale operator more general to support downscaling - i.e. GGML_OP_RESCALE

[ggerganov]

Please update the kernel to use the tensor strides - see the CPU and Metal implementations as an example. Otherwise, this would work only for contiguous data

When ready, try to see if you can add tests in test-backend-ops that exercise non-contiguous data

[balisujohn]

OK done.

[balisujohn]

This implementation ignore half-pixel effects - I'm not sure how the reference PyTorch operators behave, but it's something that needs a deeper look

I also wonder if we should take the time to make the upscale operator more general to support downscaling - i.e. GGML_OP_RESCALE

Alright so wrt behaving similarly to PyTorch nearest neighbor interpolation I can confirm

cur = ggml_upscale_ext(ctx0, cur, 187, 1024, 1,1);

behaves the same way (only tested with cuda backend) as

expanded_code_emb = F.interpolate(code_emb, size=187, mode='nearest')

for my internal test case tensor shaped [43,1024]. (dimensions read left to right)

I think it would be nice to support downscaling but I also think it might be good to leave that to a future PR.

Ready for review again :^)

[balisujohn]

Also is there somewhere inputs and output target literals are defined for the test cases? I found it a bit strange that I didn't have to specify the actual elementwise values of input literals and output target literals when defining the tests.

[ggerganov]

The test-backend-ops tool does not provide mechanism for specific input / output values. You can write a standalone test, similar to test-conv1d.cpp for example. Would be useful to demonstrate that the numerical results match with the results from PyTorch.

Also, does PyTorch support floating-point scaling factor? If so, we should consider extending the functionality