ResampleOp.resample() doesn't clamp premultiplied alphas

[sachindshinde]

The code for ResampleOp.resample(), when working with source images that have premultiplied alphas, doesn't ensure destination color samples are clamped to the alpha value. This can result in other ImageIO code somewhat mysteriously failing (in my case, I had ComponentColorModel.extractComponent() returning a value beyond its expected 8-bit range).

Regarding alpha handling, it seems like the ideal thing to do (based on this post in ImageMagick forums, and this part of ImageMagick's resize.c) would be something like:

• If there's a source alpha channel and it's not premultiplied, when multiplying source sample values by the weight, also multiply by the alpha for the source sample and divide by the source alpha's channel max. We'd additionally need to keep track of the sum of (weight * alpha / max_alpha) for the destination sample, if the destination isn't premultiplied.
• If the destination is premultiplied, then clamp the destination sample value between 0 and the min of the destination sample's channel max and the alpha for the destination sample.
• If the destination isn't premultiplied, divide the destination sample value by the sum of (weight * alpha / alpha_max ) for the destination sample, and clamp as normal between 0 and the destination sample's channel max.

[haraldk]

Hi Sachin,

Thanks for looking into this!

If you are able to convert the above into code in a PR, I'd be happy to merge it! If not, I might still look into it some day. It's a bit down on my priority list right now though, as I hardly ever work with premultiplied images.

Best regards,

--
Harald K