Add pre and post processing steps to allow non float dtypes

[ashnair1]

Changes

When using cross entropy loss with torchmetrics, masks needs to be of type int but kornia only considers float as valid data type.

Type of change

• 🔬 New feature (non-breaking change which adds functionality)

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Did you update CHANGELOG in case of a major change?

[johnnv1]

add some tests to ensure too... since the random generators use float values, maybe the output will be float anyway... I'm not sure, but that's what I'm waiting for xD

[shijianjian]

I do not think it is reasonable, as most methods considers only 0-1 value ranges. If you want to support other formats like uint8 or uint16. A preprocessing and postprocessing to 0-1 are in need I think.

[ashnair1]

I do not think it is reasonable, as most methods considers only 0-1 value ranges. If you want to support other formats like uint8 or uint16. A preprocessing and postprocessing to 0-1 are in need I think.

Agreed. Have made it so inputs are converted to float before the operations and reverted to original dtype once they're complete.

[ashnair1]

@shijianjian @johnnv1 @edgarriba Any thoughts on this?

[shijianjian]

I think this should not apply on input type.

[shijianjian]

We do not run int test in our CI. You may put only mask for int.

[shijianjian]

This does not make sense for input. Our function does not support 0-255 range.

[ashnair1]

Current issue is the type of dtype to cast to. If a non float dtype mask is detected, it is converted into a float type and later reverted back. The issue is that if it's converted to float, the slow float64 tests complain (Expected float, got double) and when it's converted to double, the slow float32 tests complain. Does it need to be aware of what dtype input is and map accordingly?

Would appreciate some pointers here as I'm not sure how to go about this.

[johnnv1]

Would appreciate some pointers here as I'm not sure how to go about this.

hey @ashnair1, sorry for the delay on this one...

i think the issue here is: the random generator creates the augmentations parameters in XXX device and YYY dtype then casts it into the input dtype (probably the same dtype as the image, since we always have the image)... here you are "hardcoding" the dtype for the other cases when you want to support nonfloating dtypes, instead you should cast it to the input dtype (or maybe the params dtype? not sure cc @shijianjian)

[ashnair1]

That does not make sense to me. Just so we're all on the same page regarding what we're trying to achieve here.

In a kornia augmentation, masks by default have to be float. This is because certain operations don't work for int tensors. But as mentioned above, masks are typically expected to be int tensors and the fact they are float here is a design convenience.

So in order to work with masks tensors of type int, we should pre-process it to float before the op (that only accepts floats) and post process it back to int after the op.

However in the tests, it requires the mask tensors to be the same dtype as input. This means that if you want to test passing in a mask tensor of type int, you would (as you suggested) need to have input be of type int as well. Which in turn means we need to add a pre and post processing step for input as well which @shijianjian had advised against (#2882 (review)) and which I agree with.

Ultimately, what I would like to see, is that kornia should ensure input is float while allowing masks to be int or float relying on the processing steps to ensure the mask tensor works with the ops.

[johnnv1]

That does not make sense to me. Just so we're all on the same page regarding what we're trying to achieve here.

In a kornia augmentation, masks by default have to be float. This is because certain operations don't work for int tensors. But as mentioned above, masks are typically expected to be int tensors and the fact they are float here is a design convenience.

So in order to work with masks tensors of type int, we should pre-process it to float before the op (that only accepts floats) and post process it back to int after the op.

yeap, but we need to cast to the right floating type, which is the same floating type as the input data, otherwise, it will mismatch and crash in some ops...

what I mean is to retrieve the data type from the input itself, like using something like

kornia/kornia/augmentation/container/augment.py

Line 408 in 25237be

inp = in_args[self.transform_op.data_keys.index(DataKey.INPUT)]

(or some other from the API itself)