Questions about codes

[nadongjin]

Hello,
I have a question about your code.

#losses.py
class ArcfaceLoss(tf.keras.losses.Loss):
[skip]
return tf.keras.losses.categorical_crossentropy(y_true, arcface_logits, from_logits=self.from_logits, label_smoothing=self.label_smoothing)

tf.keras.losses.categorical_crossentropy(y_true, arcface_logits, from_logits=self.from_logits, label_smoothing=self.label_smoothing)
===========> Multi-dimensional loss array.

Where is the reduction of the category_crossentropy result value?
I don't know where the sum, mean... part of the loss result is.
[sample]
y_true = [[0, 1, 0], [0, 0, 1]]
y_pred = [[0.0,32, 0], [0.0, 0.0, 16]]
loss = tf.keras.losses.categorical_crossentropy(y_true, y_pred,from_logits=True)
[result]
<tf.Tensor: shape=(2,), dtype=float32, numpy=array([0.0000000e+00, 2.3841855e-07], dtype=float32)>
https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy
thanks....

[leondgarse]

Yes, the result is a an array matching the batch_size. Keras handles this result in keras-team/keras losses.py. If you wanna a mean result, can get it like my print in losses.py MagFaceLoss.

[leondgarse]

• I once have done some tests over this, losses.py#L49 is specifying reduction=None for mirror strategy, and also the reduction_func usage on line 56 and line 97. But I forgot the exact results, just current behavior works... Some strategies are applied to CenterLoss, the aggregation and num_replicas_in_sync usage.
• It's told in Custom training with tf.distribute.Strategy that This reduction and scaling is done automatically in keras model.compile and model.fit, like losses_utils.py compute_weighted_loss and losses_utils.py scale_loss_for_distribution. I think this specifying reduction is needed if model compile and fit outside of strategy.scope.
• Anyway, in my understanding, even if it's needed, this reduction=None should be set for ArcFaceLoss(..., reduction=None), not for its CategoricalCrossentropy.

[nadongjin]

Your code is very good.
Soon I will buy 8GPU.
I just pray that your code will work well in 8GPU. ^^

Personally, I believe you can be the best in this field if you have more datasets and more gpu.
The EffV2S_AdamW's model proves it. :)

[nadongjin]

Definitely, the loss of two gpu is a little strange.
batch_size:224
total_batch_size=448
sch = [
{"loss": losses.ArcfaceLoss(scale=16), "epoch": 1, "optimizer": optimizer},
]
If I print the sum and average of losses in Arcface Loss,
............
total_loss=tf.keras.losses.categorical_crossentropy(y_true, arcface_logits, from_logits=self.from_logits, label_smoothing=self.label_smoothing)
tf.print('mean',tf.reduce_mean(total_loss),'sum',tf.reduce_sum(total_loss))
....................

[result]

1/400 loss 55.2847----> gpu:0 (mean:15.9840, sum:3580), gpu:1(mean:16.1062, sum:3607)

I don't understand why 55.2847 lost.

[nadongjin]

I'm a stupid.
I forgot that l2_regularizer was added to the model.
There's nothing wrong with your code.

[leondgarse]

Uh, I'm glad it works. :)
Have to admit, totally no idea how it will work on 8 GPUs. You may also try if a large speed difference between TF distribute strategies vs Horovod.