Job Status is failed, when scale-in ps.

[Mesilenceki]

In the implementation of tf-operator, it supports the scaling of replicas. For instance, if I reduce the number of parameter server (ps) replicas from 4 to 2 by using kubectl delete pod, the pods receive the SIGTERM signal and the container exits with code 137. However, the operator still checks if the pod's exit code is normal, which would result in the job exiting abnormally. Is such an implementation somewhat peculiar, or could it be considered a bug?

[tenzen-y]

IIUC, the TFJob doesn't support any scaling approaches, but there are not any validations.
For confirmation: @kubeflow/wg-training-leads

[johnugeorge]

Yes. We do not have any handle this case. I am curious if you were doing this intentionally to test the behaviour ?

[Mesilenceki]

Yes. We do not have any handle this case. I am curious if you were doing this intentionally to test the behaviour ?

Actually, we tend to make our tfjobs elastic scaling for resource utilization.I understand that supporting scaling-in within the operator should not pose any other issues, right?

[tenzen-y]

We meant that this behavior is an intended and expected one, not a bug.