Output shape of allgather(): Horovod vs Pytorch

[Alam45]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
The output of horovod.torch.allgather(tensor, name=None) is a tensor of concatenation of tensor on different rank processes. While the output of torch.distributed.all_gather(tensor_list, tensor) is a tensor list.

Pytorch output (Two GPUs):
before all_gather(): Input tensors
#tensor([1, 2]) # Rank 0
#tensor([3, 4]) # Rank 1
After all_gather(): Output tensor list
#[tensor([1, 2]), tensor([3, 4])] # Rank 0
#[tensor([1, 2]), tensor([3, 4])] # Rank 1

Horovod output (Two GPUs):
before all_gather(): Input tensors
#tensor([1, 2]) # Rank 0
#tensor([3, 4]) # Rank 1
After all_gather(): Output tensor
#tensor([1, 2, 3, 4]) # Rank 0
#tensor([1, 2, 3, 4]) # Rank 1

How can I get the output as in pytorch using horovod.torch.allgather(tensor, name=None)?

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[maxhgerlach]

@Mahbubalam4545, have you tried passing two-dimensional tensors into hvd.allgather()? E.g., they would be shaped like [[1,2]] and [[3,4]].

[Alam45]

Yes, I have split it in my code before this PR. My argument is why Horovod is merging the tensors gathered from different processes? and what are the benefits? If in a situation when the tensors are of varied sizes or when we don't know the exact size of the tensors on each of the processes, in these situations the allgather() would be useless. In short, I would request you to investigate the intuitions behind the design of Pytorch's all_gather().

[maxhgerlach]

Allgather's main purpose in Horovod is to aggregate sparse gradients.

You could also look at hvd.allgather_object if you want a convenient function that gives you a Python list.

[Alam45]

I'm using horovod 0.19.0. does it have allgather_object? I'm getting error:
"AttributeError: module horovod.torch has no attribute allgather_object "
Kindly, advise me, Which version should I use?

[maxhgerlach]

you should definitely upgrade to the latest release :)

[bryant1410]

Does allgather_object respect the gradients like dist.nn.all_gether does?