Since I'm interested in this, I wanted to take a first attempt at implementing this following the patterns that currently exist. I'll expect to have some PRs up shortly for first few items I made on check list over this week. This will be my first contribution to this community, so will hopefully be able to work well with everyone to work through details.

I made some tasks to help track how I'm plan to commit items. My initial focus is on python as it's my primary use case, though if I can I will try to do other languages as well.

@magelisk - For sinks, the batch operation is already there in the interface exposed to the developers with streaming data from the gRPC clients, I don't expect there will be too much performance enhancement to send the data with a unary call from the client in a batch.

I think we should do batching on top on gRPC streaming, we have an issue to track this #1564

Thanks for the input. I hadn't used the streaming sink so I wasn't familiar with it's interface taking the iterator, that's very useful.

Following that and the recommendation that at this be built on top of the streaming interface I think I've wrapped my head around this. I see that #1564 was already given an assignment to @yhl25 . I don't know if any work has been done on this on his part (I couldn't find anything in his accessible repos), so I did an initial draft for my own personal learning opportunity. I don't wan to step on anyone, so feel free to kick it away, but hopefully I can contribute something to help.

protodef: numaproj/numaflow-go#129
numaflow core: #1707
python client: numaproj/numaflow-python#163

The biggest question of these - I did this with a NEW gRPC endpoint, so now the mapping streamer has to Fn handlers. This is a deviation from existing patterns but allows maintaining backwards combability. Not sure how the numaflow team hope to handle these kinds of breakages generally speaking.

@magelisk, thank you for being a contributor to Numaflow. We (the Numaflow team) would like to know how the community and you are using the platform, learn about the use cases, and take some feedback. Would you be open for a chat with us based on your availability?

@syayi would be happy to talk at some point. I'm generally pretty free, and US EST based

Thanks @magelisk ! Sure, drop me a note here or on slack based on what works best for you, and happy to schedule a call

We have a conversation tomorrow, but so that there's a written record I'll put this here now to solicit and document thoughts.

The currently implementation I did was one that has new gRPC call within existing service. This works, though breaks the 'function-only' method as-is. So I think there are a few options

1. Remove/Replace existing map function which takes a single datum and let it take the multiple requiring caller to iterate over stream explicitly
   • Consistent with user defined sink patter
   • Not quite consistent with a map handler function that's typically used
2. A new Service for this which contains single endpoint.
   • A fair bit of additional overhead/new classes that are nearly duplicates of existing. Could share common classes between protos, but again, this isn't something that's currently done so would need to balance DRY vs compartmentalization
   • Should maintain full backwards compability
3. Use as-is in current MR (MapFn and BatchMapFn, and find some other work around, or adjust interface definition to have styles that still work with function-only patter
   • I think this is more complex, more moving parts, but should maintain existing backwards compatibility (I haven't prototyped this to see what final impact to the interface classes are