Function streaming

[ramicaza]

Function calling is extremely useful but it's a little tricky to implement it together with message streaming because of the weird way functions are streamed by openai. By streaming both messages and functions, it allows for fluid UX like the one displayed below. Notice how the model goes from streaming a response, right to calling the function it needs.

This PR is to enable function call streaming such that modules like ReAct and other future ones can have fluid UIs like the one below.

Screen.Recording.2023-11-19.at.1.29.49.PM.mov

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
big-agi	❌ Failed (Inspect)		Nov 19, 2023 6:37pm

[enricoros]

Thanks for this request. I see it's failing the check, let me know if you have an update. It's very important to get the function streaming right, and in particular to not overfit the architecture to openai (as we have many vendors). I do like the functionality, but OpenAI tends to overload APIs which won't even work between models (e.g. -vision- has different types), which makes it a mess for the ecosystem.

I'll review this, but not with the highest priority. If you have an update that solves the build issue (some issues with the message type not in sync), and that offers a great abstraction so that other vendors one day will be able to use streaming functions, I'ld love to see it!

[enricoros]

@ramicaza I intend starting to merge this as soon. Important questions:

• Does this implementation work with Multiple function calls, that OpenAI announced recently?
• And do you have example code on how to use this? I want to make sure the architecture is optimal to support downstream applications

[enricoros]

Large progress has been made. Not function streaming yet, but almost. Every model vendor has been abstracted to support a better Streaming and Non-Streaming generation API. Continuing... ⌛

[enricoros]

Note - evaluating this. Uncertain whether to terminate (execute the corresponding functions) on the client side, or on the server side.