-
Notifications
You must be signed in to change notification settings - Fork 777
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature Request] Function Calling for mlx_lm.server #784
Comments
It would be pretty cool to add this and perhaps not too difficult. I believe function calling requires a few things:
Marked as an enhancement. I will leave it open if someone is interested in working on it. |
Are we able to integrate with open source framework? For example langchain autogen, etc. |
Maybe we can take a look at ollama-mistral:v0.3 Prompt
Response
Any model base on mistral 0.3 should work the same. |
I wrote a library that constrains LLM output to a JSON schema in a performant way, and implemented a function calling/tools server example for MLX with it. I find that it works quite well even with models that have not been fine-tuned for function calling specifically. You can check it out here: https://github.com/otriscon/llm-structured-output If you want to give it a try, I'm happy to answer any questions and open to suggestions for improvement. |
Hello, thanks for the amazing repo. I would like to request support for function calling feature for the mlx_lm server, similar to OpenAI's implementation.
Please let me know if this is on the roadmap, or if there are good frameworks that already implements this.
The text was updated successfully, but these errors were encountered: