Adding parameter support for Local LLMs and GROQ #808
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for this @gpapp ! We have an experimental groq branch, which uses their official SDK instead of making manual requests (which is what we'd like to do in the future with the default openai compatible client function). Do you think you could adapt your changes to work on that? This would also make them less risk (in terms of how different people have set up environments wihch may inadvertenly cause problems with extra env flags), as it would only apply to Groq so far. Btw the prompt mismatch is a good catch, I was convinced that was a shortcoming of Mixtral. |
Added more aggressive file content parsing to CodeMonkey.py Added more aggressive response parsing to function_calling.py
Moved return type enforcement to function_calling This should work with OpenAI as well.
merging groq from @senko with own implementation
|
Updated to use the SDK (which in fact uses the openAI compatible API for now) readded all the modifiers that can be passed as parameters to groq. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some locally served LLMs need more fine-tune in their parameters, than the OpenAI API.
GROQ also opened their API-s, which has a limited parameters, that can be passed over in the API
The prompt example containts a timeout quoted, while it expects numeric value. This is a waste of many tokens to resolve.