Improve: Research Agent Regex Parsing #469
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Improvements
Resolved Issues
The list may not be exhaustive.
Related Pull Requests
Explanations
The current version of Devika expects to receive a specific JSON or MARKDOWN response format from LLMs in order to parse them correctly. Unfortunately, LLMs (especially smaller models like LLAMA3) do not always manage to respond perfectly to this task and bury their JSON between explanations, making their response impossible to parse. This pull request aims to use REGEX for better extraction of responses from LLMs.
Furthermore, in order to simplify this method and generalize it to all agents easily and without duplicating code, I propose in this pull request a new parent class for agents AgentTemplate which offers generic render & parse_answer methods that automatically extract the expected format from prompt.jinja2 and perform the expected validations. This modification significantly simplifies the code by reducing about 2/3 of the code for each agent, while avoiding repetition and limiting errors of the type "Invalid response from the model, trying again..."
For now, and as an example and test, I have only modified the RESEARCH agent, which now works perfectly with all LLMs I have tested: GPT4, CLAUDE OPUS, MISTRAL 8x7B, LLAMA3, and GEMMA. Once this pull request is validated, I am committed to adapting all the other agents in the same way.