langgraph/tutorials/multi_agent/hierarchical_agent_teams/

[giscus[bot]]

langgraph/tutorials/multi_agent/hierarchical_agent_teams/

Build language agents as graphs

https://langchain-ai.github.io/langgraph/tutorials/multi_agent/hierarchical_agent_teams/

[gregorybarnes]

The 2nd test to "Write an outline for poem and then write the poem to disk." didn't seem to work with gpt-4o when I initially tried it. I retried using gpt-4 which seemed to work fine. I re-ran the test again using gpt-4o so that I could capture the stack trace and it actually worked the 2nd time around! I'm not 100% certain but I vaguely remember the error complaining about the function call format on the first run.

Anyone else testing with gpt-4o may need to re-run a few times to get it to work correctly.

Also, the last test to "Write a brief research report on the North American sturgeon. Include a chart." triggered rate limit errors when running with gpt-4o:

RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4o in organization org-xxxxxxxxxx on tokens per min (TPM): Limit 30000, Requested 38324. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

I'm too lazy to add code to correctly handle rate limits...

I wasn't ever able to get the 3rd and final test to complete. gpt-4-1106-preview, gpt-4, and gpt-3.5-turbo all caused the same error on multiple runs:
BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'messages[3].name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.", 'type': 'invalid_request_error', 'param': 'messages[3].name', 'code': 'invalid_value'}}

One time when running the last test with gpt-4-1106-preview I got the following error:
ValueError: An output parsing error occurred. In order to pass this error back to the agent and have it try again, pass handle_parsing_errors=Trueto the AgentExecutor. This is the error: Could not parse tool input: {'arguments': 'scrape_webpages ', 'name': 'functions'} because thearguments is not valid JSON.

I even went back and updated all of the teams to use gpt-4-1106-preview but the 3rd and final test never actually worked.

I get the gist of things here and I understand that I may need to tweak things a bit with prompting but should I be concerned about reliability? So far while running the tutorials in the "Multi-Agent Systems" section things seem pretty brittle and very sensitive/dependent on specific model versions. Are there tricks to getting around this or do I just need to continue working through the other tutorials and learn other design patterns to account for these issues?

[hinthornw]

I don't personally think multi agent architectures are necessary or optimal for most applications, if that's your question

[gregorybarnes]

Ya, I think that cuts down to the core of my question. Basically, what you're suggesting is I should try to define a single agent architecture for most applications because each added layer of agents may expose opportunities and vectors for formatting issues in their responses?

I do have a very specific application that I'm building for Model Evaluations. The goal is to orchestrate multiple agents playing the social deduction game called "Mafia" (sometimes called Werewolf) together.

Do you know if any design patterns tend to be more reliable or optimal compared to others when working with multi agent applications (ex. Supervision design is better than Hierarchical design because it has fewer layers)? Or is reliability more of an exercise in Prompt Engineering (ex. specify example response formats). Or worst case scenario is it an exercise in fine-tuning &/or pre-training?

[hinthornw]

Oh good question. The honest answer is "it depends" and that I'd typically start with fewer moving parts and only add complexity if my evals identify something I can't easily solve with the simpler design.

I think a supervisor pattern or something similar (can do fun things with speaker selection, for instance) would work for this.

1 note: I view prompt engineering and fine-tuning different shades of the same thing -> ways of inducing better task-specific behavior while potentially (increase bias, decrease variance)

[hinthornw]

Cool application btw! Mafia/Werewolf is a really fun game

[goern]

@gregorybarnes could you have a look at #608 and let me know if it fixes your problem?

[hiravebapu]

@goen

great article. i created simple supervision agent system for arithmetic operation, just to get idea around.

lets say, i have agent "addition", it has its tool to do the job. That tool requires two arguments to run the addition, seems like supervisor and agent stuck in loop during multi-turn conversation

User: add numbers
{'supervisor': {'next': 'Addition'}}
----
{'Addition': {'messages': [HumanMessage(content='Please provide the two numbers you would like to add.', name='Addition')]}}
----
{'supervisor': {'next': 'Addition'}}
----
{'Addition': {'messages': [HumanMessage(content='Please provide the two numbers you would like to add.', name='Addition')]}}
----
{'supervisor': {'next': 'Addition'}}
----

any suggestion ?