[BUG] - Tracing for Langchain functions agent fails.

[LokeshShelva]

Describe the bug
The tracing fails for Langchain OpenAI function agent. I have a openai functions agent that uses langchain tools as functions. The tracing fails after a function call from the agent.

How To Reproduce the bug
Steps to reproduce the behavior, how frequent can you experience the bug: Always

1. Create a flow with the following tool. This is an example flow created to reproduce the bug.

from promptflow import tool
from promptflow.tracing import start_trace
from langchain_openai import AzureChatOpenAI
from langchain.agents import (
    create_openai_functions_agent,
)
from langchain.agents.agent import (
    AgentExecutor,
    RunnableAgent,
)
from langchain.tools import Tool
from langchain_core.messages.system import SystemMessage
from langchain_core.messages.human import HumanMessage
from langchain_core.prompts.chat import ChatPromptTemplate, MessagesPlaceholder

start_trace()

def tool_func(nums: str) -> int:
    a, b = [int(x.strip()) for x in nums.split(",")]
    return a + b
# The inputs section will change based on the arguments of the tool function, after you save the code
# Adding type to arguments and return value will help the system show the types properly
# Please update the function name/signature per need
@tool
def my_python_tool(num1: int, num2: int) -> str:
    tools = [
        Tool(
            name="adder",
            description="Adds two numbers. Input is comma seperated two numbers. Eg: 1, 2",
            func=tool_func 
        )
    ]
    llm = AzureChatOpenAI(
        model="gpt-4",
        azure_deployment="gpt-4",
        azure_endpoint="<endpoint>",
        api_key="<key>",
        api_version="<version>"
    )

    messgaes = [
        SystemMessage(content="You are a agent designed to use tools. Use the given tools to solve the problem."),
        HumanMessage(content=f"What is the sum of {num1} and {num2}?"),
        MessagesPlaceholder(variable_name="agent_scratchpad")
    ]

    prompt = ChatPromptTemplate.from_messages(messages=messgaes)

    agent = RunnableAgent(
        runnable=create_openai_functions_agent(llm, tools, prompt),
        return_keys_arg=["output"],
    )

    executor = AgentExecutor(
        name="Tools user",
        agent=agent,
        tools=tools,
        verbose=True
    )

    return executor.invoke({})

3. Run the flow

Expected behavior
The flow should tun without any error.

Actual behavior
The flow fails with the following error

Prompt flow service has started...
WARNING:root:'from promptflow import tool' is deprecated and will be removed in the future. Use 'from promptflow.core import tool' instead.
Prompt flow service has started...
Prompt flow service has started...
Prompt flow service has started...
2024-05-06 11:11:52 +0000    2702 execution.flow     INFO     Start executing nodes in thread pool mode.
2024-05-06 11:11:52 +0000    2702 execution.flow     INFO     Start to run 1 nodes with concurrency level 16.
2024-05-06 11:11:52 +0000    2702 execution.flow     INFO     Executing node llm. node run id: 799b20f3-b9cb-4485-9c4f-545e143446ac_llm_0
2024-05-06 11:11:52 +0000    2702 execution.flow     INFO     [llm in line 0 (index starts from 0)] stdout> 

> Entering new Tools uaer chain...
You can view the trace detail from the following URL:
http://localhost:23333/v1.0/ui/traces/?#collection=test-flow&uiTraceId=0x8dbd07b142006e206a1cca4ce537561f
2024-05-06 11:11:55 +0000    2702 execution.flow     INFO     [llm in line 0 (index starts from 0)] stdout> 
Invoking: `adder` with `1232324, 123142412`



2024-05-06 11:11:55 +0000    2702 execution.flow     INFO     [llm in line 0 (index starts from 0)] stdout> 
2024-05-06 11:11:55 +0000    2702 execution.flow     INFO     [llm in line 0 (index starts from 0)] stdout> 124374736
2024-05-06 11:11:55 +0000    2702 execution.flow     INFO     [llm in line 0 (index starts from 0)] stdout> 
WARNING:azure.monitor.opentelemetry.exporter.export._base:Retrying due to server request error: <urllib3.connection.HTTPSConnection object at 0x7fb6d9ba51f0>: Failed to resolve 'eastus-8.in.applicationinsights.azure.com' ([Errno -5] No address associated with hostname).
2024-05-06 11:11:57 +0000    2702 execution          ERROR    Node llm in line 0 failed. Exception: Execution failure in 'llm': (TypeError) expected string or buffer.
Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 182, in _invoke_tool_inner
    return f(**kwargs)
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 469, in wrapped
    output = func(*args, **kwargs)
  File "/workspaces/abb.genix.generative.ai.poc/genix-copilot-test/genix-copilot-core-test/test-flow/llm.py", line 62, in my_python_tool
    return executor.invoke({})
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain/chains/base.py", line 163, in invoke
    raise e
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain/chains/base.py", line 153, in invoke
    self._call(inputs, run_manager=run_manager)
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain/agents/agent.py", line 1432, in _call
    next_step_output = self._take_next_step(
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain/agents/agent.py", line 1138, in _take_next_step
    [
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain/agents/agent.py", line 1138, in <listcomp>
    [
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain/agents/agent.py", line 1166, in _iter_next_step
    output = self.agent.plan(
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain/agents/agent.py", line 397, in plan
    for chunk in self.runnable.stream(inputs, config={"callbacks": callbacks}):
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 2875, in stream
    yield from self.transform(iter([input]), config, **kwargs)
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 2862, in transform
    yield from self._transform_stream_with_config(
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 1880, in _transform_stream_with_config
    chunk: Output = context.run(next, iterator)  # type: ignore
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 2826, in _transform
    for output in final_pipeline:
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 1283, in transform
    for chunk in input:
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 4722, in transform
    yield from self.bound.transform(
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/runnables/base.py", line 1300, in transform
    yield from self.stream(final, config, **kwargs)
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 245, in stream
    raise e
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_core/language_models/chat_models.py", line 225, in stream
    for chunk in self._stream(messages, stop=stop, **kwargs):
  File "/home/vscode/.local/lib/python3.9/site-packages/langchain_openai/chat_models/base.py", line 395, in _stream
    for chunk in self.client.create(messages=message_dicts, **params):
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/tracing/contracts/generator_proxy.py", line 34, in generate_from_proxy
    yield from proxy
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/tracing/contracts/generator_proxy.py", line 19, in __next__
    item = next(self._iterator)
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 198, in traced_generator
    enrich_span_with_llm_if_needed(span, original_span, inputs, generator_output)
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 178, in enrich_span_with_llm_if_needed
    token_collector.collect_openai_tokens_for_streaming(span, inputs, generator_output, parser.is_chat)
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/tracing/_trace.py", line 60, in collect_openai_tokens_for_streaming
    tokens = calculator.get_openai_metrics_for_chat_api(inputs, output)
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/tracing/_openai_utils.py", line 102, in get_openai_metrics_for_chat_api
    metrics["prompt_tokens"] = self._get_prompt_tokens_from_messages(
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/tracing/_openai_utils.py", line 137, in _get_prompt_tokens_from_messages
    prompt_tokens += len(enc.encode(value))
  File "/home/vscode/.local/lib/python3.9/site-packages/tiktoken/core.py", line 116, in encode
    if match := _special_token_regex(disallowed_special).search(text):
TypeError: expected string or buffer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 90, in invoke_tool
    result = self._invoke_tool_inner(node, f, kwargs)
  File "/home/vscode/.local/lib/python3.9/site-packages/promptflow/_core/flow_execution_context.py", line 206, in _invoke_tool_inner
    raise ToolExecutionError(node_name=node_name, module=module) from e
promptflow._core._errors.ToolExecutionError: Execution failure in 'llm': (TypeError) expected string or buffer
2024-05-06 11:11:57 +0000    2702 execution.flow     WARNING  Failed to calculate metrics due to exception: expected string or buffer.
2024-05-06 11:11:57 +0000    2702 execution.flow     ERROR    Flow execution has failed. Cancelling all running nodes: llm.
pf.flow.test failed with UserErrorException: TypeError: Execution failure in 'llm': (TypeError) expected string or buffer
WARNING:azure.monitor.opentelemetry.exporter.export._base:Retrying due to server request error: <urllib3.connection.HTTPSConnection object at 0x7fb6d9aaf790>: Failed to resolve 'eastus-8.in.applicationinsights.azure.com' ([Errno -5] No address associated with hostname).

Running Information(please complete the following information):

• Promptflow Package Version using pf -v:

{
 "promptflow": "1.10.1",
 "promptflow-core": "1.10.1",
 "promptflow-devkit": "1.10.1",
 "promptflow-tracing": "1.10.1"
}

• Operating System: Debian 12 (in dev container - image mcr.microsoft.com/devcontainers/python:3.9)
• Python Version using python --version: Python 3.9.19

Additional context
This is what i was able to debug so far.

These are the messages after a function call going to the get_openai_metrics_for_chat_api function .

[
{"role": "system", "content": "You are a agent designed to use tools. Use the given tools to solve the problem."}, 
{"role": "user", "content": "What is the sum of 1232324 and 123142412?"}, 
{"role": "assistant", "content": None, "function_call": {...}}, {"role": "function", "content": "124374736", "name": "adder"}
]

When content field is None for the message that is a function call, the call to tiktoken fails.

[lumoslnt]

Hello @LokeshShelva, what is the langchain version and langchain-openai version you use?

[LokeshShelva]

Hello @lumoslnt, here are the version I am using:

langchain - 0.1.13
langchain-community - 0.0.32
langchain-core - 0.1.41
langchain-openai - 0.0.6

[lumoslnt]

@LokeshShelva In the short-term, we are implementing a try-catch block for the current code segment to ensure that any errors encountered during token calculation do not interrupt the flow execution. This solution will be included in the upcoming release. Also we are planning to support calculating tokens for function calling in our long-term strategy. I will keep you updated on our progress.

[LokeshShelva]

Thank you for looking into the issue.