DDGS Ratelimit

[thierryteisseire]

When using DuckDuckGo Search
: Search the web with DuckDuckGo. Getting a ratelimit error as seen in LangSmith. If someone had the same or have got an idea on why?
Thanks all!

DuckDuckGoSearchException('_get_url() https://links.duckduckgo.com/d.js DuckDuckGoSearchException: Ratelimit')Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search.py", line 49, in _get_url
raise DuckDuckGoSearchException("Ratelimit")

duckduckgo_search.exceptions.DuckDuckGoSearchException: Ratelimit

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1595, in _atransform_stream_with_config
chunk: Output = await asyncio.create_task( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 238, in tap_output_aiter
async for chunk in output:

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 525, in _atransform
_interrupt_or_proceed(done, inflight, step)

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 700, in _interrupt_or_proceed
raise exc

File "/usr/local/lib/python3.11/site-packages/langgraph/pregel/init.py", line 838, in _aconsume
async for _ in iterator:

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4144, in astream
async for item in self.bound.astream(

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2449, in astream
async for chunk in self.atransform(input_aiter(), config, **kwargs):

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2432, in atransform
async for chunk in self._atransform_stream_with_config(

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1595, in _atransform_stream_with_config
chunk: Output = await asyncio.create_task( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 238, in tap_output_aiter
async for chunk in output:

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2402, in _atransform
async for output in final_pipeline:

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/passthrough.py", line 269, in atransform
async for chunk in self._atransform_stream_with_config(

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1560, in _atransform_stream_with_config
final_input: Optional[Input] = await py_anext(input_for_tracing, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 62, in anext_impl
return await anext(iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 97, in tee_peer
item = await iterator.anext()
^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3711, in atransform
async for output in self._atransform_stream_with_config(

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1595, in _atransform_stream_with_config
chunk: Output = await asyncio.create_task( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 238, in tap_output_aiter
async for chunk in output:

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3680, in _atransform
output = await acall_func_with_variable_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/backend/app/agent_types/openai_agent.py", line 70, in call_tool
responses = await tool_executor.abatch(actions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4119, in abatch
return await self.bound.abatch(
^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 568, in abatch
return await gather_with_concurrency(configs[0].get("max_concurrency"), *coros)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/utils.py", line 51, in gather_with_concurrency
return await asyncio.gather(*coros)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 565, in ainvoke
return await self.ainvoke(input, config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3527, in ainvoke
return await self._acall_with_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1295, in _acall_with_config
output: Output = await asyncio.create_task(coro, context=context) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 3474, in _ainvoke
output = await acall_func_with_variable_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langgraph/prebuilt/tool_executor.py", line 73, in _aexecute
output = await tool.ainvoke(tool_invocation.tool_input, config=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tools.py", line 238, in ainvoke
return await self.arun(
^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tools.py", line 492, in arun
raise e

File "/usr/local/lib/python3.11/site-packages/langchain_core/tools.py", line 451, in arun
else await self._arun(*tool_args, **tool_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tools.py", line 303, in _arun
return await run_in_executor(None, self._run, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 493, in run_in_executor
return await asyncio.get_running_loop().run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_community/tools/ddg_search/tool.py", line 39, in _run
return self.api_wrapper.run(query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_community/utilities/duckduckgo_search.py", line 81, in run
results = self._ddgs_text(query)
^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_community/utilities/duckduckgo_search.py", line 57, in _ddgs_text
return [r for r in ddgs_gen]
^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_community/utilities/duckduckgo_search.py", line 57, in
return [r for r in ddgs_gen]
^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search.py", line 95, in text
for i, result in enumerate(results, start=1):

File "/usr/local/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search.py", line 146, in _text_api
resp = self._get_url("GET", "https://links.duckduckgo.com/d.js", params=payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search.py", line 53, in _get_url
raise DuckDuckGoSearchException(f"_get_url() {url} {type(ex).name}: {ex}") from ex

duckduckgo_search.exceptions.DuckDuckGoSearchException: _get_url() https://links.duckduckgo.com/d.js DuckDuckGoSearchException: Ratelimit

[samuelp-mw]

I also faced the issue : Upgrading duckduckgo-search package to the latest version made it work on my side. I do not have more information about what the issue was however.

[thierryteisseire]

cool, thanks; did you update the pyproject.toml to 5.0?

[samuelp-mw]

I upgraded duckduckgo-search = "^5.1.0".
The requirements in my pyproject.toml are however slightly different so I would need to test with main to confirm that it works with the current state.

I followed the thread here as reference : joaomdmoura/crewAI#136

[thierryteisseire]

I confirm it works! Just did poetry update duckduckgo-search..thanks!

[danielgen]

The weird thing is that this is (for me at least) is not a rate limit problem: e.g.

  File "/opt/miniconda3/envs/crew/lib/python3.11/site-packages/duckduckgo_search/duckduckgo_search_async.py", line 124, in _aget_url
    raise RatelimitException(f"{resp.url} {resp.status_code} Ratelimit")
duckduckgo_search.exceptions.RatelimitException: https://links.duckduckgo.com/d.js?q=teletubbies&kl=wt-wt&l=wt-wt&p=&s=0&df=y&vqd=4-23477882074055204179171254865267777697&ex=-1 202 Ratelimit

which goes:

        if resp.status_code == 200:
            return cast(bytes, resp.content)
        self._exception_event.set()
        if resp.status_code in (202, 301, 403):
            raise RatelimitException(f"{resp.url} {resp.status_code} Ratelimit")

the link being returned contains results, so duckduckgo did execute the request.
I tried inspecting duckduckgo_search_async.py, line 124 but so far no luck.

Anyhow 202 is "the request has been accepted for processing, but the processing has not been finished yet"
I would expect 429 for rate limit problems...