Issue: Evaluators don't run after a ConnectionError

[jennmueng]

Issue you'd like to raise.

We've been getting intermittent connection errors after an evaluation run right before evaluators are run:

Error evaluating run 6058a864-6395-4df9-9800-237d90e7199f with DynamicRunEvaluator: LangSmithConnectionError("Connection error caused failure to get https://api.smith.langchain.com/examples/ace841ac-123c-4f38-bf58-87cc6637e133  in LangSmith API. Please confirm your internet connection.. ConnectionError(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))")

This results in a run without any of the evaluator results.

Suggestion:

Looks like there could be connection issues but even if there are, the evaluators shouldn't be skipped and there should be a retry

[hinthornw]

Have seen dropped runs more frequently recently too - will work on resolving.

Errors don't impact subsequent evaluator calls, however.

[hinthornw]

Are you still running into this?

[jennmueng]

@hinthornw Haven't had connection errors recently so can't confirm whether this error still occurs