LLM Isnights (rate limit handling and so on )

[mdarweash]

Does it support rate limit handling (like backoff, Retry) or is it on the path of being implemented?

any sort of collecting insights about LLM usage (number of tokens used ...etc) for later push to analytics (like prometheus or any data store)

[kwilliams-halosight]

I have another similar question, but I can answer in part. If your AI Service returns a Response , the object has a Usage property object that contains token usage separated by input/output/etc.

[langchain4j]

Not yet implemented

[langchain4j]

A simple "retry after a rate limite error" strategy can be implemented quite easily I guess, but for a more robust solution we need to think if it should be a part of a library or not.