🪢 Open source LLM engineering platform: Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23

🐢 Open-Source Evaluation & Testing for LLMs and ML models

Test your prompts, models, and RAGs. Catch regressions and improve prompt quality. LLM evals for OpenAI, Azure, Anthropic, Gemini, Mistral, Llama, Bedrock, Ollama, and other local & private models with CI/CD integration.

The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.

The LLM Evaluation Framework

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界.

Open-Source Evaluation for GenAI Application Pipelines

A list of LLMs Tools & Projects

Python SDK for running evaluations on LLM generated responses

Awesome papers involving LLMs in Social Science.

A collection of hand on notebook for LLMs practitioner

Framework for LLM evaluation, guardrails and security

Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute, relative and much more. It contains a list of all the available tool, methods, repo, code etc to detect hallucination, LLM evaluation, grading and much more.

A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.

Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate.

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

Evaluating LLMs with CommonGen-Lite

FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts

Code and data for ACL ARR 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"

FactScoreLite is an implementation of the FactScore metric, designed for detailed accuracy assessment in text generation. This package builds upon the framework provided by the original FactScore repository, which is no longer maintained and contains outdated functions.