-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Issues: EleutherAI/lm-evaluation-harness
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Add TensorRT-LLM support
feature request
A feature that isn't implemented yet.
#1910
opened Jun 1, 2024 by
taewan2002
OpenaiCompletionsLM invokes the completions API with max_tokens set to 0
#1903
opened May 29, 2024 by
chimezie
can we add C4 and PTB tasks for PpL?
feature request
A feature that isn't implemented yet.
#1884
opened May 25, 2024 by
123wujiao
Add Regression Testing
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1883
opened May 24, 2024 by
haileyschoelkopf
Llama2-hf-q40.gguf model got very poor results on lambada_openai tasks, but was fine on other tasks.
#1866
opened May 21, 2024 by
intellinjun
--device cuda:3 not honored when using --model vllm
bug
Something isn't working.
documentation
Improvements or additions to documentation.
#1846
opened May 15, 2024 by
LGLG42
AssertionError: aggregation named 'mean' conflicts with existing registered aggregation!
#1839
opened May 14, 2024 by
hunter2009pf
Evaluation results of llama2 with lm-evaluation-harness using wikitext-2
#1833
opened May 13, 2024 by
l2002924700
Using Language Models as Evaluators
feature request
A feature that isn't implemented yet.
#1831
opened May 13, 2024 by
lintangsutawika
Previous Next
ProTip!
no:milestone will show everything without a milestone.