Skip to content

Issues: huggingface/lighteval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

38 Open 36 Closed
38 Open 36 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Issue when saving results with fsspec==2023.12.1
#191 opened May 15, 2024 by Hamza-Alobeidli
Zero scores on cnn-dm benchmark from HELM
#188 opened May 13, 2024 by hicleo
Evaluate EncoderDecoderModels
#183 opened May 3, 2024 by Bachstelze
2
Performance compared to lm-evaluation-harness
#179 opened Apr 29, 2024 by geoalgo
6
Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!
#176 opened Apr 26, 2024 by gody7334
3
Error: ModuleNotFoundError: No module named 'openai'.
#175 opened Apr 26, 2024 by PhilipMay
5
Add Sympy equivalence for MATH / GSM8K?
#170 opened Apr 23, 2024 by lewtun
1
Could not initialize the JudgeOpenAI model and openi import error bug Something isn't working
#166 opened Apr 22, 2024 by lewtun
@NathanHB
1
DROP Evaluation with Llama3 (vs. lm-evaluation-harness)
#165 opened Apr 21, 2024 by vipulraheja
1
Expose a few model predictions / gold answers in the logs
#164 opened Apr 21, 2024 by lewtun
1
MMLU evaluation fails with Mistral
#159 opened Apr 18, 2024 by sanchit-gandhi
6
add transformers model to be used as judge feature request New feature/request
#153 opened Apr 10, 2024 by NathanHB
2
LatexTableWriter created but never used.
#151 opened Apr 9, 2024 by PhilipMay
1
Add Code-Centric Interface to LightEval for Enhanced Usability
#148 opened Apr 7, 2024 by adithya-s-k
4
Add a nanotron model to the test suite
#144 opened Apr 4, 2024 by clefourrier
Do an intro notebook on how to use lighteval documentation Improvements or additions to documentation
#143 opened Apr 3, 2024 by clefourrier
2
[New Task] Add AlpacaEval LC feature request New feature/request new task
#139 opened Apr 2, 2024 by YannDubs
9
Add a logger in the metric functions feature request New feature/request
#135 opened Mar 29, 2024 by NathanHB
@NathanHB
Winogrande degraded results bug Something isn't working
#132 opened Mar 27, 2024 by opherlieber
@clefourrier
5
human eval run
#130 opened Mar 26, 2024 by meitalbensinai
5
Homogeneize logging system
#118 opened Mar 19, 2024 by clefourrier
1
@clefourrier
Add EQ Bench new task
#114 opened Mar 18, 2024 by lewtun
1
Collate items in GenerativeTaskDataset by similar EOS token bug Something isn't working
#100 opened Mar 7, 2024 by clefourrier
To remember for version upgrades to keep
#97 opened Mar 6, 2024 by clefourrier
Align GPQA zero-shot / few-shot prompts with paper? feature request New feature/request
#70 opened Feb 27, 2024 by lewtun
@lewtun
3
ProTip! What’s not been updated in a month: updated:<2024-04-16.