Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporarily disable expensive text metrics #3953

Open
arnavgarg1 opened this issue Mar 5, 2024 · 0 comments
Open

Temporarily disable expensive text metrics #3953

arnavgarg1 opened this issue Mar 5, 2024 · 0 comments

Comments

@arnavgarg1
Copy link
Contributor

arnavgarg1 commented Mar 5, 2024

This issue has been created to track that the following text metrics have been temporarily disabled because they are expensive to compute:

  1. BLEU
  2. ROUGE
  3. WordErrorRate
  4. CharacterErrorRate

We can re-enable them at a different time by allowing users to customize which metrics they want to see during training, leaving the onus on the user to decide how much extra time they are willing to spend training to get more training metrics.

For reference, this is how BLEU and ROUGE scale with a sample of 200 rows over different sequence lengths:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant