Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Option to plot train/eval plots with epoch instead of step on x-axis #700

Open
tmostak opened this issue May 8, 2024 · 0 comments
Labels
type/feature Feature request

Comments

@tmostak
Copy link

tmostak commented May 8, 2024

馃殌 Feature

Allow epoch to be optionally used as the x-axis of training/eval charts for easier comparison between runs with different amounts of training pairs.

Motivation

I'm often experimenting with adding or removing subsets of training data, either at dataset creation time by adding or deleting categories of prompts, or via the sampling slider in the H2O LLM Studio UX. Currently, all training and eval loss and perplexity graphs (and any other metric graphs) use step as the x-axis, but it would be easier to compare training runs with different numbers of training pairs if you could switch the x-axis to epoch, that way two training runs X% of the way finished would line up. See the image below for an example of how it is hard to compare two training runs with dissimilar numbers of training pairs (here for the shorter run I removed various prompt templates teaching specific skills/tasks from the training dataset, as I want to see if that improves performance on one specific task I care about and which solely makes up the eval set).

Screenshot 2024-05-08 at 2 14 21鈥疨M
@tmostak tmostak added the type/feature Feature request label May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature Feature request
Projects
None yet
Development

No branches or pull requests

1 participant