New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
experiment.evaluate() shows stale evaluation results #79
Comments
Your observation is correct. Currently, if a metric already exists (which is If you change the metric name given in the second We are open to considering overwriting it even when the existing metric already exists. Let us know what you think. |
Thank you for this issue, I changed the variable name but still, the response column is stale. Any leads on this issue? I use python version 3.11.5 |
Hi @Sruthi5797, Can you post a minimal code snippet of what you are running? Also, are you seeing any warning message? |
馃悰 Describe the bug
Hi folks,
Thanks again for your work on this library.
I noticed an issue where similarity scores do not get updated when I change my
expected
fields. Only when I re-run the experiment are the values updated.Bug
Steps to reproduce:
In my opinion,
evaluate()
should re-compute metrics every time it is run, rather than depending/being coupled to another function (run()
). I haven't tested it on other eval_fns, but it could be worth testing if this is the case as well.The text was updated successfully, but these errors were encountered: