-
-
Notifications
You must be signed in to change notification settings - Fork 206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Importing config vars from a .csv file #679
Comments
Hey @alexclaydon, is it possible this import from CSV functionality does what you want? It works at the
And the CSV includes a bunch of vars mapped to columns. |
Thanks Ian! First of all, we're loving promptfoo and are going all in on it - thanks for all of your excellent work! Having thought about this issue for a few days and discussed with a colleague, we realised that I should have framed it differently. Please allow me to provide some additional context on the use-case to flesh out the issue as we see it. It's always possible that we've misunderstood the paradigm. We're making extensive use of "scenarios" in our setup, like so: description: ...
providers: ...
prompts: file://prompt_templates.txt # prompt template containing nunjucks tags for `topic` and `description`
scenarios:
- config:
- vars:
topic: "Liabilities for Business Losses"
description: file://case01.txt
- vars:
topic: "Liabilities for Data Losses"
description: file://case02.txt
- vars:
topic: "Liabilities for Other Losses"
description: file://case03.txt
tests: './data.csv' # contains 4 columns: `text` (str), `definitions` (str), `data_loss` (bool), `business_loss` (bool) The crux of the issue for us is that, in scenarios, it doesn't seem to be possible to specify assertions under the Accordingly, if we want to check, for example, whether some output matches the value of one of the bool columns in our data spreadsheet, we need to go down each row and change the name of that column to Ideally, we'd prefer to write a one-line "wrapper" assertion, leveraging I hope that all makes sense and would really appreciate any guidance you might be able to provide! |
Just had an additional thought on your response above which might clear things up for us a bit. You mentioned that "the CSV includes a bunch of vars mapped to columns." Does that imply that I can access column names / vars from within the config file in an assert immediately following description: 'Extraction'
prompts: ['{{ clause }}']
providers:
- id: 'python:../shims/extract.py'
label: 'Extraction'
config:
evaluateOptions.maxConcurrency: 1
evaluateOptions.showProgressBar: true
tests:
- '../testdata/data.csv'
- assert:
- type: icontains-any
value: '{{ entities }}' # where the `.csv` file contains at least `clause` and `entities` columns I've been trying to get that working today, but wasn't able - I also had a look through the If this kind of thing is possible, that would enable us to conveniently "wrap" asserts around row data as I described above - so a complete solution! |
I think the right construction here is: defaultTest:
assert:
- type: icontains-any
value: '{{ entities }}'
tests: '../testdata/data.csv' The |
Thank you Ian! The confirmation that this is supposed to work is very helpful. I had actually tried this approach, but without success. I think I now know why: it appears to relate to the way arrays are handled. Since this particular assert is In addition, I tried enclosing the value of the defaultTest:
assert:
- type: icontains-any
value: ['{{ entities }}']
tests: '../testdata/data.csv' That seemed to resolve the above error about the assert requiring an array, but also resulted in the actual value of the Could you advise on the correct approach here? Unless I missed it, there seems to be a bit of a gap in the documentation here. I would be very happy to submit a documentation PR if we could get to the bottom of this and if that would be helpful? Cheers |
Ok, I see the problem! I went through the whole troubleshooting and debugging process (e.g. using I think the best way to fix this is add comma-delimited string support for defaultTest:
assert:
- type: icontains-any
value: '{{ entities }}'
tests: '../testdata/data.csv' and you can include an #755 implements this (although it naively splits on any comma, and won't respect CSV-style quotes like |
Perfect! Thanks so much for this, really appreciate the attention. That's the final piece of the puzzle for us on pushing Promptfoo to adoption across our whole team, incredibly helpful! |
Sorry for the delay here @alexclaydon - it's merged and will be in the next release! |
Not at all - we really appreciate you talking the time on this! |
@typpo thank you for this explanation, it helped me to be able to use csv files to test on large datasets. |
Hello, I am also testing LLM from CSV files. However I am led to use a custom Python script to calculate some metrics in the assertions. In the
My project structure looks like this: ├── src The folder After executing the command line at the root project:
It returns the following error:
Is there a nice hack to make it work ? |
Hello, having had a look through the documentation (Import vars from separate files) and examples, it appears as though importing vars from external files is limited to yaml, txt, Python and Javascript files.
Are there any plans to allow import from a csv file, where each column header is a variable name, and each row is one single complete configuration? We feel this would be particularly helpful when composing "scenarios" where you have complex setup config to apply to each test.
The text was updated successfully, but these errors were encountered: