Hi @ananegru, thanks for opening this issue!

The reason this error is being raised is because the tiiuae/falcon-7b tokenizer doesn't have a cls_token set. This is used in the example to indicate when an q/a isn't answerable e.g. if there isn't an answer or the answer happens outside of the span.

In order to get this example to work for this model, I'd suggest setting the tokenizer's cls token to an equivalent token that can represent this, or adapting the script to filter out these problem examples.

cc @Rocketknight1

Yes, this script is rather outdated - with modern models, it is more common to do QA by just providing the text and asking an instruct model directly!

We can probably work around this by just setting the value to 0 (and emitting an 'empty' answer at the start of the sequence) when no CLS token is present, since the CLS token location is only used to create 'dummy' spans for impossible answers. I'll make a PR now!

@ananegru a PR is ready. Can you try it out and let us know if it works for you? You can install the PR branch with this command:

pip install --upgrade git+https://github.com/huggingface/transformers.git@fix_qa_example

Let us know if you can train Falcon after running it!

Hi, thanks a lot for the help! However, when I try running the command you sent above I get the following error:

Collecting git+https://github.com/huggingface/transformers.git@fix_qa_example
Cloning https://github.com/huggingface/transformers.git (to revision fix_qa_example) to /gpfs/scratch1/nodespecific/int4/anegru/pip-req-build-ji9_wocz
Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /gpfs/scratch1/nodespecific/int4/anegru/pip-req-build-ji9_wocz
WARNING: Did not find branch or tag 'fix_qa_example', assuming revision or ref.
Running command git checkout -q fix_qa_example
error: pathspec 'fix_qa_example' did not match any file(s) known to git
error: subprocess-exited-with-error

× git checkout -q fix_qa_example did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git checkout -q fix_qa_example did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

@ananegru I think this is because the branch was deleted after merge. The commit is now on main - you can get this by installing from source pip install git+https://github.com/huggingface/transformers

Unfortuantely, the same issue regarding the valueerror for the CLS token is persisting, even after running the command you sent above as well, anything else that might be done to fix it?

@ananegru Is this occurring when using the falcon checkpoint? Unfortunately, it's not possible to use this model here - the QA script is designed for MLM, whereas falcon is a CLM.

Ah yes, its occurring at the checkpoint. Okay so I just can't use this script at all for fine-tuning falcon? Do you perhaps know of any other scripts to fine-tune a CLM for QA?

There isn't a script currently in the library, however as there's a FalconForQuestionAnswering head, this should probably be supported cc @Rocketknight1

I'll investigate! But also @ananegru, is there a reason you specifically want a CLM for this kind of span-extraction task? The most common approaches for question answering in 2024 are:

• Use a masked language model to extract spans from the input text
• Use a chat language model, give it the text, and directly ask it questions

The second option is harder to fine-tune for, but the base accuracy will be very high if you use a state-of-the-art chat model like LLaMA-3, DBRX, Mixtral or Command-R.