Do we need to consider the chat template when doing DPO/KTO training?

[ZeroYuHuang]

Hi! I closely check the data processing in the KTO trainer and DPO trainer. I find that the chatting template is not considered when combining the prompt and its completion.

For DPOTrainer: full_tokenized = self.tokenizer(prompt + answer, add_special_tokens=False)
And for KTOTrainer: prompt_and_completion = [prompt + completion for prompt, completion in zip(batch["prompt"], batch["completion"])]

Do we need to consider the chat template when combining them, something like: [INST] + prompt + [\INST] + completion.

[younesbelkada]

cc @kashif @kawine

[kashif]

let me check! Thanks for the report!

[yaoxiao1999]

let me check! Thanks for the report!

Hi, didn't mean to chase you up, but do you have any updates on this question? Thanks!