ConstantLengthDataset Ignore Some Texts

[TianyiPeng]

Not a big concern. But the current implementation will ignore the last chunk of the data which is the remainder of the self.seq_length. Maybe worth adding the last chunk back with padding tokens.

            for i in range(0, len(all_token_ids), self.seq_length):
                input_ids = all_token_ids[i : i + self.seq_length]
                if len(input_ids) == self.seq_length:
                    examples.append(input_ids)

https://github.com/huggingface/trl/blob/main/trl/trainer/utils.py#L475-L478

[younesbelkada]

Thanks @TianyiPeng for the suggestion, the reason we don't add that is because we'll have to introduce custom attention masks. This will lead to slower training for setups that use for instance Flash Attention 2