Packing的问题咨询

[447428054]

Reminder

• I have read the README and searched the existing issues.

Reproduction

total_length = len(input_ids)
block_size = data_args.cutoff_len
# we drop the small remainder, and if the total_length < block_size, we exclude this batch
total_length = (total_length // block_size) * block_size
# split by chunks of cutoff_len
for i in range(0, total_length, block_size):
    if not all(label == IGNORE_INDEX for label in labels[i : i + block_size]):
        model_inputs["input_ids"].append(input_ids[i : i + block_size])
        model_inputs["attention_mask"].append([1] * block_size)
        model_inputs["labels"].append(labels[i : i + block_size])

return model_inputs

packing是将输入按照指定长度切分开，所以单轮数据的拼接是在数据集层面自己拼接吗？

Expected behavior

No response

System Info

No response

Others

No response

[codemayq]

不是很明白你的问题是什么，数据不管是单轮还是多轮都按照数据集的格式要求分字段存入即可。