Maximum input token count 4919 exceeds limit of 4096 for train data in 03_Model_customization/03_continued_pretraining_titan_text.ipynb

[FireballDWF]

"Maximum input token count 4919 exceeds limit of 4096 for train data" in model-customization-job/amazon.titan-text-lite-v1:0:4k/nhjsh25oes0i in notebook 03_Model_customization/03_continued_pretraining_titan_text.ipynb

[HiDhineshRaja]

I too face the same issue. I tried to reduce the chunksize to 10000 even though getting the same error after training about 2 hours

[jicowan]

I am also getting this error.

[nmudkey000]

I was able to fix the issue by reducing the chunk size and chunk overlap to 5000 and 1000, respectively; 5000 was an assumption, I am sure it would still create the model with anything below 10000 (someone above observed that the model would not get created for a chunk size equaling 10000):

text_splitter = RecursiveCharacterTextSplitter(
# Set a really small chunk size, just to show.
chunk_size = 5000, # assumption
chunk_overlap = 1000, # overlap for continuity across chunks
)

docs = text_splitter.split_documents(document)