You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
would it make sense for there to be a version of chunk_by_title() that operates on tokens rather than characters? using the same soft window and small chunk combining logic, except using tokens rather than characters.
i see that there is also a chunk_by_attention_window() that's part of the staging/destination side of things, so it seems like one could use the 2 in succession.
The text was updated successfully, but these errors were encountered:
would it make sense for there to be a version of
chunk_by_title()
that operates on tokens rather than characters? using the same soft window and small chunk combining logic, except using tokens rather than characters.i see that there is also a
chunk_by_attention_window()
that's part of the staging/destination side of things, so it seems like one could use the 2 in succession.The text was updated successfully, but these errors were encountered: