Update caching and add tokenizer to create_states_mapping
#911
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brandonwillard
force-pushed
the
use-persistent-tokenizer-hash
branch
2 times, most recently
from
8136133
to
5d795cf
Compare
brandonwillard
added
transformers
brandonwillard
force-pushed
the
use-persistent-tokenizer-hash
branch
from
5d795cf
to
bf5e3f8
Compare
brandonwillard
changed the title
Tokenizer hash and add tokenizer to cache signaturecreate_states_mapping caching
brandonwillard
force-pushed
the
use-persistent-tokenizer-hash
branch
from
bf5e3f8
to
6fc7c3d
Compare
rlouf
approved these changes
May 22, 2024
brandonwillard
force-pushed
the
use-persistent-tokenizer-hash
branch
from
6fc7c3d
to
47f6bd8
Compare
brandonwillard
changed the title
create_states_mapping cachingcreate_states_mapping
brandonwillard
force-pushed
the
use-persistent-tokenizer-hash
branch
3 times, most recently
from
917e2c9
to
3401637
Compare
brandonwillard
force-pushed
the
use-persistent-tokenizer-hash
branch
from
3401637
to
ab47129
Compare
These changes make the `cache` decorator operate more like `diskcache`'s existing `memoize` method. They also remove the use of hash value as store keys.
brandonwillard
force-pushed
the
use-persistent-tokenizer-hash
branch
from
ab47129
to
f2d6464
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes the use of hash values as the keys to the underlying function call cache. It also adds the tokenizer to the arguments of
create_states_mapping.The main change here is that
hash_argumentshas been removed and replaced with aDisksubclass,CloudpickleDisk, that usescloudpickle. This allows all the same objects to be cacheable as before, while also removing the use of hash values as keys. The latter is especially important, since hash collisions at the key level can return the wrong results from the cache.Likewise, the
key_functionargument tocachehas been removed. If one needs this sort of functionality, they can wrap the corresponding arguments with objects that properly implement serialization.Closes #872