token probabilities are not included

[dakinggg]

It seems the config isn't quite right to include token probabilities. I'm not 100% sure of the solution, but this issue should help (explosion/spaCy#6388 (comment)). The vocab files include the probabilities, but I guess that isn't enough.

[dakinggg]

This is how you would add the probabilities from the base spacy models

import spacy
from spacy.lookups import load_lookups
nlp = spacy.load("en_core_web_sm")
lookups = load_lookups("en", ["lexeme_prob"])
nlp.vocab.lookups.add_table("lexeme_prob", lookups.get_table("lexeme_prob"))