The passed-in tokenizer is not being used in the get_scores method.

[Alkacid]

I noticed that a custom tokenizer can be passed in during initialization to tokenize the input documents, but the tokenizer is not used to tokenize the query in the get_scores method. This means that the query needs to be tokenized manually externally. Would it be possible to add the following content at the beginning of the get_scores method：

if self.tokenizer:
    query = self.tokenizer(query)