Code example of using SpTokenizer

[dripp55]

Can someone please point me at a complete code example of constructing and using SpTokenizer to tokenize a given sentence? I saw in the JavaDocs that its constructor requires a model argument - what can I enter there?
Can I train it from scratch without a pre-trained model?

[frankfliu]

@dripp55

You can find SpTokenizer examples in this test code: https://github.com/deepjavalibrary/djl/blob/master/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTokenizerTest.java#L31

You can also find text embedding examples here: https://github.com/deepjavalibrary/djl/blob/master/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTextEmbeddingTest.java

We only support run inference with pre-trained model. You can train it with python code and use DJL to load the model.

By the way, we have Huggingface tokenizer support in DJL, it can also load sentencepiece model.

[dripp55]

Thank you, @frankfliu !
From your answer I understand that SpTokenizer is not what I am looking for. I need to find (or I may need to build) a good Java tokenizer that I can train from scratch on my data.

[frankfliu]

@dripp55
Why you have to train tokenizer in Java? Are trying to implement a online training feature? If you can train offline, it doesn't matter much using Python or java.

[dripp55]

I will need the same tokenizer to train on my data and then to be used online for tokenizing based on the same vocabulary that it created from my specific data in training and in the same methodology. I also prefer to run everything in Java for performance and maintenance purposes. If I need to tune or improve the tokenizer - it must be in one code and not in two languages.

[frankfliu]

I agree from maintenance purpose, the same language for inference and train is the best choice. But in reality, we don't have demand on training. Most of our customer train offline using Python and run inference in Java. That's why we didn't expose training feature for SpTokenizer.

Both Python and use use the same C++ native library. Java and Python only provide the frontend API. The model trained in python can be used in Java, and the result are identical.