About the "limited to a maximum of three words" problem of multi-word-synonyms

[loulazynote]

I saw on document that there is a limit of 3 single words in multi-word synonyms. It because I mostly index document in Chinese. I don't know how to count "words" in Chinese. The following sample is my real synonyms. It actually can't be found when I search for the synonym.

{
  "信託業商業同業公會金融專業訓練機構暨信託業務相關課程認可標準":
    [
       "申請成為訓練機構",
       "申請成為金融訓練機構", 
       "認可標準", 
       "金融機構認可標準"
    ],
   "申請成為金融訓練機構":
     [
        "信託業商業同業公會金融專業訓練機構暨信託業務相關課程認可標準",
        "申請成為訓練機構",
        "認可標準",
        "金融機構認可標準"
     ]
} 

So my main question is that how can I make these terms be able to searched in Meilisearch? Is it already above the multi-word-synonyms limit? Can I expand the limit by simply change some configuration? It not, can I modify the Meilisearch source code and build my own version?

[oluademola]

Hello @loulazynote. Indeed this often happens when you've exceeded the multi-word limit. The Jeiba tokenizer is responsible for splitting Chinese text into words. Typically this would analyze the text with a Chinese dictionary to infer segments. Today, meilisearch doesn't support customizing the tokenizer, but I'd let @ManyTheFish or the @meilisearch/engine-team give you the best answer as regards this issue.

[ManyTheFish]

Hello @loulazynote and @oluademola,
The first synonym is segmented as ["信", "託", "業", "商", "業", "同", "業", "公", "會", "金融", "專", "業", "訓", "練", "機", "構", "暨", "信", "託", "業", "務", "相", "關", "課", "程", "認", "可", "標", "準"], unfortunately it would be complicated to handle this with the current version of Meilisearch.
However, it seems that you want to set complete phrases as synonyms which is a quite exotic usage for this feature, we probably have to rethink it 🤔

[doggy8088]

I see you removed the note of the limitation from the doc. Why?

[ManyTheFish]

Yes, you are right. It's a mistake; I've created an issue to fix the documentation:
meilisearch/documentation#2676 (comment)

Thank you for the report.

[doggy8088]

@ManyTheFish Do you have any plan increase the limitation? 3 is way to small for me.

[macraig]

@doggy8088 can you tell us a bit more about your needs? What number of words would make sense for your use case?

However, the dictionary setting gives an interesting workaround by allowing you to force Meilisearch to consider a certain character sequence as a word and bypass the synonym limitation.
It may not work for everything but it should enhance your experience with synonyms. 😄

Have you tried the workaround @ManyTheFish suggested above?

[ManyTheFish]

@macraig, Yes, it works for the synonyms, but using the dictionary setting imposes a specific shape to any substring matching the word in the dictionary, which may be restrictive and inconsistent in languages where the segmentation is based on the context, like Chinese or Japanese.
Because the same substring of character can be segmented differently based on the context. :S