On the issue of Chinese full-text search matching

[jerryyang-git]

I use Docker's latest image deployment
My data is like this

        {
            "alias": "ナルキッソス\nナルキ\n水仙",
            "vvid": 22,
            "id": "v10",
            "image": "cv8900"
        }

I set alias as the search field and use /indexes/{index_uid}/settings/separator token to set the delimiter to \n
But when I search for 水仙 , I cannot obtain this data, but if I add extra words to the search, such as キ水仙, I can find it. I don't understand the problem with this

I have another piece of data in a similar form, and when I search for 死馆, I can accurately match it

        {
            "alias": "Shi ni Yuku Kimi, Yakata ni Mebuku Zouo\nにくにく\nNikuniku\n死馆",
            "vvid": 39669,
            "id": "v19233",
            "image": "cv29481"
        }

At present, I suspect that Meilisearch segmented the search terms during Chinese search, which seems to be incorrect. However, I am just a beginner in Python and I am confused about this. I need help

Also, English is not my native language. I use a translator, and although I have confirmed through back and forth translation, there are strange words in the sentences. Please do not hesitate to ask your questions

[ManyTheFish]

Hello @jerryyang-git,
I may know the source of your issue. Meilisearch has difficulties in searching in documents containing both Chinese and Japanese Languages.
I have a question: why do you have Japanese characters in a Chinese document?

Thank you for our issue,
see you

[jerryyang-git]

Hello @ManyTheFish
I'm very sorry for taking so long to reply. Thank you very much for your attention
My data was obtained from vndb.org, where alias represents the alias of the game. As this is not an official name, it may be mixed with many languages, such as Chinese, English, Japanese, Korean, etc. They use \n to separate each name

[ManyTheFish]

Ok, I understand,
unfortunately, Meilisearch has issues when Chinese and Japanese are mixed in the same index, and even more when both languages are mixed in the same attribute. It's due to the segmentation and the normalization of both Languages that are treated completely differently by Meilisearch, meaning that the same character will be transformed differently depending on how Meilisearch detects the language. 😞

The only fix I see is to implement a setting to let you choose the primary language of your index. Could you please upvote the related product discussion?
https://github.com/orgs/meilisearch/discussions/702

Thanks!

[jerryyang-git]

Thank you very much for replying to me
I used the Japanese version of Docker image and I am very pleased that the issue above has been resolved
But due to the mandatory Japanese language, I completely lost the search function in Chinese
Perhaps my situation is more extreme, or perhaps I should classify the data to some extent