Wordforms are not effective when exceptions are present

[sanikolaev]

Bug Description:

If you specify both wordforms and exceptions in a recent dev version, the wordforms are not effective:

MRE:

snikolaev@dev2:~$ echo "a > b" > /tmp/wf; echo "123 => 345" > /tmp/exceptions; mysql -P9306 -h0 -v -e "drop table if exists t; create table t(f text) wordforms='/tmp/wf' exceptions='/tmp/exceptions'; call keywords('a 123', 't')"
--------------
drop table if exists t
--------------

--------------
create table t(f text) wordforms='/tmp/wf' exceptions='/tmp/exceptions'
--------------

--------------
call keywords('a 123', 't')
--------------

+------+-----------+------------+
| qpos | tokenized | normalized |
+------+-----------+------------+
| 1    | a         | a          |
| 2    | 345       | 345        |
+------+-----------+------------+

Expected b in the normalized column:

+------+-----------+------------+
| qpos | tokenized | normalized |
+------+-----------+------------+
| 1    | a         | b          |
| 2    | 345       | 345        |
+------+-----------+------------+

which can be achieved if I remove the exceptions:

snikolaev@dev2:~$ echo "a > b" > /tmp/wf; echo "123 => 345" > /tmp/exceptions; mysql -P9306 -h0 -v -e "drop table if exists t; create table t(f text) wordforms='/tmp/wf'; call keywords('a 123', 't')"
--------------
drop table if exists t
--------------

--------------
create table t(f text) wordforms='/tmp/wf'
--------------

--------------
call keywords('a 123', 't')
--------------

+------+-----------+------------+
| qpos | tokenized | normalized |
+------+-----------+------------+
| 1    | a         | b          |
| 2    | 123       | 123        |
+------+-----------+------------+

Can't reproduce in 6.2.12

Manticore Search Version:

Manticore 6.2.13 56052bf8a@24051721 dev (columnar 2.2.5 a915428@24051706) (secondary 2.2.5 a915428@24051706) (knn 2.2.5 a915428@24051706)

Operating System Version:

Ubuntu Jammy (dev2)

Have you tried the latest development version?

• Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Task estimated
• Specification created, reviewed, and approved
• Implementation completed
• Tests developed
• Documentation updated
• Documentation proofread
• Changelog updated

[tomatolog]

seems strange but on windows box I see

call keywords('a 123', 't')
--------------

+------+-----------+------------+
| qpos | tokenized | normalized |
+------+-----------+------------+
| 1    | a         | b          |
| 2    | 345       | 345        |
+------+-----------+------------+

and at linux box

mysql> call keywords('a 123', 't');
+------+-----------+------------+
| qpos | tokenized | normalized |
+------+-----------+------------+
| 1    | a         | b          |
| 2    | 123       | 123        |
+------+-----------+------------+

seems the exception does not get applied but wordforms works as usual or all works as usual

[tomatolog]

seems dev2 box issue with files at the /tmp/
as I tried with the files at the home directory and all works fine

mysql> create table t(f text) wordforms='/home/stas/bin/txt/wf1' exceptions='/home/stas/bin/txt/e1';
mysql> call keywords('a 123', 't');
+------+-----------+------------+
| qpos | tokenized | normalized |
+------+-----------+------------+
| 1    | a         | b          |
| 2    | 345       | 345        |
+------+-----------+------------+

[tomatolog]

added comment to #2154 to make sure all cases are covered and does not pop up from nother as described here

[sanikolaev]

Can't reproduce it either anymore. Closing then.