How to use lemmatization together with YAKE?

[PrimozGodec]

Dear authors, thank you for the great keyword extracting methods.

When extracting keywords with YAKE it usually happens that in-between keywords there are two words with the same lemma (e.g. tree, trees). Since it would be difficult (and exposed to errors) to lemmatize text and the contact text together before using YAKE, I was thinking to lemmatize words after keyword extraction. I am wondering what is the best way to aggregate scores after lemmatization. Should I use sum, mean or max function for aggregating scores of the words with the same lemma or do you suggest any alternative method? Do you have any experience with my issue?

[juhoinkinen]

Hi, Just dropping here to say that I have been wondering the same. Our case is how to combine the scores of keywords being synonyms (a vocabulary/ontology is used to match the synonyms).

Currently the approach is first to transform the scores (limiting to zero from below, then inverting the score range from interval [0, Inf] to [1, 0], so the score of best possible keyword is 1; this transformation is needed in our application in any case). Then, just for combining two scores, the scores are scaled to range [0.5, 1], conflated and finally scaled back to [0, 1], code here. This way finding several forms (synonyms) of the same subject will always increase the score of the subject (never decrease as when taking mean), while the upper limit of 1 is approached smoothly (in contrast to summing).

But I would like hear if anyone has found a better way to this, or how to match keywords to vocabulary? Now the matching is done basically by lemmatizing both vocabulary terms and keywords, but it's not working very well.

[VictorSuarezL]

@PrimozGodec I came here with the same doubt. However, why do you say not to lemmatize before the keyword extraction? I think it should definitely solve the problem. Am I missing something?

[PrimozGodec]

@VictorSuarezL I agree with you that it would be better to lemmatize before the keyword extraction, but if I understand YAKE correctly it needs complete text on the input including stopwords and sentences. I got the idea to split sentences on tokens lemmatize them and then join them back together (including punctuation - as I understand it is important). It is possible but I am worried about errors I would make in this procedure. Do you have any other ideas?