Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functionality to return the lemmas of words used in a corpus. #3256

Open
Sion1225 opened this issue May 21, 2024 · 1 comment
Open

Add functionality to return the lemmas of words used in a corpus. #3256

Sion1225 opened this issue May 21, 2024 · 1 comment

Comments

@Sion1225
Copy link

Sion1225 commented May 21, 2024

I have been working with natural language processing and often needed to know which words were used in certain corpora. Many dictionaries are comprised of word stems, requiring the extraction of stems from sentences. For example, specific words can be key clues or carry important information, necessitating the extraction of sentences using these words, or processing sentence information with them.

In this context, I have developed a class called AutoLemmatizer in stem/wordnet.py that automatically performs tokenization and part-of-speech-based lemmatization, returning the lemmas of all words used in a sentence.

I also considered converting 'n't' to 'not,' but have not implemented because I can't sure that is good idea.

>>> from nltk.stem import AutoLemmatizer
>>> auto_wnl = AutoLemmatizer()
>>> print(auto_wnl.auto_lemmatize('Proverbs are short sentences drawn from long experience.'))
['Proverbs', 'be', 'short', 'sentence', 'draw', 'from', 'long', 'experience', '.']
>>> print(auto_wnl.auto_lemmatize('proverbs are short sentences drawn from long experience.'))
['proverb', 'be', 'short', 'sentence', 'draw', 'from', 'long', 'experience', '.']

Resolves: nltk:#3257

@Sion1225
Copy link
Author

This issue is solved as #3257 full request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant