Skip to content

Term frequency–inverse document frequency for Chinese novel/documents implemented in python.

License

Notifications You must be signed in to change notification settings

Jasonnor/tf-idf-python

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

35 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

tf–idf-python

tf-idf, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.

preview

Enter Chinese novel "笑傲江湖" files, each of which is a chapter in the novel, and output the Top-K words and their weights in each chapter.

The purpose of this project is to implement tf-idf, input given a set of files with a specific relationship, and output the tf-idf weight value of each file. Specifically, the "word" with the highest k is displayed and its weight value, as shown in the figure above. Alternatively, you can enter a word and output a weight value for that word in all files.

English can be segmented by blanks, but Chinese cannot. So we used Jieba Chinese text segmentation to collect the corpus of word. The word weighting value is then obtained using the tf-idf algorithm.

In fact, jieba also has built-in "keyword extraction based on td-idf algorithm", but according to its source code, jieba actually only reads one file to calculate TF. The IDF part reads their own custom corpus, so the result is not accurate (not based on the set of related files to calculate the inverse frequency). Specifically, you can try simple tf-idf jieba version here.

Requirements

  • Python 3
  • jieba

Getting Started

Console

git clone https://github.com/Jasonnor/tf-idf-python.git
cd tf-idf-python/src/
python -u tf_idf.py

Sample GUI

python -u main_gui.py

Preview

preview

Sample GUI Result

preview

A list of the weights of the chapters in the "笑傲江湖" dataset, you can see the important keyword rankings for each chapter.

preview

The weight of the word "任我行" in each chapter. You can see that "任我行" played the most in Chapter 28, and the part with the value of 0 can tell that he did not appear.

Contributing

Please feel free to open issues or submit pull requests.

Reference

License

tf–idf-python is released under the MIT License. See the LICENSE file for details.

About

Term frequency–inverse document frequency for Chinese novel/documents implemented in python.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages