We are not able to use raw text to fitting a machine learning or deep learning model. We need clean our text first, which means splitting it into words and handling punctuation and case. In fact, there is a whole suite of text preparation methods that we may need to use, and the choice of methods really depends on our natural language processing task. This notebook consists of the following parts:
-
How to get started by developing your own very simple text cleaning tools.
-
How to take a step up and use the more sophisticated methods in the NLTK library.
But, before starting let's select our dataset. In this notebook, I will use text from Jalil Mammadguluzadeh who is notable Azerbaijani writer, great dramatist and democrat man-of-letter. The file includes a piece from writer’s narrative "Kishmish oyunu" story.