IUST HTMLCharDet is a java tool for detecting Charset Encoding of HTML web pages. HTMLCharDet stands for HTML Charset Detector and IUST stands for Iran University of Science & Technology.
This tool is in connection with a paper entitled:
Charset Encoding Detection of HTML Documents
A Practical Experience
which was presented In Proceedings of the 11th Asia Information Retrieval Societies Conference (pp. 215-226), Brisbane, Australia, 2015.
Although we wrote a paper to describe the algorithm, but this tool is not just an academic effort to solve charset encoding detection problem for HTML web pages. In fact this tool is an industrial product which is now actively used in a large-scale web crawler, under a load of over than 1 billion web pages.
In order to determine the precision of IUST HTMLCharDet, we compared it with the two famous charset detector tools namely IBM ICU and Mozilla CharDet, against two test scenarios including Encoding-Wise and Language-Wise. Results of the comparisons are presented in the paper, but you can take a glance at them, below. Interested readers are referred to TIKA-2038 for more evaluations and explanations.
Note: In these images Hybrid is the same IUST HTMLCharDet, we called it Hybrid in the paper because it is actually a hybrid mechanism.
In this test scenario, we compared IBM ICU, Mozilla CharDet and the Hybrid mechanism against a corpus of HTML documents. To create this corpus, we wrote a multi-threaded crawler and then we gathered a collection of nearly 2700 HTML pages with various charset encoding types. The code which we wrote for creating this corpus is available in the ./eval/src/main/java/encodingwise/corpus folder of this repository and the created corpus is available via ./eval/test-data/encoding-wise/corpus.zip. Below find the results of comparison ...
Usually, graphical presentation of results makes a better sense ...
I know that the results are odd and incredible indeed, specially for Windows-1251 and Windows-1256 (Cyrillic and Arabic specific charset encodings respectively), but it is just as well. Refer to the paper to know why IUST HTMLCharDet is so accurate from theoretical point of view and to see its accuracy in practice yourself you can find a prepared evaluation project inside the ./eval folder.
In this test scenario, we compared our hybrid mechanism with the two others from language point of view. In this connection, we gathered a collection of URLs of web pages with different languages. The URLs were selected from the URLs of Alexa's top 1 million websites visited from throughout the world. In order to get URLs in the list of a specific language, we investigated URLs with the Internet Top Level Domain (TLD) name of that language/country. For example, the URLs of Japanese web pages were collected from .jp TLD. The results of the evaluation for eight different languages are shown in details in the following table ...
More details about this test can be found in ./eval/test-data/language-wise/results/.
As you can see from this diagram, in this test scenario the improveness in mean average accuracy of IUST HTMLCharDet aginst two other tools is less that from which in the previous test scenario (i.e. 0.14 and 0.10 in Lang-Wise versus 0.38 and 0.69 in Enc-Wise). It is because over than 85% of the websites use UTF-8 as their charset encoding (ref) and as we know from the previous test, compared to other charsets, both IBM ICU and Mozilla CharDet are more accurate when the charset is UTF-8.
<dependency>
<groupId>ir.ac.iust</groupId>
<artifactId>htmlchardet</artifactId>
<version>1.0.1</version>
</dependency>libraryDependencies += "ir.ac.iust" % "htmlchardet" % "1.0.1"In case of using version 1.0.1:
String charset = HTMLCharsetDetector.detect(htmlByteArray, false);
// or
String charset = HTMLCharsetDetector.detect(htmlByteArray, true); // to involve charsets in meta tagsFor the current 1.0.2-SNAPSHOT version:
HTMLCharsetDetector htmlCharsetDetector = new HTMLCharsetDetector();
String charset = htmlCharsetDetector.detect(htmlInputStream);
// or
String charset = htmlCharsetDetector.detect(htmlInputStream, true); // to involve charsets in meta tags