Skip to content

Proof of concept of training a simple Region Classifier using PdfPig and ML.NET (LightGBM). The objective is to classify each text block in a pdf document page as either title, text, list, table and image.

Notifications You must be signed in to change notification settings

BobLd/PdfPigMLNetBlockClassifier

Repository files navigation

PdfPig ML.Net Block Classifier v2

Proof of concept of training a simple Region Classifier using PdfPig and ML.NET. The objective is to classify each text block in a pdf document page as either title, text, list, table and image.

AutoML model builder was used. The model was trained on a subset of the PubLayNet dataset. See their license here.

Generation MicroAccuracy MacroAccuracy
v1 0.937 0.748
v2 0.952 0.801

For v1 model results, see here.

Results

Results are based on PubLayNet's validation dataset, where the page segmentation is known. For real life use, a page segmenter will be needed (see PdfPig's PageSegmenters). The quality of the page segmentation will impact the results.

Metrics for multi-class classification model

MicroAccuracy = 0.9523, a value between 0 and 1, the closer to 1, the better
MacroAccuracy = 0.8009, a value between 0 and 1, the closer to 1, the better
LogLoss = 3.5333, the closer to 0, the better

LogLoss for class 0 (title)         = 4.6289, the closer to 0, the better
LogLoss for class 1 (image)         = 1.849, the closer to 0, the better
LogLoss for class 2 (text)          = 2.5834, the closer to 0, the better
LogLoss for class 3 (list)          = 28.8412, the closer to 0, the better
LogLoss for class 4 (table)         = 2.8326, the closer to 0, the better

F1 Score for class 0 (title)       = 0.9305, a value between 0 and 1, the closer to 1, the better
F1 Score for class 1 (image)       = 0.9412, a value between 0 and 1, the closer to 1, the better
F1 Score for class 2 (text)        = 0.9691, a value between 0 and 1, the closer to 1, the better
F1 Score for class 3 (list)        = 0.2963, a value between 0 and 1, the closer to 1, the better
F1 Score for class 4 (table)       = 0.9611, a value between 0 and 1, the closer to 1, the better

Confusion table

          ||=======================================================
PREDICTED ||     0 |     1 |     2 |     3 |     4 | Total | Recall
TRUTH     ||=======================================================
(title) 0 || 1,848 |     0 |    89 |     1 |     0 | 1,938 | 0.9536
(image) 1 ||     0 |    72 |     3 |     0 |     1 | 76    | 0.9474
(text)  2 ||   185 |     1 | 8,837 |    16 |     9 | 9,048 | 0.9767
(list)  3 ||     1 |     0 |   225 |    52 |     2 | 280   | 0.1857
(table) 4 ||     0 |     4 |    35 |     2 |   654 | 695   | 0.9410
          ||=======================================================
Precision ||0.9086 |0.9351 |0.9617 |0.7324 |0.9820 |

Permutation Feature Importance

PFI works by taking a labeled dataset, choosing a feature, and permuting the values for that feature across all the examples, so that each example now has a random value for the feature and the original values for all other features. The evaluation metric (e.g. micro-accuracy) is then calculated for this modified dataset, and the change in the evaluation metric from the original dataset is computed. The larger the change in the evaluation metric, the more important the feature is to the model. PFI works by performing this permutation analysis across all the features of a model, one after another. - Source

Micro Accuracy

Feature Description Change in MicroAccuracy 95% Confidence in the Mean Change in MicroAccuracy
wordsCount Words count -0.1144 0.0008237
charsCount Characters count -0.09849 0.0009115
linesCount Lines count -0.04255 0.0005726
blockAspectRatio Ratio between the block's width and height -0.04159 0.0005134
bestNormEditDistance Minimum edit distance between bookmark title and block text -0.04016 0.0006307
deltaToHeight Average delta to average page glyph height -0.03689 0.0003823
pctNumericChars % of numeric characters -0.03375 0.0005174
pctSymbolicChars % of symbolic characters -0.01035 0.000358
pctAlphabeticalChars % of alphabetical characters -0.00859 0.0003075
pctBulletChars % of bullet characters -0.004071 0.0002063
pathsCount Paths count -0.003661 0.0001184
pctHorPaths % of horizontal paths -0.003431 0.0001309
imageAvgProportion Average area covered by images -0.001684 4.68E-05
pctVertPaths % of vertical paths -0.001448 8.139E-05
pctOblPaths % of oblique paths -0.0001883 1.734E-05
imagesCount Images count -9.692E-05 1.127E-05
pctBezierPaths % of Bezier curve paths 6.212E-22 1.352E-05

Macro Accuracy

Feature Description Change in MacroAccuracy 95% Confidence in the Mean Change in MacroAccuracy
charsCount Characters count -0.1339 0.002263
linesCount Lines count -0.08378 0.0009925
deltaToHeight Average delta to average page glyph height -0.06461 0.001223
blockAspectRatio Ratio between the block's width and height -0.05896 0.001163
bestNormEditDistance Minimum edit distance between bookmark title and block text -0.05039 0.001419
pctNumericChars % of numeric characters -0.0475 0.001258
pathsCount Paths count -0.0252 0.000614
wordsCount Words count -0.01651 0.001655
pctSymbolicChars % of symbolic characters -0.01166 0.001505
pctHorPaths % of horizontal paths -0.00984 0.0003186
pctVertPaths % of vertical paths -0.008558 0.0002925
pctAlphabeticalChars % of alphabetical characters -0.006882 0.0009913
imageAvgProportion Average area covered by images -0.006148 0.0001146
pctBulletChars % of bullet characters -0.005319 0.0008375
pctOblPaths % of oblique paths -0.002747 6.657E-05
imagesCount Images count -0.00268 3.903E-05
pctBezierPaths % of Bezier curve paths -4.078E-05 5.263E-05

TO DO

Features

  • Add a decoration score/flag
  • Add block's area ratio: the ratio between block area and the page area, cf.
  • Add block's font style: an enumerated type, with possible values: regular, bold, italic, underline, cf.
  • Add % sparse lines in a block, for better table recognition cf.
  • Font color distance from most common color

About

Proof of concept of training a simple Region Classifier using PdfPig and ML.NET (LightGBM). The objective is to classify each text block in a pdf document page as either title, text, list, table and image.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages