Skip to content

Evaluation of the Optical Character Recognition (OCR) capabilities of GPT-4V(ision)

Notifications You must be signed in to change notification settings

SCUT-DLVCLab/GPT-4V_OCR

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

69 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GPT-4V_OCR


[arXiv 2310.16809]Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation

This paper presents a comprehensive evaluation of the Optical Character Recognition (OCR) capabilities of the recently released GPT-4V(ision), a Large Multimodal Model (LMM). We assess the model's performance across a range of OCR tasks, including scene text recognition, handwritten text recognition, handwritten mathematical expression recognition, table structure recognition, and information extraction from visually-rich document. The evaluation reveals that GPT-4V performs well in recognizing and understanding Latin contents, but struggles with multilingual scenarios and complex tasks. Based on these observations, we delve deeper into the necessity of specialized OCR models and deliberate on the strategies to fully harness the pretrained general LMMs like GPT-4V for OCR downstream tasks. The study offers a critical reference for future research in OCR with LMMs.

Scene Text Recognition

results_str
Scene Text Recognition (STR) aims to recognize textual information in natural scene pictures.

Handwritten Text Recognition

results_htr
Handwritten Text Recognition (HTR) aims to recognize handwritten text.

Handwritten Mathematical Expression Recognition

results_hmer
Handwritten Mathematical Expression Recognition (HMER) aims to recognize handwritten mathematical formulas.

Visual Information Extraction

results_vie results_vie recommendations_vie

To learn more about Visual Information Extraction, please refer to Document-AI-Recommendations.

Visual Information Extraction(VIE) aims at mining, analyzing, and extracting key fields entities contained in visually rich documents. For example, given an image of a receipt, the VIE algorithms will tell information such as store name, product details, price, etc. For documents like forms, VIE algorithms will tell the key-value pairs contained.

Table Structure Recognition

results_tsr recommendations_tsr

To learn more about Table Structure Recognition, please refer to Document-AI-Recommendations.

Table Structure Recognition(TSR) aims to recognize the cellular structures of tables from table images by extracting the coordinates of cell boxes and row/column spanning information. This task is very challenging since tables may have complex structures, diverse styles and contents, and become geometrically distorted or even curved during an image capturing process.

Data Download

Citation

@misc{shi2023exploring,
      title={Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation}, 
      author={Yongxin Shi and Dezhi Peng and Wenhui Liao and Zening Lin and Xinhong Chen and Chongyu Liu and Yuyi Zhang and Lianwen Jin},
      year={2023},
      eprint={2310.16809},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

About

Evaluation of the Optical Character Recognition (OCR) capabilities of GPT-4V(ision)

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages