benchmark

• Peformance numbers
• Pre-built binaries
• Building
  • Android
  • iOS
  • Windows
  • Generic GCC
  • Raspberry Pi (Raspbian OS)
• Testing
  • Usage
  • Examples

It's easy to assert that our implementation is fast without backing our claim with numbers and source code freely available to everyone to check.

Rules:

• We're running the processing function within a loop for #100 times.
• The positive rate defines the percentage of images with at least #2 MRZ lines. For example, 20% positives means we will have #80 negative images (no MRZ lines) and #20 positives (with MRZ lines) out of the #100 total images. This percentage is important as it allows timing both the detector and recognizer.
• We're using high accuracy for the segmenter and bilinear interpolation.
• All positive images contain a least #2 MRZ lines.
• The initialization is done outside the loop.

The concept of negative and positive images is very important because in most use cases you'll:

• 1. Start the application
• 2. Move the application to the MRZ zone to recognize the data

You only need a single "good frame" to recognize the MRZ lines. But, between step #1 and step #2 the application has probably processed more than #200 frames (40fps * 5sec). So, in such scenario the application have to process #201 frames before reaching the "good frame": #200 negatives and #1 positive. If processing negative frame is very slow then, the application won't be able to catch this "good frame" at the right time. A slow application will do one of these strategies:

• 1. Drop frames to keep the impression that the application is running at realtime: In such scenario the positive frames will most likely be dropped (probability = 1/201 = 0.49%) which means reporting the time when this single "good frame" is caught.
• 2. Enqueue the frames and process them at the application speed: This is the worse solution because you could run out of memory and when the application is running slowly then, you can spend several minutes before reaching this single "good frame".

A fast application will run at 40fps and catch this "good frame" as soon as it's presented for processing. This offers a nice user experience.

Peformance numbers

Some performance numbers on high-end (Core i7) and low-end (Raspberry Pi 4) devices using 720p (1280x720) images:

	0.0 rate	0.2 rate	0.5 rate	0.7 rate	1.0 rate
Core i7-4790K (Windows 8) (AMD RADEON R9 M290X GPU)	877 millis 114 fps	1975 millis 50.61 fps	3736 millis 26.76 fps	4901 millis 20.40 fps	6526 millis 15.32 fps
Intel® Xeon® E3 1230v5 (Ubuntu 18) (NVIDIA GTX 1070 GPU)	933 millis 107.07 fps	1995 millis 50.11 fps	3660 millis 27.32 fps	4702 millis 21.26 fps	6320 millis 15.82 fps
iPhone7 (iOS 13)	1990 millis 50.23 fps	4325 millis 23.11 fps	7982 millis 12.52 fps	10595 millis 9.43 fps	14201 millis 7.04 fps
Galaxy S10+ (Android 10)	2825 millis 35.39 fps	7575 millis 13.20 fps	12960 millis 7.71 fps	17636 millis 5.67 fps	21069 millis 4.74 fps
Raspberry Pi 4 (Raspbian Buster)	4335 millis 23.06 fps	13555 millis 7.37 fps	27878 millis 3.58 fps	37399 millis 2.67 fps	47797 millis 2.09 fps

Some notes:

• The GPGPU acceleration is done using OpenCL and works on GPUs (Intel, NVIDIA, AMD...)
• Please note that even if Raspberry Pi 4 has a 64-bit CPU Raspbian OS uses a 32-bit kernel which means we're loosing many SIMD optimizations.

Pre-built binaries

If you don't want to build this sample by yourself then, use the pre-built versions:

• Windows: benchmark.exe under binaries/windows/x86_64
• Linux: benchmark under binaries/linux/x86_64. Built on Ubuntu 18.
• Raspberry Pi: benchmark under binaries/raspbian/armv7l
• Android: check android folder
• iOS: check ios folder

On Windows, the easiest way to try this sample is to navigate to binaries/windows/x86_64 and run binaries/windows/x86_64/benchmark.bat . You can edit this file to use your own images and configuration options.

Building

This sample contains a single C++ source file and is easy to build. The documentation about the C++ API is at https://www.doubango.org/SDKs/mrz/docs/cpp-api.html.

Android

Please check android folder for Android samples.

iOS

Please check iOS folder for iOS samples.

Windows

You'll need Visual Studio to build the code. The VS project is at benchmark.vcxproj. Open it.

1. You will need to change the "Command Arguments" like the below image. Default value: --loops 100 --rate 0.2 --positive $(ProjectDir)..\..\..\assets\images\Passport-Australia_1280x720.jpg --negative $(ProjectDir)..\..\..\assets\images\Passport-France_1200x864.jpg --assets $(ProjectDir)..\..\..\assets
2. You will need to change the "Environment" variable like the below image. Default value: PATH=$(VCRedistPaths)%PATH%;$(ProjectDir)..\..\..\binaries\windows\x86_64

You're now ready to build and run the sample.

Generic GCC

Next command is a generic GCC command:

cd ultimateMRZ-SDK/samples/c++/benchmark

g++ main.cxx -O3 -I../../../c++ -L../../../binaries/<yourOS>/<yourArch> -lultimate_mrz-sdk -o benchmark

• You've to change yourOS and yourArch with the correct values. For example, on Linux x86_64 they would be equal to linux and x86_64 respectively.
• If you're cross compiling then, you'll have to change g++ with the correct triplet. For example, on Android ARM64 the triplet would be equal to aarch64-linux-android-g++.

Raspberry Pi (Raspbian OS)

To build the sample for Raspberry Pi you can either do it on the device itself or cross compile it on Windows, Linux or OSX machines. For more information on how to install the toolchain for cross compilation please check here.

cd ultimateMRZ-SDK/samples/c++/benchmark

arm-linux-gnueabihf-g++ main.cxx -O3 -I../../../c++ -L../../../binaries/raspbian/armv7l -lultimate_mrz-sdk -o benchmark

• On Windows: replace arm-linux-gnueabihf-g++ with arm-linux-gnueabihf-g++.exe
• If you're building on the device itself: replace arm-linux-gnueabihf-g++ with g++ to use the default GCC

Testing

After building the application you can test it on your local machine.

Usage

Benchmark is a command line application with the following usage:

benchmark \
      --positive <path-to-image-with-a-plate> \
      --negative <path-to-image-without-a-plate> \
      [--assets <path-to-assets-folder>] \
      [--loops <number-of-times-to-run-the-loop:[1, inf]>] \
      [--rate <positive-rate:[0.0, 1.0]>] \
      [--tokenfile <path-to-license-token-file>] \
      [--tokendata <base64-license-token-data>]

Options surrounded with [] are optional.

• --positive Path to an image (JPEG/PNG/BMP) with a license plate. This image will be used to evaluate the recognizer. You can use default image at ../../../assets/images/Passport-Australia_1280x720.jpg.
• --negative Path to an image (JPEG/PNG/BMP) without a license plate. This image will be used to evaluate the decoder. You can use default image at ../../../assets/images/Passport-France_1200x864.jpg.
• --assets Path to the assets folder containing the configuration files and models. Default value is the current folder.
• --loops Number of times to run the processing pipeline.
• --rate Percentage value within[0.0, 1.0] defining the positive rate. The positive rate defines the percentage of images with MRZ lines.
• --tokenfile Path to the file containing the base64 license token if you have one. If not provided then, the application will act like a trial version. Default: null.
• --tokendata Base64 license token if you have one. If not provided then, the application will act like a trial version. Default: null.

The information about the maximum frame rate (114fps on core i7 and 50fps on iPhone7) is obtained using --rate 0.0 which means evaluating the negative (no MRZ lines) image only. The minimum frame rate could be obtained using --rate 1.0 which means evaluating the positive image only (all images on the video stream have MRZ lines). In real life, you only need a single positive frame to recognize the MRZ data.

Examples

For example, on Raspberry Pi you may call the benchmark application using the following command:

LD_LIBRARY_PATH=../../../binaries/raspbian/armv7l:$LD_LIBRARY_PATH ./benchmark \
    --positive ../../../assets/images/Passport-Australia_1280x720.jpg \
    --negative ../../../assets/images/Passport-France_1200x864.jpg \
    --assets ../../../assets \
    --loops 100 \
    --rate 0.2

On Linux x86_64 you may use the next command:

LD_LIBRARY_PATH=../../../binaries/linux/x86_64:$LD_LIBRARY_PATH ./benchmark \
    --positive ../../../assets/images/Passport-Australia_1280x720.jpg \
    --negative ../../../assets/images/Passport-France_1200x864.jpg \
    --assets ../../../assets \
    --loops 100 \
    --rate 0.2

On Windows x86_64, you may use the next command:

benchmark.exe ^
    --positive ../../../assets/images/Passport-Australia_1280x720.jpg ^
    --negative ../../../assets/images/Passport-France_1200x864.jpg ^
    --assets ../../../assets ^
    --loops 100 ^
    --rate 0.2

Please note that if you're cross compiling the application then you've to make sure to copy the application and both the assets and binaries folders to the target device.