Benchmark: Lossless/Lossy Floating Point Compression. TurboPFor vs zfp & blosc

[powturbo]

• 2D/3D datasets:
  +Float Compression dataset : description
  +Datasets for Benchmarking Floating-Point Compressors

• Lossless floating point compression:
  icapp sq1024x1024x4.f32 -R0 -Ff -I15 -J15 -e105,143,80,102 -Ezstd,15
  option -R0 = automatic dimensions determination from file name

• Lossy compression with point wise relative error bound:
  icapp sq1024x1024x4.f32 -R0 -Ff -I15 -J15 -e105 -g.001 -Ezstd,15

• Lossy compression with zfp:
  icapp sq1024x1024x4.f32 -R0 -Ff -I15 -J15 -e142 -z.001
  option -z: for compressors with builin lossy error bound (ex. zfp)

E MB/s     size   ratio   D MB/s  function floating point size=32 bits (lz=zstd,15) unsorted -1
   60    2327663  13.87%     553  105:LztpzD3byte      3D TransposeZ+zstd,15    sq1024x1024x4.f32
  230   16014760  95.46%     237  143:zfp              zfp 3D                   sq1024x1024x4.f32 
   58    2213925  13.20%     585  105:LztpzD3byte      3D TransposeZ+zstd,15    sq1024x1024x4.f32 -g.001
  225   14089880  83.98%     258  142:zfp              zfp 3D                   sq1024x1024x4.f32 -z.001
 
   16   14290026  21.29%     230  105:LztpzD3byte      3D TransposeZ+zstd,15    sq2048x2048x4.f32
  209   68256528 101.71%     209  142:zfp              zfp 3D                   sq2048x2048x4.f32
   18    7277397  10.84%     225  104:LztpxD3byte      3D TransposeX+zstd,15    sq2048x2048x4.f32 -g.001
   14    7550698  11.25%     222  105:LztpzD3byte      3D TransposeZ+zstd,15    sq2048x2048x4.f32 -g.001
  246   47138168  70.24%     293  142:zfp              zfp 3D                   sq2048x2048x4.f32 -z.001 
  
   13     672884  18.08%    2159   80:Lz               zstd,15                  float232630x4.f32
   10    2100247  56.43%    1037  102:LztpzD2byte      2D TransposeZ+zstd,15    float232630x4.f32
  214    3774768 101.42%     151  141:zfp              zfp 2D                   float232630x4.f32
   10     414119  11.13%    2506   80:Lz               zstd,15                  float232630x4.f32 -g.001
    8    1287500  34.59%    1245  102:LztpzD2byte      2D TransposeZ+zstd,15    float232630x4.f32 -g.001
  377    1686848  45.32%     303  141:zfp              zfp 2D                   float232630x4.f32 -z.001
  
   12    5821713  50.90%    1358  102:LztpzD2byte      2D TransposeZ+zstd,15    float953134x3.f32
  124   14742232 128.89%     127  141:zfp              zfp 2D                   float953134x3.f32
   10    4517811  39.50%    1205  102:LztpzD2byte      2D TransposeZ+zstd,15    float953134x3.f32 -g.001
  195    9544168  83.45%     119  140:zfp              zfp                      float953134x3.f32 -z.001
  223   12528088 109.53%     150  141:zfp              zfp 2D                   float953134x3.f32 -z.001

  331    5041816   1.88%    5087  90:lztprle           Transpose +rle+zstd,15   astro_mhd128x512x1024.f32
   90    5855745   2.18%     182  103:LztpD3byte       3D Transpose +zstd,15    astro_mhd128x512x1024.f32
  557   37367816  13.92%     829  141:zfp              zfp 2D                   astro_mhd128x512x1024.f32
  469   81566104  30.39%     598  142:zfp              zfp 3D                   astro_mhd128x512x1024.f32   
  379    2550304   0.95%    5217   90:lztprle          Transpose +rle+zstd,15   astro_mhd128x512x1024.f32 -g.001
   98    3011165   1.12%     154  103:LztpD3byte       3D Transpose +zstd,15    astro_mhd128x512x1024.f32 -g.001
  424   35967416  13.40%     810  141:zfp              zfp 2D                   astro_mhd128x512x1024.f32 -z.001
  320   53603088  19.97%     717  142:zfp              zfp 3D                   astro_mhd128x512x1024.f32 -z.001

   17  137024507  40.84%    2453   86:Lztp4z Nibble    TransposeZ +zstd,15      astro_pt512x256x640.f32
   14  140254161  41.80%     182  105:LztpzD3byte      3D TransposeZ+zstd,15    astro_pt512x256x640.f32
    9   34393724  10.25%     181  105:LztpzD3byte      3D TransposeZ+zstd,15    astro_pt512x256x640.f32 -g.001
   16   40208355  11.98%    2487   86:Lztp4z Nibble    TransposeZ +zstd,15      astro_pt512x256x640.f32 -g.001
  317  148181952  44.16%     405  142:zfp              zfp 3D                   astro_pt512x256x640.f32 -z.001
  211  322757072  96.19%     224  142:zfp              zfp 3D                   astro_pt512x256x640.f32 -z.001

   19   36640070  38.86%     248  102:LztpzD2byte      2D TransposeZ+zstd,15    rsim2048x11509.f32
  284   71839424  76.20%     214  141:zfp              zfp 2D                   rsim2048x11509.f32
   10    8529441   9.05%     243  102:LztpzD2byte      2D TransposeZ+zstd,15    rsim2048x11509.f32 -g.001
  945    8888536   9.43%    1074  141:zfp              zfp 2D                   rsim2048x11509.f32 -z.001

   11  141026747  26.27%     213 105:LztpzD3byte       3D TransposeZ+zstd,15    wave512x512x512.f32
  236  167737848  31.24%     476 142:zfp               zfp 3D                   wave512x512x512.f32
    1   12370337   2.30%     113 105:LztpzD3byte       3D TransposeZ+zstd,22    wave512x512x512.f32 -g.001
  578   13156920   2.45%    1717 142:zfp               zfp 3D                   wave512x512x512.f32 -z.001
   13   28570801   5.32%     224 105:LztpzD3byte       3D TransposeZ+zstd,15    wave512x512x512.f32 -g.001

---

• 1D compression (10 mixed 1D/2D/3D data files):
  +Float Compression dataset : description
  +Datasets for Benchmarking Floating-Point Compressors
  +Scientific IEEE 754 32-Bit Single-Precision Floating-Point Datasets

TurboPFor floating point compression is 31,6% vs 49,3% for blosc2 using byte transpose/shuffle + delta.
Blosc with the new bytedelta compression is 35,7% and still inferior. Additinally TurboPFor decompression ~35% faster.
Using lz4 instead of zstd we have 36,1% witch has similar compression ratio as the new blosc-bytedelta, but is a lot faster than blosc2 with zstd,15.
Lossy compression is only 9,7% with point wise error bound 0.001, Blosc has nothing similar.

icapp *.f32 -I16 -J16 -Ezstd,15 -e83 -b4m
E MB/s     size   ratio   D MB/s  function integer size=32 bits (lz=zstd,15) 
  113    7071465   2.63%    7012  83:Lztpz  Byte   TransposeZ +zstd,15    astro_mhd128x512x1024.f32
   23  146977334  43.80%    3510  83:Lztpz  Byte   TransposeZ +zstd,15    astro_pt512x256x640.f32
   14    1636009  43.95%    1401  83:Lztpz  Byte   TransposeZ +zstd,15    float232630x4.f32
   20    8037995  70.28%    2989  83:Lztpz  Byte   TransposeZ +zstd,15    float953134x3.f32
   70   12834243   9.20%    5102  83:Lztpz  Byte   TransposeZ +zstd,15    msg_sppm.f32
   26    6321188  10.06%    6057  83:Lztpz  Byte   TransposeZ +zstd,15    msg_sweep3d.f32
   30   47691738  50.58%    3379  83:Lztpz  Byte   TransposeZ +zstd,15    rsim2048x11509.f32
   14    3780571  22.53%    2273  83:Lztpz  Byte   TransposeZ +zstd,15    sq1024x1024x4.f32
   15   23721299  35.35%    2280  83:Lztpz  Byte   TransposeZ +zstd,15    sq2048x2048x4.f32
   24  228029425  42.47%    3498  83:Lztpz  Byte   TransposeZ +zstd,15    wave512x512x512.f32
Tot  486,101,266  31,6%

icapp *.f32 -I16 -J16 -Elz4,9 -b4m -e83
   67    8705415   3.24%    8265  83:Lztpz  Byte  TransposeZ +lz4,9       astro_mhd128x512x1024.f32
   42  164889677  49.14%    5321  83:Lztpz  Byte  TransposeZ +lz4,9       astro_pt512x256x640.f32
   28    1908166  51.27%    1856  83:Lztpz  Byte  TransposeZ +lz4,9       float232630x4.f32
   51    8428180  73.69%    2856  83:Lztpz  Byte  TransposeZ +lz4,9       float953134x3.f32
  195   15746064  11.29%    5813  83:Lztpz  Byte  TransposeZ +lz4,9       msg_sppm.f32
   66   19587682  31.16%    6285  83:Lztpz  Byte  TransposeZ +lz4,9       msg_sweep3d.f32
   83   50008850  53.04%    5955  83:Lztpz  Byte  TransposeZ +lz4,9       rsim2048x11509.f32
   23    4651669  27.73%    4283  83:Lztpz  Byte  TransposeZ +lz4,9       sq1024x1024x4.f32
   36   25760593  38.39%    4096  83:Lztpz  Byte  TransposeZ +lz4,9       sq2048x2048x4.f32
   50  254984059  47.49%    5137  83:Lztpz  Byte  TransposeZ +lz4,9       wave512x512x512.f32
Tot  554,670,354  36,10%

icapp *.f32 -I16 -J16 -Ezstd,15 -e147
  266   18371070   6.84%    5689 147:blosc  shuffle delta+zstd,15         astro_mhd128x512x1024.f32
   92  231861071  69.10%    3860 147:blosc  shuffle delta+zstd,15         astro_pt512x256x640.f32
   34    2475736  66.51%     950 147:blosc  shuffle delta+zstd,15         float232630x4.f32
   51    8531654  74.59%    2854 147:blosc  shuffle delta+zstd,15         float953134x3.f32
  295   14362282  10.30%    4440 147:blosc  shuffle delta+zstd,15         msg_sppm.f32
   60   33739276  53.67%    4097 147:blosc  shuffle delta+zstd,15         msg_sweep3d.f32
   81   50291192  53.34%    4443 147:blosc  shuffle delta+zstd,15         rsim2048x11509.f32
   24    3278693  19.54%    1826 147:blosc  shuffle delta+zstd,15         sq1024x1024x4.f32
   71   45029315  67.10%    1418 147:blosc  shuffle delta+zstd,15         sq2048x2048x4.f32
   69  349817736  65.16%    3976 147:blosc  shuffle delta+zstd,15         wave512x512x512.f32
Tot  757,758,024  49,32%

icapp -e148 \x\c\fp\*.f32 -I16 -J16 -Ezstd,15
  390    7221944   2.69%    5262 148:blosc  shuffle+bytedelta+zstd,15     astro_mhd128x512x1024.f32
   35  167134975  49.81%    2515 148:blosc  shuffle+bytedelta+zstd,15     astro_pt512x256x640.f32
   26    1688155  45.36%    1165 148:blosc  shuffle+bytedelta+zstd,15     float232630x4.f32
   54    7720069  67.50%    2424 148:blosc  shuffle+bytedelta+zstd,15     float953134x3.f32
  276   14293184  10.25%    4165 148:blosc  shuffle+bytedelta+zstd,15     msg_sppm.f32
   51   16267728  25.88%    3972 148:blosc  shuffle+bytedelta+zstd,15     msg_sweep3d.f32
   70   48424868  51.36%    2701 148:blosc  shuffle+bytedelta+zstd,15     rsim2048x11509.f32
   26    4018441  23.95%    1964 148:blosc  shuffle+bytedelta+zstd,15     sq1024x1024x4.f32
   25   22167842  33.03%    2178 148:blosc  shuffle+bytedelta+zstd,15     sq2048x2048x4.f32
   38  259533898  48.34%    2859 148:blosc  shuffle+bytedelta+zstd,15     wave512x512x512.f32
Tot    548471103  35.70%

---

Lossy compression with pointwise relative error PWE = 0.001

icapp *.f32 -Ezstd,15 -I0 -J15 -e83 -b4m -Ff -v0 -g.001
  113    4500243   1.68%    6406  83:Lztpz  Byte      TransposeZ +zstd,15  astro_mhd128x512x1024.f32
   18   38733567  11.54%    2645  83:Lztpz  Byte      TransposeZ +zstd,15  astro_pt512x256x640.f32
    9     988371  26.55%    1516  83:Lztpz  Byte      TransposeZ +zstd,15  float232630x4.f32
   17    6663777  58.26%    1853  83:Lztpz  Byte      TransposeZ +zstd,15  float953134x3.f32
   50    7769060   5.57%    4192  83:Lztpz  Byte      TransposeZ +zstd,15  msg_sppm.f32
   17    1693751   2.69%    5780  83:Lztpz  Byte      TransposeZ +zstd,15  msg_sweep3d.f32
   22   20582797  21.83%    2124  83:Lztpz  Byte      TransposeZ +zstd,15  rsim2048x11509.f32
   13    3474495  20.71%    2158  83:Lztpz  Byte      TransposeZ +zstd,15  sq1024x1024x4.f32
   16   15005005  22.36%    2022  83:Lztpz  Byte      TransposeZ +zstd,15  sq2048x2048x4.f32
   16   50267079   9.36%    2777  83:Lztpz  Byte      TransposeZ +zstd,15  wave512x512x512.f32
Total 149,678,144  9,74%