Performance on large number of small files

[shcheklein]

Hi, I'm one of the maintainers at DVC - github.com/iterative/dvc. A lot of users use Minio as a S3 on-prem alternative and from time to time we see complaints like this:

Hi. I'm looking for the best practices of DVC. I work for hundreds of thousands small files (<400KB) and s3 miniO storage. For the exp I prepared around 40GB of data and push them to the storage. When I tried to pull them on another machine I spent around 4 hours. It was too slow despite the fact that storage was located on nvme SSD. The average speed was about 15-40 Mbits per second (LAN speed is 1Gbit). So for the new experiment I upload the data with borgbackup client and extract them on the same machine that I used for DVC. It was like a miracle! The avg speed was about 400Mbits/sec and I spent about 15 minutes to get all data. So I suppose I did something wrong and I don't know what. Please tell me how to store data with DVC properly using intra network storages.

It doesn't help to change level of parallelism, aws s3 sync is also slow:

aws --endpoint-url http://<ip>:9000 s3 sync s3://myminio .

with any of these settings:

[default]
aws_access_key_id=minio
aws_secret_access_key=miniominio
s3 =
  max_concurrent_requests = 15 # [default]
aws_access_key_id=minio
aws_secret_access_key=miniominio
s3 =
  max_concurrent_requests = 15 up to 150
  max_queue_size = 10000
  multipart_threshold = 64MB
  multipart_chunksize = 16MB
  max_bandwidth = 900MB/s

gives no more than 15MiB/s.

Question: Is it known and expected or it's a client problem? what can be the root cause for this? Is there a known overhead of creating objects?

[tyommik]

Hi. My real example.

MiniO version RELEASE.2021-03-17T02-33-02Z
/etc/default/minio

MINIO_ACCESS_KEY="minio"
MINIO_VOLUMES="/data/s3/"
MINIO_OPTS="-C /etc/minio --address <addr>:9000"
MINIO_SECRET_KEY="miniominio"
MINIO_ROOT_USER="minio"
MINIO_ROOT_PASSWORD="miniominio"

MiniO volume is located on NVME ssd (3Gb/s, INTEL SSDPE2KX040T8)

I have one bucket which I plan to use with DVC. The bucket contains 455,605 files and the size is about 36Gb. ".minio.sys" dir size is 73Mb and contains 455k files also).

LAN is 1Gbit/sec. Both machines are connected to one switch.

I used an awscli for testing I got about 15Mbits per second. I also used warp, here is its output:

warp: Cleanup Done.(base) ashibaev@dl1:~/Downloads$ time warp get --host=addr:9000 --access-key=minio --secret-key=miniominio --autoterm                                                                                                                                                                                                                                                                                                                                               
Throughput 112.1MiB/s within 7.500000% for 10.311s. Assuming stability. Terminating benchmark./ 05m00s 
warp: Benchmark data written to "warp-get-2021-04-03[222121]-dyXY.csv.zst"                                                                                                                                                                                                                                                                                                                                                                                                                    

----------------------------------------
Operation: PUT
* Average: 112.07 MiB/s, 11.21 obj/s

Throughput, split into 218 x 1s:
 * Fastest: 112.8MiB/s, 11.28 obj/s
 * 50% Median: 112.1MiB/s, 11.21 obj/s
 * Slowest: 111.4MiB/s, 11.14 obj/s

----------------------------------------
Operation: GET
* Average: 112.52 MiB/s, 11.25 obj/s

Throughput, split into 40 x 1s:
 * Fastest: 124.8MiB/s, 12.48 obj/s
 * 50% Median: 113.6MiB/s, 11.36 obj/s
 * Slowest: 80.4MiB/s, 8.04 obj/s

[harshavardhana]

LAN is 1Gbit/sec. Both machines are connected to one switch.

your LAN is the bottleneck here 1Gibit/sec = 120MiB/sec max

• Average: 112.52 MiB/s, 11.25 obj/s

GET is maxing out your network.

[tyommik]

I use awscli to download the bucket with small files and got these results (it's vnstat tool). The real speed about 116Mbit/s

 enp7s0  /  traffic statistics

                           rx         |       tx
--------------------------------------+------------------
  bytes                     5,91 GiB  |       77,95 MiB
--------------------------------------+------------------
          max          169,41 Mbit/s  |     1,91 Mbit/s
      average          116,45 Mbit/s  |     1,50 Mbit/s
          min           76,17 Mbit/s  |   856,72 kbit/s
--------------------------------------+------------------
  packets                    4337478  |          524209
--------------------------------------+------------------
          max              14457 p/s  |        2161 p/s
      average               9948 p/s  |        1202 p/s
          min               6552 p/s  |         371 p/s
--------------------------------------+------------------
  time                  7,27 minutes

I will try cp with smb or rclone to compare with s3.

[harshavardhana]

I use awscli to download the bucket with small files and got these results (it's vnstat tool). The real speed about 116Mbit/s

There is not enough concurrency with these tools you should increase concurrency just like warp default is using 10MiB object with concurrency of 20 - your network is the bottleneck here.

Your NVMe is 10x more powerful than your network.

Use

warp get --obj.size 10KiB --host=addr:9000 --access-key=minio --secret-key=miniominio --autoterm --concurrent 128

[tyommik]

There seems everything is OK

$ warp get --obj.size 10KiB --host=10.0.40.123:9000 --access-key=minio --secret-key=miniominio --autoterm --concurrent 128
Throughput 107.7MiB/s within 7.500000% for 10.045s. Assuming stability. Terminating benchmark./ 05m00s                                                                                                                                                                                                                                                                                                                                                                                                                

----------------------------------------
Operation: PUT
Skipping PUT too few samples. Longer benchmark run required for reliable results.

----------------------------------------
Operation: GET
* Average: 105.87 MiB/s, 10841.27 obj/s

Throughput, split into 36 x 1s:
 * Fastest: 107.7MiB/s, 11027.37 obj/s
 * 50% Median: 106.4MiB/s, 10896.14 obj/s
 * Slowest: 97.6MiB/s, 9991.39 obj/s

Thank you very much for your support!

[tyommik]

@harshavardhana is there a way to check performance on my own bucket to remove uncertainty?