Puzzled with uploading a big zip file in chunks to back-end

[alexandis]

Hi guys.

In my Angular app I have a dropzone component where I upload a big amount of files to client, which later need to be uploaded to server.
Since the size and amount of files varies, I've decided to create a single zip archive and then upload it by fixed-size chunk.

I have a working solution using JsZip + RxJs operators (I upload chunks sequentially (to avoid mess), then merge them + unzip the resulted file on server). However I am not satisfied with the performance: for 2.2GB file, the zipping is 9 min, uploading is 5.

Now I've decided to use fflate, but I am puzzled and overwhelmed with it - too much of everything here... Could you please advice me how to accomplish my goal in the best possible way (memory usage + speed)? Not a ready code :), just schematically. Because I don't understand the workflow in my case (like - do I need to fill a whole zip object with files or I can start streaming the first chunk to the server once the zip file is filled up with [chunkSizeInBytes] bytes of data, etc.)

Thank you very much.

[alexandis]

I ended up with the following code. But it did not bring me the speeding up compared to JsZip:

      const chunks: Uint8Array[] = [];
    
      const flateStreamHandler = (error: FlateError, chunk: Uint8Array, final: boolean) => {
        if (error) {
          observer.error(error);
        }
        else if (final) {
          //Upload
          //Sequentially upload each chunk
          concat(...chunks.map((chunk, index) => uploadChunk(chunk, measureEquipmentCode, index, chunks.length - 1 === index))).subscribe(observer);
        }
        else {
          if (chunks.length === 0 || (chunks[chunks.length - 1].length + chunk.byteLength) > AbxMeasureProtocols.maxChunkBytes) {
            // If there are no chunks yet or adding the current chunk would exceed the maximum chunk size,
            // create a new chunk and add it to the chunks array
            chunks.push(new Uint8Array(chunk));
          }
          else {
            // Otherwise, append the current chunk to the last chunk in the chunks array
            const lastChunk = chunks[chunks.length - 1];
            const concatenatedChunk = new Uint8Array(lastChunk.length + chunk.byteLength);
            concatenatedChunk.set(lastChunk, 0);
            concatenatedChunk.set(chunk, lastChunk.length);
            chunks[chunks.length - 1] = concatenatedChunk;
          }
          zippingProgressCallback(chunks[chunks.length - 1].byteLength, chunks.length);
        }
      };

      this.zip = new Zip(flateStreamHandler);

      const processFilesSequentially = (files: File[]): Promise<void> => {
        let promise = Promise.resolve();
        files.forEach(file => promise = promise.then(() => processFile(file)));
        return promise;
      }

      const processFile = (file: File): Promise<void> => {
        const zipFile = new ZipDeflate(file.name.replace(/^\/+/, ''), { level: 0 });
        this.zip.add(zipFile);
        return new Promise<void>((resolve, reject) => {
          file.arrayBuffer()
            .then(fileData => {
              zipFile.push(new Uint8Array(fileData), true);
              resolve();
            })
            .catch(error => {
              reject(error);
            });
        });
      }
    
      processFilesSequentially(files).then(() => this.zip.end());

Besides, when unzipping at back-end, I've got the exception "End of Central Directory record could not be found" (I used standard System.IO.Compression.ZipFile class in .NET - obviously some format incompatibilities).

[101arrowz]

I'll send a hopefully faster working solution tonight.

[101arrowz]

You should probably make use of the WHATWG streams API to maximize performance here. If fetch supported ReadableStream in all browsers you could do the entire thing with WHATWG streams, but as it does not, you can do it as follows.

First you need a way to read the file into a zip entry. Since you're using level 0 compression, I've used a ZipPassThrough instead of ZipDeflate (it's a bit faster).

import { Zip, ZipPassThrough } from 'fflate';

const addFileToZip = async (zip: Zip, file: File) => {
  const zipEntry = new ZipPassThrough(file.name.replace(/^\/+/, ''));
  zip.add(zipEntry);
  const reader = file.stream().getReader();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    zipEntry.push(value);
  }

  // finish file
  zipEntry.push(new Uint8Array(0), true);
}

Now you create a Zip file that uploads the chunks as it receives them, instead of waiting till the end. This will save you quite a bit of time while uploading, and likely some memory too.

let prevUpload = Promise.resolve();

const zip = new Zip((err, chunk, final) => {
  if (err) {
    // handle as you see fit
  }
  // make an uploadChunkPromise that returns a promise instead of an RxJS observable
  prevUpload = prevUpload.then(() => uploadChunkPromise(chunk, ...));
  if (final) {
    promiseToObservable(prevUpload).subscribe(observer);
  }
});

Now just add your files and you're done.

// assuming you have a files array like this:
// var files: File[];

for (const file of files) {
  await addFileToZip(zip, file);
}
zip.end();

Full code

import { Zip, ZipPassThrough } from 'fflate';

const addFileToZip = async (zip: Zip, file: File) => {
  const zipEntry = new ZipPassThrough(file.name.replace(/^\/+/, ''));
  zip.add(zipEntry);
  const reader = file.stream().getReader();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    zipEntry.push(value);
  }

  // finish file
  zipEntry.push(new Uint8Array(0), true);
}

let prevUpload = Promise.resolve();

const zip = new Zip((err, chunk, final) => {
  if (err) {
    // handle as you see fit
  }
  // make an uploadChunkPromise that returns a promise instead of an RxJS observable
  prevUpload = prevUpload.then(() => uploadChunkPromise(chunk, ...));
  if (final) {
    promiseToObservable(prevUpload).subscribe(observer);
  }
});

for (const file of files) {
  await addFileToZip(zip, file);
}
zip.end();

The end-of-central-directory thing doesn't look like it should be an issue with your current code, but it definitely won't be with this. Let me know if you have any other questions!

[alexandis]

Hi.
Thank you very much for your time and efforts. Unfortunately, the project is under time pressure, so I cannot try this approach now and it was agreed to proceed with the "slow" solution we have now (based on JsZip). However, I think this discussion might be useful for the community. I myself will give it a try and compare the timings when I get a chance.