[HelpWanted] Download progress

[jawainc]

Hello,
Thanks for the great library.

I am facing problem for displaying the download progress

here's my code:

   const files = response.data.files

    totalBytes = getTotalBytes(files)

    const requests = files.map((file) => fetch(file.url)
      .then(async (response) => {
        const reader = response.body.getReader()
        const chunks = []
        while (true) {
          const { done, value } = await reader.read();
          if (done) {
            break;
          }
          chunks.push(value)
          receivedBytes += value.length;
          progress(receivedBytes)
        }
        return new Blob(chunks) //--> dont know if this is the correct way
      })
      .catch(error => {
        throw new Error(error.message + "\nURL: " + file.url)
      })
    )
    const responses = await Promise.all(requests)
    
    console.log("response: ", responses) //--> this works and output like:  (11) [Blob, Blob, Blob, Blob, Blob, Blob, Blob, Blob, Blob, Blob, Blob]

    const blob = await downloadZip(responses); //--> not working

    const link = document.createElement("a")
    link.href = URL.createObjectURL(blob)
    link.download = fileName + ".zip"
    link.click()
    URL.revokeObjectURL(link.href)
    link.remove()

I've tried solution mentioned here: #19
but it does not calculate when files are downloaded but rather I think when making zip file

Any help on this matter Please

[Touffy]

When using an intermediate Blob, the process will be divided in two steps :

1. Fetching the input files and creating the Zip file into a Blob. This depends on CPU and network.
2. "downloading" the result from that Blob. This is just moving memory to disk.

To be clear, which one do you want to measure ?

If you were not using a Blob, but streaming through a Service Worker, the two steps would be concurrent and move roughly at the same pace, so the question wouldn't really matter.

[jawainc]

Thankyou for your quick response

1st option, fetching the input file and creating zip.

I got it working with following:

const files = response.data.files

    totalBytes = getTotalBytes(files)

    const requests = files.map((file) => fetch(file.url)
      .then(async (response) => {
        const reader = response.body.getReader()
        const chunks = []
        while (true) {
          const { done, value } = await reader.read();
          if (done) {
            break;
          }
          chunks.push(value)
          receivedBytes += value.length;
          progress()
        }
        return {name: file.name, input: new Blob(chunks)} //--> changed from just returning blob
      })
      .catch(error => {
        throw new Error(error.message + "\nURL: " + file.url)
      })
    )
    const responses = await Promise.all(requests)

    const blob = await downloadZip(responses ).blob();

    const link = document.createElement("a")
    link.href = URL.createObjectURL(blob)
    link.download = fileName + ".zip"
    link.click()
    URL.revokeObjectURL(link.href)
    link.remove()

[Touffy]

Yeah, I guess that'll do it. Although it is rather inefficient. You're starting all the downloads at once, and waiting until they're all fully buffered, before downloadZip can even begin to work.

I suggest using my DownloadStream instead of Promise.all and then an async generator to make those blobs as needed from the stream. Something like this :

const blob = await downloadZip(blobAndCountBytes(new DownloadStream(files))).blob()

async function *blobAndCountBytes(downloadStream) {
  for await (const response of downloadStream) {
    // do your thing with the Reader, increasing receivedBytes and making a blob
    // and get the filename (for example by looking up the response.url in a Map)
    yield {name: filename, input: new Blob(chunks)}
  }
}

Even better (and shorter to write), you could use a TransformStream on each Response.body instead of reading the whole Response into a Blob. The transform function for the TransformStream would just increment receivedBytes and enqueue the input chunk. You can use that stream as input for downloadZip instead of a blob, which allows downloadZip to start working as soon as some data is available instead of the whole file.

const blob = await downloadZip(countBytes(new DownloadStream(files))).blob()

async function * countBytes(downloadStream) {
  for await (const response of downloadStream) {
    const stream = response.body.pipeThrough(new TransformStream({
      transform(chunk, ctrl) {
        receivedBytes += chunk.length
        ctrl.enqueue(chunk)
      }
    }))
    // get the filename (for example by looking up the response.url in a Map)
    yield {name: filename, input: stream}
  }
}

[ahamelers]

If you were not using a Blob, but streaming through a Service Worker, the two steps would be concurrent and move roughly at the same pace, so the question wouldn't really matter.

Hi @Touffy, how would you track progress (or even just trigger stuff on completion) from the client page using a service worker with client-zip and dl-stream?

[Touffy]

The client page isn't using JavaScript to download, so you have to do the tracking in the Service Worker (using any of the methods already discussed) and then feed the tracking information to the client. You could send progress events with postMessage.

Or maybe create a dedicated MessageChannel, transfer the receiving port over an initial postMessage to the client (fired when you begin creating the archive), and then post all further updates on that channel. That way, the client's global onmessage doesn't need to handle the complexity of following multiple streams (should that happen…) because you have a separate channel for each.

[Touffy]

So many people seem interested in tracking the downloads that I am considering adding a hook in client-zip just for that — it would be easier than what you can do from the outside — but I'm afraid it would hurt performance for people who aren't tracking client-zip's progress. Some tests are needed…

[ahamelers]

I really just need a way to note that long downloads are complete (to clear firefox keep alive requests among other things). I love the serviceworker implementation that begins the download of the zip while the download of individual files is ongoing, and can't seem to find a way to mark the progress/completion of the download that doesn't interrupt this

[Touffy]

I'm thinking of a new option for downloadZip where you give it a writable MessagePort, and it will post a messages to it every time a file is fully processed, when chunks from a large file are consumed, and a final message at the end. In your case, you can just listen to that final message. It should be easy to implement, but then, like I said, I need to measure how much it will cost when the option is not used (and how much when it is used, too).

[Touffy]

I decided to post tracking messages after each file is completely processed, but not each chunk of larger files. That could be added later, probably with some throttling. Indeed, when using the new feature with my set of 12k very small files, creating the archive took nearly 10% longer. Keep in mind that's an extreme case.

Even better news : performance is unaffected by the new code as long as you don't actually use the optional MessagePort.

[ahamelers]

That's great! In my implementation in the meantime, I made worker scripts for predictLength and makeZip, and using .tee() with makeZip and recreating the downloadZip response, I managed to note when the download was complete while still downloading the zip while it was created with dl-stream. However, when it came time to act on the message in the client, I realized the form submission had refreshed the page and console, removing or uninitializing the message port I'd created there 🤦 . So when you do release the tracking messages, I'm not sure what I'll be able to do with them!