[FEATURE] Generate ranged responses with `downloadZip`

[Harsh-br0]

Is your feature request related to a problem? Please describe.
I have found this only client library that generates zip in streaming fashion to use in cloudfare workers.
But the response doesn't seems to be truely streaming because of the lack of partial content responses

Why is your feature relevant to client-zip?
Since cloudfare Workers have browser based architecture ( having whatwg compliant streaming APIs like ReadableStream , TransformStream ) , so i needed a client library that is capable of generating zips in streaming fashion for large outputs

Describe the solution you'd like
At the moment , I'm not sure but seems like this function

client-zip/src/zip.ts

Lines 51 to 116 in a338414

	export async function* loadFiles(files: ForAwaitable<ZipEntryDescription & Metadata>, options: Options) {
	const centralRecord: Uint8Array[] = []
	let offset = 0n
	let fileCount = 0n
	let archiveNeedsZip64 = false
	// write files
	for await (const file of files) {
	const flags = flagNameUTF8(file, options.buffersAreUTF8)
	yield fileHeader(file, flags)
	yield file.encodedName
	if (file.isFile) {
	yield* fileData(file)
	}
	const bigFile = file.uncompressedSize! >= 0xffffffffn
	const bigOffset = offset >= 0xffffffffn
	// @ts-ignore
	const zip64HeaderLength = (bigOffset * 12 | bigFile * 28) as Zip64FieldLength
	yield dataDescriptor(file, bigFile)
	centralRecord.push(centralHeader(file, offset, flags, zip64HeaderLength))
	centralRecord.push(file.encodedName)
	if (zip64HeaderLength) centralRecord.push(zip64ExtraField(file, offset, zip64HeaderLength))
	if (bigFile) offset += 8n // because the data descriptor will have 64-bit sizes
	fileCount++
	offset += BigInt(fileHeaderLength + descriptorLength + file.encodedName.length) + file.uncompressedSize!
	archiveNeedsZip64 ||= bigFile
	}
	// write central repository
	let centralSize = 0n
	for (const record of centralRecord) {
	yield record
	centralSize += BigInt(record.length)
	}
	if (archiveNeedsZip64 || offset >= 0xffffffffn) {
	const endZip64 = makeBuffer(zip64endRecordLength + zip64endLocatorLength)
	// 4.3.14 Zip64 end of central directory record
	endZip64.setUint32(0, zip64endRecordSignature)
	endZip64.setBigUint64(4, BigInt(zip64endRecordLength - 12), true)
	endZip64.setUint32(12, 0x2d03_2d_00) // UNIX app version 4.5 | ZIP version 4.5
	// leave 8 bytes at zero
	endZip64.setBigUint64(24, fileCount, true)
	endZip64.setBigUint64(32, fileCount, true)
	endZip64.setBigUint64(40, centralSize, true)
	endZip64.setBigUint64(48, offset, true)
	// 4.3.15 Zip64 end of central directory locator
	endZip64.setUint32(56, zip64endLocatorSignature)
	// leave 4 bytes at zero
	endZip64.setBigUint64(64, offset + centralSize, true)
	endZip64.setUint32(72, 1, true)
	yield makeUint8Array(endZip64)
	}
	const end = makeBuffer(endLength)
	end.setUint32(0, endSignature)
	// skip 4 useless bytes here
	end.setUint16(8, clampInt16(fileCount), true)
	end.setUint16(10, clampInt16(fileCount), true)
	end.setUint32(12, clampInt32(centralSize), true)
	end.setUint32(16, clampInt32(offset), true)
	// leave comment length = zero (2 bytes)
	yield makeUint8Array(end)
	}

here it can accept byteRange property as 2 element array or start and end properties in options object to yield only those bytes whose index lies inbetween this range.

Describe alternatives you've considered
Currently it is possible to achieve same thing manually with makeZip output stream and wrapping around with own implemented function

Additional Context
Thanks for such library, expecting more features like a modular approach same as node-archiver
So the implementation can be platform , format agnostic

[Touffy]

Hi. Interesting use case, and cool that you're using Edge Workers. I wish my clients would dump their bloated cloud platforms and did the same thing ;) Anyway…

As you might have noticed, there's a predictLength function that can tell exactly how long the archive will be, given input names and sizes. I think we can usually solve your problem efficiently if you can provide those metadata (and if you never want to use compression, because compression would make the length impossible to predict). But it won't work for the end of the file.

So, assuming those two things, this is how it would go :

1. run a modified predictLength until its internal offset is >= the desired range start
   if necessary, backtrack one file (if the previous step did not find an exact solution)
2. keep running predictLength until the offset is >= the desired range end
3. now we now which files to fetch, so run loadFiles with just those files and the initial offset obtained at step 1
4. dump as many bytes as necessary from the beginning and end of the stream (this is where large files make it potentially wasteful)

That works as long as the desired range does not include the central repository (it's the index at the end of the archive). If it does, the problem is that we need the CRC32 of every file (not just the ones that are part of the range) to create the central repository, and normally the only way to obtain that information is to read every byte of those files. In that case, it would be terribly inefficient to serve just that part using the current design. The only reasonable way to solve this for large archives would be for you to provide CRC32s to client-zip (probably as a new property in the metadata items) and then client-zip could create the central repository without reading each file.

So, before we go any further, would you be able to quickly obtain filenames, sizes, and CRC32s whenever a request comes in with a Range header ?

(Or, you know… just cache the whole Response the first time it's requested — ignoring range — and serve the desired range from cache, using the Cache API. Of course, you probably can't do that if there are many different archives or very large ones.)

[Touffy]

expecting more features like a modular approach same as node-archiver
So the implementation can be platform , format agnostic

About that, the answer is no. The point of client-zip is to make ZIP files because — sadly — that's the format with the widest support and the only format necessary to accomplish its intended purpose.

Maybe someday I will write a similar library (or three) to make TAR-GZIP-PGP archives (which, as I've said elsewhere, are a much nicer format than ZIP) and the API would look similar, maybe even the same. But it won't be in the same library.

[Harsh-br0]

So, assuming those two things, this is how it would go :

1. run a modified predictLength until its internal offset is >= the desired range start
   if necessary, backtrack one file (if the previous step did not find an exact solution)
2. keep running predictLength until the offset is >= the desired range end
3. now we now which files to fetch, so run loadFiles with just those files and the initial offset obtained at step 1
4. dump as many bytes as necessary from the beginning and end of the stream (this is where large files make it potentially wasteful)

I see , sounds like heavylifting needs to be done before streaming but the logic is fair to process only required files

So, before we go any further, would you be able to quickly obtain filenames, sizes, and CRC32s whenever a request comes in with a Range header ?

In that case , this checksum can be cached by calculating it before serving, in the storage itself like filename.crc32
Just one time hassle...

Anyways, thanks for insights , your quick response means a lot :)

[Harsh-br0]

About that, the answer is no. The point of client-zip is to make ZIP files

Makes sense, it's in the name itself that clears the purpose

[Touffy]

You can also compute the CRC32 when a file is uploaded, instead of when it is accessed. Since the upload involves streaming all the data from network to storage, just passing the stream through CRC32 shouldn't add too much overhead.

So the next step should be to add a new mode to client-zip, where the metadata option is mandatory and has to also provide CRC32s for all the files. In this mode, client-zip will use the provided metadata instead of computing the size and CRC32 itself.

Once we've done that, we can implement the partial behaviour on top.

I'm not sure if all this should be included in the default client-zip. It will probably add a few kilobytes of code, all of it completely useless in the browser, and it relies on exact length prediction which will be incompatible with compression and maybe encryption too. So maybe a new major branch with its own NPM tag ? e.g. client-zip@partial where all the code for computing sizes and CRC32s is removed.

[Harsh-br0]

You can also compute the CRC32 when a file is uploaded, instead of when it is accessed. Since the upload involves streaming all the data from network to storage, just passing the stream through CRC32 shouldn't add too much overhead.

Yeh , but seems like the tool i use for uploading doesn't supports calculating hashsums on the go, instead an inefficient way to calculate them after uploading exists.
Probably, need to implement it with own uploading logic

I'm not sure if all this should be included in the default client-zip. It will probably add a few kilobytes of code, all of it completely useless in the browser, and it relies on exact length prediction which will be incompatible with compression and maybe encryption too. So maybe a new major branch with its own NPM tag ? e.g. client-zip@partial where all the code for computing sizes and CRC32s is removed.

Yeh , a seperate branch is good idea to integrate this