Tests to prove bulk avoids EMFILE errors

[adam-lynch]

Appends multiple entries from passed array of src-dest mappings. A lazystream wrapper is used to prevent issues with open file limits.

Would it be worth it to have tests on this? Or maybe you're ok with the lazystream wrapper (not sure what it is) presumably having its own tests for that?

Spawned from nwutils/nw-builder#118

[soyuka]

I'm using bulk with a lot of data with no issues.

[ctalkington]

while the description of file should most likely be updated, it also uses lazystream wrapper. i haven't heard of too many EMFILE errors since implementing lazystream. what version of archiver would this be? theres been a bit of churn with the async and stat queue changes in last 2 releases.

[adam-lynch]

I'm not really saying anything is broke. Tests would just be good for people considering using this and also would help catch any PRs / commits that introduce a change that breaks this.

[adam-lynch]

Although, for graceful-ncp (adam-lynch/graceful-ncp#1). I have tried to write tests that prove an EMFILE would be thrown in one case and not another, but completely failed 😦.

[soyuka]

@adam-lynch I've made a small E2E test based on the graceful-fs test.

This will throw EMFiles:

var fs = require('fs')
var n = 1024, fds = [];
for(var i = 0; i < n; i++) {
  go()
}

function go() {
  fs.open(__filename, 'r', function(er, fd) {
    if(er) throw er
    fds.push(fd) 
  })
}

process.on('uncaughtException', function(err) {
  console.error(err) 

  for(var i in fds) {
    fs.close(fds[i], function(er) { if(er) throw er })
  }
  process.exit()
})

So, if we open about 1024 times the __filename (the current file) EMFILES is happening.

Let's make the same test by archiving 1024 times __filename with bulk :

var fs = require('fs')
var archiver = require('./lib/archiver.js')
var archive = archiver('zip')

var n = 1024, fds = [];
for(var i = 0; i < n; i++) {
  fds.push({src: __filename, data: {name: 'test'}})
}

archive.bulk(fds)

archive.on('error', function(er) {
  throw er
})

archive.pipe(fs.createWriteStream('./test.zip'))

process.on('uncaughtException', function(err) {
  console.error(err) 

  process.exit()
})

It's working with no EMFIles. QED.

Note that I made this really fast it may not worth anything but I think this is a good start for a proof of concept.

[adam-lynch]

@soyuka great! Now it would be great to be able to run both of these and "pass" (exit with 0) if the first throws an error and the second one doesn't. It might be easier with two scripts instead of two function calls and trying to catch with Mocha like I was trying for graceful-ncp. Then it could be ran by Travis 😄.

[soyuka]

I don't think it's really needed nor usefull. It's the whole point of Lazystream!

[adam-lynch]

@soyuka yeah, I wasn't sure if there was a need.

[soyuka]

@ctalkington you can close this I think ;)