Add streaming API to model upload

[brettlangdon]

"If I may presume to make one suggestion going forward, it would be that you consider adding a streaming API for uploading the models themselves. For performance reasons, I think that this is absolutely essential, especially when you are consuming larger models produced by external processes or redirected from disk. Globbing an entire mesh into memory for each upload can quickly consume all of the limited heap available to a node process and cause server crashes. Streams also implement back pressure which optimally handles performance and routing problems for throttling. I half-heartedly tried to get something like this to work in the older SOAP api, but was unsuccessful mostly due to the undue hassle of inserting a stream into the middle of a SOAP request and the general crappiness of the current options for handling SOAP/XML in node. However with the current REST system, I believe that this should be much easier to implement and that it really ought to be exposed as the default method for uploading."

From @mikolalysenko via e-mail.

[bdickason]

+1 for this. I was running into a similar problem when writing my test client.

[mikolalysenko]

Thanks for the mention. From my experience working on the previous client, (and from hacking around in node in general) I think that streaming should be the default/main method for all of the API methods that push big chunks of data. Like:

• addModel
• getModelFile
• addModelFile
• getModelPhoto
• addModelPhoto

Regarding the addModel method, maybe you could do something like this:

var uploadStream = shapewaysClient.addModel({
   filename: "mymodel.stl",
   //User specified data goes here
})
myModelData.pipe(uploadStream)
uploadStream.pipe(toJSON(function(err, result) {
  if(e) {
     console.log("upload failed:", err)
  }
  console.log("upload succeeded, result:", result)
})

Or for a simpler example where you just fire-and-forget the response, here is a sketch of how a 1-line streaming photo upload could work:

myPhotoStream.pipe(shapewaysClient.addModelPhoto(myModelId, { /* optional params */ })

Again, addModelFile could do basically the same thing. Streams would also work for downloads too. For example,

shapewaysClient.getModelFile(myModelId).pipe(fs.createWriteStream("download.stl"))

Anyway, what do you think about this proposal?