2gb upload limit fix #352
2gb upload limit fix #352
Conversation
demiankatz
left a comment
demiankatz
left a comment
There was a problem hiding this comment.
Thanks, @Jason-Benson, great work -- this looks like a solid solution to the problem. See below for the usual nitpicks and suggestions, many of which are just about possibly removing log messages that are no longer needed. :-)
demiankatz
left a comment
demiankatz
left a comment
There was a problem hiding this comment.
Thanks for the further progress, @Jason-Benson -- looking even better now. I had one more minor idea while reviewing, and it looks like there are still a couple of comments pending from the last review, but I've gone through and resolved all the threads that are finished to clarify what remains.
demiankatz
left a comment
demiankatz
left a comment
There was a problem hiding this comment.
See below for a couple of optional suggestions. We're definitely nearly done!
Co-authored-by: Demian Katz <demian.katz@villanova.edu>
demiankatz
left a comment
demiankatz
left a comment
There was a problem hiding this comment.
Thanks, @Jason-Benson, this looks great!
See below for an incredibly trivial suggestion.
Beyond that, one other small request: can you add a line to the "Fixed" section of the changelog that says something like "Eliminated 2GB file limit by adding streaming support."
...and then, unless you're aware of outstanding problems, I think this can be merged!
demiankatz
left a comment
demiankatz
left a comment
There was a problem hiding this comment.
Thanks, @Jason-Benson -- merging now!
2 participants
To get around node's 2gb limit on handling, I reworked the ingest and metadata handling to stream the files in chunks.