Error: Buffer is not large enough

[qharlie]

I'm getting this error a lot , here is the full stack trace:

Error: Buffer is not large enough. Expected 1745879154, have 36 new Response (/opt/deploy/formhog/node_modules/mongolian/node_modules/buffalo/lib/mongo.js:255:15),Connection. (/opt/deploy/formhog/node_modules/mongolian/lib/server.js:227:28),Connection.EventEmitter.emit (events.js:88:17),Socket. (/opt/deploy/formhog/node_modules/mongolian/lib/connection.js:61:22),Socket.EventEmitter.emit (events.js:88:17),TCP.onread (net.js:397:14)

[marcello3d]

I have seen this occasionally as a cascade from a program error/exception. Can you reproduce it?

[qharlie]

Ahh thats probably it. Let me try something.

[awwright]

Yeah I have this problem too, though usually only when I haven't properly handled an error somewhere else. I haven't been able to reliably reproduce it, it just sort of shows up among a bazillion other errors when things go wrong.

[awwright]

I've only been able to reproduce it after another error. I'm suspecting there may be a missing var or similar, somewhere. But I can't see anything obviously wrong with the code!

2013-03-14T15:41:51.285Z - Uncaught Exception: Error: Mongo query should have updated 1 documents, but only updated 0
    at lib/transform.HTTPAuto_typeMongoDB_Post.js:178:13
    at mongolian/lib/collection.js:97:13
    at mongolian/lib/util.js:19:13
    at mongolian/lib/db.js:69:25
    at mongolian/lib/util.js:19:13
    at MongolianCursor.next (mongolian/lib/cursor.js:167:9)
    at mongolian/lib/cursor.js:171:18
    at mongolian/lib/util.js:19:13
    at mongolian/lib/cursor.js:125:9
    at mongolian/lib/util.js:19:13
2013-03-14T15:42:02.393Z - Uncaught Exception: Error: Buffer is not large enough. Expected 1159, have 101
    at new Response (buffalo/lib/mongo.js:255:15)
    at Connection.<anonymous> (mongolian/lib/server.js:227:28)
    at Connection.EventEmitter.emit (events.js:94:17)
    at Socket.<anonymous> (mongolian/lib/connection.js:61:22)
    at Socket.EventEmitter.emit (events.js:94:17)
    at Socket.<anonymous> (_stream_readable.js:648:5)
    at Socket.EventEmitter.emit (events.js:91:17)
    at emitReadable (_stream_readable.js:358:10)
    at onread (_stream_readable.js:343:7)
    at ReadableState.onread (_stream_readable.js:64:5)
2013-03-14T15:42:04.999Z - Uncaught Exception: Error: Buffer is not large enough. Expected 1701080942, have 101
    at new Response (buffalo/lib/mongo.js:255:15)
    at Connection.<anonymous> (mongolian/lib/server.js:227:28)
    at Connection.EventEmitter.emit (events.js:94:17)
    at Socket.<anonymous> (mongolian/lib/connection.js:61:22)
    at Socket.EventEmitter.emit (events.js:94:17)
    at Socket.<anonymous> (_stream_readable.js:648:5)
    at Socket.EventEmitter.emit (events.js:91:17)
    at emitReadable (_stream_readable.js:358:10)
    at onread (_stream_readable.js:343:7)
    at ReadableState.onread (_stream_readable.js:64:5)

[awwright]

If only half a message of the stream were to come in (which is perfectly legal), what would happen here?

node-mongolian/lib/connection.js

Lines 59 to 79 in f1a6bdc

	while (writeIndex - readIndex >= expectedMessageSize) {
	if (readMessageLength) {
	thiz.emit('message', buffer.slice(readIndex, readIndex += expectedMessageSize))
	if (readIndex == writeIndex) {
	readIndex = writeIndex = 0
	}
	expectedMessageSize = 4
	readMessageLength = false
	} else {
	expectedMessageSize = (buffer[readIndex]) |
	(buffer[readIndex+1] << 8) |
	(buffer[readIndex+2] << 16) |
	(buffer[readIndex+3] << 24)
	readMessageLength = true
	if (expectedMessageSize > maxMessageSize) {
	thiz.emit('error', 'message too large: ' + expectedMessageSize + ' (max=' + maxMessageSize + ')')
	thiz.close()
	return
	}
	}
	}

It appears that there's no test to make sure we have a complete message before emitting a "have packet" event.

[marcello3d]

The while loop condition should handle that case.

[awwright]

So this probably is the problem then?

I'm currently re-writing the file to avoid Buffer re-allocations, among other things

[marcello3d]

The solution might be to move the emitting on line 61 to the bottom of that block. That way if the message handler throws an exception, the buffer is in a consistent state.

[awwright]

That might be the case for the second and beyond errors ("Expected 1701080942") but the first one seems to simply be an incomplete packet, with no prior error having been thrown.

Oh wait...

[awwright]

Yeah, that's exactly the problem: whenever an Exception is thrown in an emitter hook, it interrupts the stream handling and an entire segment goes unparsed. So, how should errors be handled? Hmmm

[awwright]

Should it really be disconnecting on error by default:

node-mongolian/lib/server.js

Lines 212 to 215 in f1a6bdc

	connection.on('error', function(error) {
	mongolian.log.error(self+": "+require('util').inspect(error))
	if (!connected) callback(error)
	self.close(error)

?

Here's my file that seems to fix: https://gist.github.com/Acubed/5167758
And available in commit awwright/node-mongolian@2b48848

[awwright]

Perhaps the event should be called with process.nextTick? That would isolate any errors from affecting the main loop. I'm not so sure if it should disconnect from the database, but may be that too.