Message gets redelivered in loop during the time of high CPU processing on the client side #123
Description
This is the flow of the issue
- A queue message which requires high amount of CPU processing reaches the client.
- Client is not able to respond to the heartbeat from the broker.
- Broker redelivers the same message and also gives "unexpected ACK" error as the ack Id changes on redelivery of same message and previous ack id becomes invalid
- This seems to happen again and again for a while
- I receive connect time out error around - 250000 times after the first message acknowledges
- Consumer count shows up as 0 in broker console, for all queues in the nodeJs application, not only the one which the messages were being sent to
- Consumers are not able to reconnect back, broker always shows consumers as 0
When I disable both heartbeat everything seems to work fine
When I disable the broker to client heartbeat, message occasionally redelivers but the consumers still go to 0 in the broker console
I can't disable the heartbeat to make it work as it has it's own advantages
In all of these cases I am receiving high amounts of connect time out error.
During low or average CPU loads on the nodejs server, everthing seems to work fine. Issue is only when the load is high
This is the code for connection
const stompit = require('stompit')
const keys = require('./keys')
const logger = require('../utils/logger.util')
const reconnectOptions = {
useExponentialBackOff: true
}
let ssl = !!parseInt(keys.activeMQ.ssl)
const serverPrimary = {
host: keys.activeMQ.host,
port: keys.activeMQ.port,
ssl: ssl,
connectHeaders: {
host: '/',
login: keys.activeMQ.username,
passcode: keys.activeMQ.password,
'heart-beat': '5000,5000',
},
}
const serverFailover = {
host: keys.activeMQ.host1,
port: keys.activeMQ.port,
ssl: ssl,
connectHeaders: {
host: '/',
login: keys.activeMQ.username,
passcode: keys.activeMQ.password,
'heart-beat': '5000,5000',
},
}
connManager = new stompit.ConnectFailover(
[serverPrimary, serverFailover],
reconnectOptions,
)
connManager.on('error', function (e) {
const connectArgs = e.connectArgs
const address = connectArgs.host + ':' + connectArgs.port
logger.error({ error: e, customMessage: address })
})
channelPool = new stompit.ChannelPool(connManager)
module.exports = channelPool
@gdaws Could you please help me out on this. I have been stuck on this for weeks
Main issues being
- Redelivery in loop of the same message on loss of heartbeat from the application during high load
- Consumers becoming 0 in broker console. Hence new messages piling up on the broker
- Consumer Connection doesn't reestablish after broker console becomes 0.