Moped is utilizing 'bad' Connections from the connection Pool

[steve-rodriguez]

I have a 3 node replica set with 1 primary and 2 secondaries. We read from Primary. We are using the 4.0.1 Mongoid gem, 2.0.3 Moped gem and the 2.1.1 connection pool gem. When we directly kill a mongo node and a new Primary is selected, 'broken' connections remain in the pool and when the app grabs one it throws the ConnectionPool::ShuttingDownError. When we gracefully step_down the primary and a new primary is selected, 'old' connections to the 'old' primary still exist in the pool and the app throws a 13435 error of "not master and slaveOK=false". Any time a connection is 'broken' or not valid useable anymore, it should NOT be utilized by Mongoid. Attaching screenprints of the error and the mongodb rest interface status ( for the 'killed' scenario ). Please advise ASAP. Thx!

[dblock]

This looks like a more detailed description from #345. @arthurnn can you please take a look? This seems new in the most recent version of moped and is affecting production for an increasing amount of people.

[dblock]

I also opened #348 which seems possibly related.

[argami]

I'm not sure how much can help, but most of my errors are related and are this:

A Timeout::Error occurred in background at 2015-02-07 01:02:20 UTC :

 Waited 0.5 sec
 path/shared/bundle/ruby/2.0.0/gems/connection_pool-2.0.0/lib/connection_pool/timed_stack.rb:42:in `block (2 levels) in pop'

I've change my pool timeout but seems to still be using 0.5

A Moped::Errors::ConnectionFailure occurred in users#show:

 Could not connect to a primary node for replica set #<Moped::Cluster:100465340 @seeds=[<Moped::Node resolved_address="10.101.1.100:27017">]>
 app/controllers/api/base_api_controller.rb:83:in `authenticate_user_from_token!'

Not sure if is because authenticate_user_from_token is the most used one but seems that the error came mainly from here. Wich is a look for a user in the db and sign_in with devise.

Two more things that can be related with the error is the way i look for the user:

def find_by_username username
    Rails.cache.fetch(username, :expires_in => 30.seconds) do
        self.where(username: /^#{username}$/i).first
    end
end

[argami]

I was able to fixed, In my case i found out that the error it was 2 things.

First the pool_size, and pool_timeout wasn't enough and second, for some reason my postfix queue was sending old messages as new ones, so from my point of view none of my modifications had any effect.

Anyways with a small pool_size and pool_timeout you should have those errors, i think that is the expected behaviour.

[dblock]

@argami What are the values you use?

[klamontagne]

The "not master and slaveOK=false" after a stepdown or restart of the primary was fixed for us by #351 .

[argami]

@dblock this are my values:

      options:
        pool_size: 200
        pool_timeout: 15
        retry_interval: 1
        timeout: 60
        max_retries: 20
        refresh_interval: 30

But beause i had a lot of noise on the mail stuff probably the last value i used (100) probably was good too

Im using Puma 3 workers 0,8 threads

[mateusdelbianco]

This is fixed in #351

[dblock]

Thanks @mateusdelbianco! Looking forward to a moped release soon with these.

[mateusdelbianco]

You should thank @wandenberg for his patch! :)

[steve-rodriguez]

So are BOTH issues fixed? The master/slave issue AND the one where the ConnectionPool gem throws the ConnectionPool::PoolShuttingDownError when you restart a VM where a mongo primary is running? If so when will a new release of Moped be cut? Thanks!

[niedfelj]

Please see my PR #348

[steve-rodriguez]

Still seeing this error when a mongo db node is killed. Moped should evict this bad connection. I am assuming since this issue is still open that this was NOT resolved with 2.0.4. I am using Mongoid 4.0.2 and Moped 2.0.4 and Connection_Pool 2.1.1. Any ideas when this will be addressed? thanks!

2015-02-23T15:06:11-07:00 INFO : [ebdb5491-645e-49b7-9535-07ec2412573f] [ 10.0.32.86] [TBD] [visitor_id: 140555603] Completed 500 Internal Server E rror in 21ms
2015-02-23T15:06:11-07:00 FATAL : [ebdb5491-645e-49b7-9535-07ec2412573f] [10.0.32.86] [TBD] [visitor_id: 140555603]
ConnectionPool::PoolShuttingDownError (ConnectionPool::PoolShuttingDownEr ror
controller: portfolio_summary/portfolios
action: show
params: {"action"=>"show", "controller"=>"portfolio_summary/portfoli os"}

[michaelrobinette]

Still seeing it with Moped 2.0.4 as well

[steve-rodriguez]

To provide more information on this issue, I have a 3 node replica set ( node1, node2 and node3 ). Node1 is the primary. When node2 (secondary node ) is shutdown via the instructions on Mongodbs website for upgrading a replicaset, Mongoid/Moped throws a ConnectionPool::PoolShuttingDownError when it attempts to grab a connection. WHY would it throw this error if this error is generated from a SECONDARY node that should NOT be in the pool anyway???? The mongoid.yml default read option is primary. It should never be creating pool connections to SECONDARY nodes. It appears to me that Mongoid/Moped is throwing SECONDARY node exceptions to the client when the read preference is PRIMARY. And if this error is thrown from logic that is monitoring the replica set, it should NEVER be returned to the client. Mongoid/Moped should always return a connection to the PRIMARY node if the primary node is UP and AVAILABLE. Seems like a fairly MAJOR bug.