router: make full map_callrw with split args

[mrForza]

This patch introduces a new way of map_callrw execution by which we can pass some arguments to all storages and split buckets' arguments to those storages that have at least one bucket of bucket_ids. To achieve this we introduce a new string option - mode to map_callrw api.

@TarantoolBot document

Title: vshard: mode option for router.map_callrw()

This string option regulates on which storages the user function will be executed via map_callrw. Possible values:

1. full - the user function will be executed on all storages in
   cluster.
2. partial - the user function will be executed on certain sotrages on
   which buckets from bucket_ids were found.

After that changes map_callrw works in 4 different scenarios depending on mode and bucket_ids options:

1. <mode = 'full', bucket_ids = nil> - user function will be executed
   with args on all storages.
2. <mode = 'full', bucket_ids = {[bid_1] = {b_arg_1}, ...}> - storages
   that have at least one bucket of bucket_ids will execute user
   function with args and additional buckets' arguments. Other storages
   will execute user function only with args.
3. <mode = 'partial', bucket_ids = {bid_1, ...}> - user function will be
   executed with args on storages that have at least one bucket of
   bucket_ids.
4. <mode = 'partial', bucket_id = {[bid_1] = {b_arg_1}, ...}> - the same
   as the 3rd scenario but buckets' arguments (b_arg_1, ..., b_arg_n) will
   be added as additional arguments to user function.

Also now map_callrw ends with error in cases of <mode = 'full', bucket_ids = {1, 2, ...}> and <mode = 'partial', bucket_ids = nil>.

Closes #559

[Gerold103]

Thanks 🔥! I sure hope it will help the customers 😁.

[Gerold103]

The commit message doesn't mention the ticket. You need to write Closes #559 right before the TarantoolBot doc request.

[Serpentian]

Thank you for the patch! It looks great, but I'm afraid, it's a little bit broken now, let's fix and test the new mode with broken cache on the router

[Serpentian]

Thank you for the patchset. Very good! Seems like the last comments from me

[Serpentian]

Thank you a lot! Clean work, IMHO. Hope, it'll help TDG make requests safer

[Gerold103]

I don't understand. Why do you need "refed replicasets"? The stages are Ref + Map.

Ref works in 2 ways:

• Ref all storages regardless of buckets. This was done previously by router_ref_storage_all() without any replicasets referenced in advance.
• Or ref specific storages which have at least one bucket. For that we had router_ref_storage_by_buckets(). It was already doing all the checks to make sure that the referenced storage has the needed buckets.

This patch was only supposed to add additional arguments to these already existing ways of execution. Not change the way of referencing. It was previously "ref all or ref by buckets" and remains like this.

What changed here? What am I missing?

[mrForza]

Why do you need "refed replicasets"?

We need refed_replicasets in router_ref_storage_all in order to check that storages which were referenced by router_ref_storage_by_buckets still have refs (in other words they are not expired). Also if we were just use router_ref_storage_by_buckets and then router_ref_storage_all without refed_replicasets, we would have to complicate the logic in this functions to avoid doubled refs for previously refed storages. See Nikita's comment related to this problem.

What changed here? What am I missing?

To show what has changed, I'll draw a table of map_callrw scenarios:

mode	bucket_ids	ref-functions which we use
partial	{bid_1, ..., bid_n}	router_ref_storage_by_buckets
partial	{bid_1 = {bargs_1}, ..., bid_n = {bargs_n}}	router_ref_storage_by_buckets
full	nil	router_ref_storage_all
full	{bid_1 = {bargs_1}, ..., bid_n = {bargs_n}}	router_ref_storage_by_buckets + router_ref_storage_all

Our main goal is to implement full map_callrw with split args with minimal cost. We don't need to introduce new ref-functions or new ways of referencing for it. We only want to minimally modify existing ref-functions and combine it in some way. If we want to have full map_callrw with split args we need 2 important things:

1. ref all storages
2. build a table of bucket arguments

router_ref_storage_by_buckets + router_ref_storage_all in minimally modified way can help us to achieve these 2 things. Also as you can see, for the 1-3 map_callrw scenarios we continue to use the old ref-functions. In my opinion this patch implements full map_callrw with split args without any serious complication of map_callrw business logic.

[Gerold103]

So now in the full mode with buckets we call router_ref_storage_by_buckets() and then router_ref_storage_all()? Why not call just router_ref_storage_all()? The effect would be the same - all storages receiving a ref.

[mrForza]

No, in full map_callrw with split args we must invoke router_ref_storage_by_buckets, because without using this function the router_group_map_callrw_args will build incorrect table of form {rs_id = {b_id = {b_args}}} as the router's cache may be broken. router_ref_storage_by_buckets finds the buckets in the cluster, updates the router's cache - router.route_map and refs storages which relate to given buckets ids whereas the router_ref_storage_all doesn't do that. See Nikita's comment related to this problem.

[Gerold103]

The possibility of the router's cache being broken is quite low. We just introduced one more network hop just for that.

Would it work to perhaps pass grouped bucket IDs to replicasets as arguments of router_ref_storage_all()? Like make this one call ref_storage_with buckets for storages where we expect specific buckets, and just "ref" for storages where we don't expect any specific buckets.

At least then we wouldn't have to do 2 network hops to each replicaset just to build/check the refs. Building refs must be possible with a single hop.