Skip to content

Introduce API to retrieve vshard-topology for storage and router for etcd.cluster.vshard #30

@ochaton

Description

@ochaton

We have several installations with moonlibs/config as Configuration Driver for multiple vshard-clusters. All of them have to copy-paste function build_sharding which retrieves entire configuration under etcd/prefix and constructs vshard acceptable configuration.

After introducing master_selection_policy = etcd.cluster.vshard and having config.etcd.get_all actually we are able to standardize the way every installation should configure it's vshard cluster.

I propose to specify configuration of vshard like that:

common:
  vshard: // main vshard section shared across all instances
    bucket_count: XXX // 
    collect_bucket_garbage_interval: ...
    collect_lua_garbage: ...
    sync_timeout: ...
    rebalancer_disbalance_threshold: ...
    rebalancer_max_sending: ...
    discovery_mode: ...
    consts: // the way to override vshard consts. The moonlibs/config should not override somehow vshard.consts by himself.
        BUCKET_CHUNK_SIZE: 3000
    weights: // the way to specify distances between zones (the higher the longer)
      region-1:
        region-2: 100 // distances between zones, (default is none)
clusters:
  shard_001:
    vshard:
      weight: 1 // weight of replicaset in terms of rebalancing/resharding (default none)
instances:
  instance_001_01:
    vshard:
      zone: "region-1" // replica weight in terms of router API (not the same as replicaset weight)

moonlibs/config should not call vshard.storage.cfg or vshard.router.cfg. It should provide API as config.vshard_get("storage") and config.vshard_get("router") to fetch constructed topology for storage or router instead.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions