Store seshat:field_specs/0 in persistent terms

[the-mikedavis]

Previously each writer, replica reader and replica stored their field spec in seshat, so the ETS group table in seshat contained many duplicate field spec values. Seshat accepts {persistent_term, term()} for field specs to avoid this duplication. We can update osiris_writer, osiris_replica_reader and osiris_reader to eagerly create those persistent terms with their field specs and avoid the duplication in ETS. This saves a significant amount of memory consumed by the ETS table and also saves time and garbage when querying the ETS table, for example when building the counters map with seshat:counters(osiris).

[the-mikedavis]

Testing this on RabbitMQ main with 2000 streams:

• make start-cluster
• Create 2000 streams: perf-test -sq -qpf 1 -qpt 2000 -qp sq-%d -x 1 -y 0 --time 1
• Measure in a remote_shell, sbin/rabbitmq-diagnostics -n rabbit-1 remote_shell

main:

(rabbit-1@mango2)> tprof:profile(seshat, counters, [osiris], #{type => call_memory}).

****** Process <14366.42911.0>  --  100.00% of total *** 
FUNCTION                                 CALLS    WORDS  PER CALL  [    %]
seshat_counters_server:get_table/1           1        3      3.00  [ 0.00]
ets:first_lookup/1                           1      691    691.00  [ 0.01]
erts_internal:counters_get/2             39994    15845      0.40  [ 0.31]
seshat:build_counters_map/3               5999    17997      3.00  [ 0.36]
seshat:'-counters/1-fun-0-'/2             5999   282386     47.07  [ 5.57]
seshat:'-build_counters_map/3-fun-0-'/3  39994   505929     12.65  [ 9.98]
ets:next_lookup/2                         5999  4244618    707.55  [83.76]
                                                5067469            [100.0]
ok
(rabbit-1@mango2)> {Time, _} = timer:tc(seshat, counters, [osiris]), Time.
19727
(rabbit-1@mango2)> ets:info(seshat_counters_server:get_table(osiris), memory).
4134044

Persistent term field specs:

(rabbit-1@mango2)> tprof:profile(seshat, counters, [osiris], #{type => call_memory}).

****** Process <14366.54497.0>  --  100.00% of total *** 
FUNCTION                                 CALLS    WORDS  PER CALL  [    %]
seshat_counters_server:get_table/1           1        3      3.00  [ 0.00]
ets:first_lookup/1                           1       68     68.00  [ 0.01]
seshat:build_counters_map/3               6000    18000      3.00  [ 1.50]
seshat:'-counters/1-fun-0-'/2             6000   282350     47.06  [23.52]
ets:next_lookup/2                         6000   394121     65.69  [32.83]
seshat:'-build_counters_map/3-fun-0-'/3  40000   506000     12.65  [42.15]
                                                1200542            [100.0]
ok
(rabbit-1@mango2)> {Time, _} = timer:tc(seshat, counters, [osiris]), Time.
12147
(rabbit-1@mango2)> ets:info(seshat_counters_server:get_table(osiris), memory).
268713

[michaelklishin]

So, a ≈ 38.5% percent speedup for seshat:counters/1 with that specific common input.

[michaelklishin]

Note that besides a 38% execution time gain and a 90+% ETS table memory footprint gain (on isolated benchmarks), this practice is already used in Ra for its counters https://github.com/rabbitmq/ra/blob/main/src/ra_counters.erl#L28.

[michaelklishin]

@the-mikedavis thank you!