Use routing fees z-score to remove outliers #6
Description
Description
Channels having a fee rate PPM higher than 10,000 or a base fee higher than 100 satoshis are discarded to remove outliers from the data set.
This is to avoid having too high values that would mess up the scoring system.
A channel's score is determined relative to the maximum and minimum values found in all the channels from the network. This is the formula:
score = (value-lowest * (1 / highest-lowest)) * weightHaving a maximum value too high messes up the final score as the differences on low values becomes smaller.
For example, consider the following scenarios where we have these values
chan_a_value = 50
chan_b_value = 100
min_value = 1
Higher maximum value scenario
max_value = 100000
chan_a_score = 0.00049
chan_b_score = 0.00098
Lower maximum value scenario
max_value = 2000
chan_a_score = 0.024
chan_b_score = 0.049
The impact in the overall score in the second scenario is three orders of magnitude lower.
An easy solution would be to lower the 10,000 value to something closer to 2,000, but if on-chain fees increase it is likely that lightning fees will follow, so the limit would become outdated and leave out channels that shouldn't be discarded.
A better solution is to calculate the set of values standard deviation and remove outliers that have a z-score value of X (to be defined).