Fixing spatial anomaly for htmcore WIP

[breznak]

So, we have a problem.. hopefully a combination of the params for HTMcore is not optimal.

The Numenta detectors have an "artificial" way of detecting "spatial anomaly" (which is a running upper/lower bounds threshold detector)

If I disable such (fake) detector for our htmcore the scores become pretty bad:

    "htmcore": {
-        "reward_low_FN_rate": 66.1922106936328,
-       "reward_low_FP_rate": 58.7930712907694,
-        "standard": 63.081419488725054
+        "reward_low_FN_rate": 28.85203529943721,
+       "reward_low_FP_rate": 23.591809889328452,
+       "standard": 26.783416390999154

This means most of the work in htmcore with these settings is done in the threshold detector.

Luckily, the numenta detectors & settings do not suffer this drop when spatial-threshold is removed.

TL;DR: we must optimize our net without this "spatial anomaly" threshold-detector.

For #6

[breznak]

CC @ctrl-z-9000-times

[breznak]

One interesting fact,
using synapse competition from htm-community/htm.core#584
leads to better utilization of synapses (probably counters some ill combination of params) which leads to our results getting back to reasonable again.

HTMcore without "spatial anomaly" and with "synapse competition":

--- a/results/final_results.json
+++ b/results/final_results.json
@@ -15,9 +15,9 @@
"standard": 16.43666922426724
},
"htmcore": {
-        "reward_low_FN_rate": 66.1922106936328,
-        "reward_low_FP_rate": 58.7930712907694,
-        "standard": 63.081419488725054
+        "reward_low_FN_rate": 59.48811919632727,
+        "reward_low_FP_rate": 53.995064634304576,
+        "standard": 56.90459258759435
},

EDIT:
Ok, scratch that. I apparently ran NAB on an incorrect branch, the results are still bad w/ syn_competition and w/o spatial anomaly.

[ctrl-z-9000-times]

Yea I don't know what to say here. Numenta did several non-biological things in order score higher on the NAB benchmark.

I think we should disregard both the spatial anomaly and the backtracking TM, since both are non-biological methods which improve scores on this particular benchmark (but probably not all benchmarks).

[breznak]

Numenta did several non-biological things in order score higher on the NAB benchmark. ..backtrackingTM

yep, we're catching up to "HTM" with backTM, even Numenta lists the numentaTM detecot, which uses biological TM and our htmcore detector has results close to that one:
htm-community/htm.core#391

disregard both the spatial anomaly

definitely we should. I was really worried even Numenta's detector suffer the problem (most work done by "spatial anomaly"), fortunately it is not a problem (so the NAB paper is valid!), unfortunately, our TM suffers w/o the spatial anomaly.
For param optimization we must not use the "spatial anomaly". I'll adapt the PR not-to use it on our detector (which is the correct approach). Not sure what to do with Numenta's, probably leave as is?

[breznak]

@ctrl-z-9000-times I'll try to revisit this again, the issue is a weird/severe performance regression. Please help me investigating if you have time, thanks

[breznak]

Update:
When running on only the artificial/synthetic labels, our results are quite good:
python run.py -d htmcore --detect --score --optimize --normalize --windowsFile labels/synthetic.json -n 8

htmcore detector benchmark scores written to /mnt/store/devel/HTM/NAB/results/htmcore/htmcore_reward_low_FN_rate_scores.csv

Running score normalization step
Final score for 'htmcore' detector on 'standard' profile = 84.84
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 84.36
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 88.37
Final scores have been written to /mnt/store/devel/HTM/NAB/results/final_results.json.

[Zbysekz]

I merge it and i got

Running score normalization step
Final score for 'htmcore' detector on 'standard' profile = 66.35
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 57.11
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 71.82
Final scores have been written to /home/ub/HTM/NAB/results/final_results.json.

It is the "good" score?

I can merge it with this branch if you like
EDIT: i will commit it there

[Zbysekz]

I don't have write permissions for this NAB repo.

The branch is there: https://github.com/Zbysekz/NAB/tree/fixing_spatial_anomaly

[Zbysekz]

Now i got even better: (assume it is because of seed==0)

Final score for 'htmcore' detector on 'standard' profile = 70.51
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 60.67
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 75.74

[Zbysekz]

Previous was with spatial anomaly=True,

with false it is:

Final score for 'htmcore' detector on 'standard' profile = 60.91
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 52.14
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 65.60

Not sure if it should be set to true or false in this branch?

[breznak]

OK, i take a look at this. Panda remake is now "finished", so now it uses NetworkAPI only.

thank you 👍
ad NetworkAPI: Does it mean we won't be able to use it with this detector? Since the detector is not in NAPI.

It is the "good" score?

that was slightly below good :), I think I got same-as Numenta scores. (hope with correct settings too)
But I did use the current htmcore master build, not the latest tag from pip (there's the difference in improving changes).

Not sure if it should be set to true or false in this branch?

we want spatial=False here, and everywhere.

[breznak]

@Zbysekz btw, the htmcore detector was not in your PR, so the conflict is still there

[Zbysekz]

ad NetworkAPI: Does it mean we won't be able to use it with this detector? Since the detector is not in NAPI.

Yes, it would be not possible anymore. I coudn't keep Panda in this way. With using NAPI, everything is more general and generated automatically in contrast of manually handing over specific data & instances.

But i can rewrite this nab detector to use NAPI, what i saw it should be quite simple.

[Zbysekz]

btw, the htmcore detector was not in your PR, so the conflict is still there

Hmm there is a bit complication, because i wanted to revert pandaVis changes, but these are not in this branch yet so that is why it was not reflected... i will start new PR and we will just link it from here

[breznak]

But i can rewrite this nab detector to use NAPI, what i saw it should be quite simple.

I was thinking about it...let's keep it for a while as is, and if we stabilize the (top) scores, we can rewrite to NAPI. What I wonder is overhead of the NAPI, for c++->py it's about 30%.

[Zbysekz]

What I wonder is overhead of the NAPI, for c++->py it's about 30%.

What you mean by this overhead of 30% ?
Like performance overhead/complexity or line count?

[breznak]

What you mean by this overhead of 30% ?

compute speed overhead.

[Zbysekz]

What you mean by this overhead of 30% ?

compute speed overhead.

Hmm yes 30% overhead is good, i would expect more.

[Zbysekz]

I am getting

Final score for 'htmcore' detector on 'standard' profile = 60.91
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 52.14
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 65.60

without spatial anomaly and with latest htm.core

[gotham29]

I am getting

Final score for 'htmcore' detector on 'standard' profile = 60.91
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 52.14
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 65.60

without spatial anomaly and with latest htm.core

Hi @Zbysekz!

Could I possibly see your implementation at all?
I ask because I'm getting NAB score around 50 with my htm.core implmenetation (htm_streamer), and itching to figure out the difference.
I know its been a while now but just wondering!

Thanks,

Sam

[gotham29]

Scores with random (seed=0), without any further optimizations

 "htmcore": {
-        "reward_low_FN_rate": 76.5626293570994,
-        "reward_low_FP_rate": 61.359926511549155,
-        "standard": 71.3094612770284
+        "reward_low_FN_rate": 73.96624112193469,
+        "reward_low_FP_rate": 59.253001477255935,
+        "standard": 68.40111896667753

Hey @breznak!

Could I possibly see your implementation that got these scores if you still have it??
I am trying to validate my own htm.core implementation (htm_streamer) on NAB and only scoring around 50 - and dying to get to the bottom of it.
Thank you!

Sam

[Zbysekz]

I am getting

Final score for 'htmcore' detector on 'standard' profile = 60.91
Final score for 'htmcore' detector on 'reward_low_FP_rate' profile = 52.14
Final score for 'htmcore' detector on 'reward_low_FN_rate' profile = 65.60

without spatial anomaly and with latest htm.core

Hi @Zbysekz!

Could I possibly see your implementation at all? I ask because I'm getting NAB score around 50 with my htm.core implmenetation (htm_streamer), and itching to figure out the difference. I know its been a while now but just wondering!

Thanks,

Sam

Hello Sam,
I was just debugging connection with pandaVis, i was not improving the NAB scoring. So my setup was just this branch compiled.

What about this? "good as Numenta results without the fake spatial anomalies" as Breznak writes in one comment. I thnik we discuss this somewhere, but it is already so long ago. I am not i this project active right now.
Trying to remember, but i think there was some mathematic simplification for calculating spatial anomalies.

Good luck

[gotham29]

Hi @Zbysekz!
Thanks for your reply!
Do you mean that you you might've added the spatial anomaly functionality to your htm.core implementation to score > 60?
I see in the NumentaTM detector they declare spatial anomaly for anything more than 5% above the max or below the min.

[Zbysekz]

Hi @Zbysekz! Thanks for your reply! Do you mean that you you might've added the spatial anomaly functionality to your htm.core implementation to score > 60? I see in the NumentaTM detector they declare spatial anomaly for anything more than 5% above the max or below the min.

No i just ran latest htm.core with this branch (that is not merged to NAB master) with spatial anomaly off..
I wish i could help more.

[gotham29]

Hi @Zbysekz! Thanks for your reply! Do you mean that you you might've added the spatial anomaly functionality to your htm.core implementation to score > 60? I see in the NumentaTM detector they declare spatial anomaly for anything more than 5% above the max or below the min.

No i just ran latest htm.core with this branch (that is not merged to NAB master) with spatial anomaly off.. I wish i could help more.

All good man thank you!
I now have 60+ NAB score when using spatial anomaly, which is comparable to NuPIC & htm.java so I'm happy!
With that done I'm definitely curious to look into your visualization functionality too!

[ctrl-z-9000-times]

Hi @Zbysekz and @gotham29,
If you're interested in working on NAB, I will give you write permissions to this repo.

[gotham29]

Hi @ctrl-z-9000-times, yes thank you that'd be awesome!
I'd love to have a look!

[ctrl-z-9000-times]

Ok @gotham29, you should now have write permission for this repo.
Feel free to improve this project in whatever way you see fit.