proposal for additional data source -- full real-time BGP table

[joe-bp]

Hi,

Not sure if this is the right place to propose this, but I'd like to propose an additional source of data for asmap -- the full internet routing table.

This can be done by running FRR or BIRD in a container or a VM (cloud, etc), getting an eBGP peer with a provider that's willing to provide the full table, and then extracting the data into a format that can be used with asmap. Multiple people/orgs can run these, and the table can be received from multiple networks. The BGP peer for the table doesn't have to be part of any data plane, it can be just for sending the table, no routes accepted, etc. It can also be remote (not ttl 1).

Would this be useful to this project and bitcoin core?

I can do this work and provide code and steps, just want to see if it's worthwhile first.

[jurraca]

Hey @joe-bp thanks for the idea. If I understand correctly, this BGP peer would be limited to its provider's announcements, even if they are "global". We use a few sources which do a lot of this work for us, namely get route information from many BGP peers and collate it into one data source. We use multiple sources (and want to add more) to be able to compare them against each other and not rely/trust a single provider's view of the routing table.
That being said, more data could be good if we take the time to validate that it gives us something RPKI/IRR/Routeviews doesn't, and why. We can definitely do more here so thanks for the input!

[joe-bp]

Hi @jurraca thanks for the reply.

This sounded like a yes to the usefulness question, so I started the work.

Couple of points I'd like to make:

1. In my opinion, the RIPE RIS data is the most useful (for reasons below), however using it relies on a centralized data source that could change access or even manipulate data at will. Using direct BGP routing table data from BGP itself is a means of obtaining that same kind of data but without having to rely on a single upstream organization (particularly not one under a hostile jurisdiction)
2. IRR and RADB records are often stale, incomplete, sometimes completely incorrect, and occasionally even malicious. Also, many times these databases may say one thing, but what ends up being used by routing policies on carrier and edge networks can be completely different, particularly the further down from tier1/2 networks you go. Pulling and using subnet+origin ASN from the actual routing tables provides the clearest and most accurate view on what's really going on
3. A couple people have asked/mentioned things like "but who's routing table?", and it's a very good point/question. What I'm hoping to provide is a means for anyone interested in bitcoin infrastructure to be able to run this and pull their own BGP table data from multiple networks. There are possibly thousands of tier2/3 network operators that will provide this kind of route feed ebgp peer, remotely, for a small fee monthly. Just have to know what is being asked.

So, with all that, the first step was to grab the data in a useful format with some python. Overall takes about 4 seconds to run on a small VM that has all ~1m internet routes:

ewr1-asdata1:~# time vtysh -c "show ip bgp" > bgp_table.txt; time python3 ./asmap_input_generate.py

real	0m1.500s
user	0m0.134s
sys	0m0.303s
Generated asmap_input.txt with prefix to ASN mappings.

real	0m2.424s
user	0m2.179s
sys	0m0.241s    

I've attached that data file, just has subnet and origin/announced by ASN (routes that were accepted into the global routing table by carriers and ISPs)

asmap_input.txt

Next step is to get it into asmap

I'll be heading to tabconf next week -- is there anyone around that might want to meet and discuss this there?

[joe-bp]

Also, here's what a full bgp table looks like from FRR, and the python script to convert it into subnet/asn text file. (subnet and announcing ASN are primarily what matters -- announcing ASN is the last one in the as path field)

bgp_table.txt.gz

asmap_input_generate.py.gz

[fjahr]

@joe-bp Very cool, thanks for your interest in asmap and going with this alternative data sourcing approach. It's definitely interesting and it goes a bit along the lines of what I was thinking of when I wrote the "Using your own source" bit here: asmap/kartograf#11 Though I had the few people in mind that already have the necessary BGP infrastructure running anyway, so their overhead would be minimal. Still, the general idea is similar I think.

A few comments/questions to make sure we are on the same page:

In my opinion, the RIPE RIS data is the most useful (for reasons below)

You probably mean most useful in terms of data completeness (getting all BGP announcements), in which case I would agree. That's both good and bad though, aside from the single data source issue the declared goal of RIPE RIS is to collect data for research so if there is a huge leak/hijack going on they really want to have it in their data set for researching it later which goes completely against our goals of course. FWIW, did you see our write-up on data sources? https://asmap.org/sourcing-data/ I am not sure what you mean by the reasons below though. You only speak about IRR and RADB being suboptimal but that's not what we are using.

IRR and RADB

We are not using RADB for the same reasons you laid out and in terms of other IRRs we only use the IRRs which are run by the RIRs. The reasoning is that they know their customer base (KYC FTW ;) ) and what each of them controls and they can filter the entries based on that, so the risk of wrong/outdated entries is minimized. Our primary source, however, is RPKI and we are getting less and less data from the IRRs as RPKI network adoption grows steadily. I am somewhat hopeful that we can kick out IRRs altogether as a source in the near future. (The last time I checked we got about 10% of the asmap entries out of IRRs. When I started the work on asmap I think it was more like 20%.) We are then using preprocessed data from Routeviews to backfill everything we don't have yet. Routeviews uses the same approach as RIPE RIS to collect their data so I see them as equivalent and due to the trust issues you mentioned yourself we believe there is no upside to processing the data ourself.

What I'm hoping to provide is a means for anyone interested in bitcoin infrastructure to be able to run this and pull their own BGP table data from multiple networks. There are possibly thousands of tier2/3 network operators that will provide this kind of route feed ebgp peer, remotely, for a small fee monthly. Just have to know what is being asked.

That would be cool and I don't want to discourage you from trying this but I have to say that I am sceptical that this will get high numbers of adoption. Paying a small fee to several network operators is probably not a small fee anymore? I also would suspect that paying them implies some business relationship between the node runner and the network operator which would require some version of doxxing yourself and your setup without significant additional anonymity-preserving effort. If this infrastructure is run continuously (to achieve real-time) it would probably require a separate server from the node as well. Can you say a bit about how this relationship would work in practice: Would you maintain a public contact list of network operators that provide this service to node runners for a pre-negotiated price or something like that? How many network operators would you recommend for a normal node runner and how would you discourage people from connecting to only one and possibly all the same one (the cheapest/most convenient)?

I would also recommend that the user runs an RPKI validator and checks the routes they are receiving for their validity status before including them in the asmap. This is really the single best thing to do for data quality. I am curious how you plan to reconcile the differences in the table data received from multiple network operators? Do you configure a preference sequence or something?

And, just to clarify, this means you don't see a point in comparing hashes of the final asmap with other peers because everyone will have a different result anyway right? (different network operator sources, different starting times etc.)

I'll be heading to tabconf next week -- is there anyone around that might want to meet and discuss this there?

I can't make it this time, unfortunately.

[fjahr]

Not sure if this is the right place to propose this

Neither am I for what it's worth but I think here is a bit better than kartograf because I think this would best be implemented as a separate project because so far it sounds like the requirements could be quite different (continuously running etc.) and we could pull out parts of kartograf that are shared between the projects.

[joe-bp]

Thanks for the reply

I did review the write-up and your other comments -- think we're definitely on the same track on this.

A few comments/questions to make sure we are on the same page:

In my opinion, the RIPE RIS data is the most useful (for reasons below)

You probably mean most useful in terms of data completeness (getting all BGP announcements), in which case I would agree. That's both good and bad though, aside from the single data source issue the declared goal of RIPE RIS is to collect data for research so if there is a huge leak/hijack going on they really want to have it in their data set for researching it later which goes completely against our goals of course. FWIW, did you see our write-up on data sources? https://asmap.org/sourcing-data/ I am not sure what you mean by the reasons below though. You only speak about IRR and RADB being suboptimal but that's not what we are using.

I think this is the key area where my initial thoughts/approach to this are a bit different. I'm coming from a background in carrier networking and routing -- forgive me if I'm saying things that everyone already knows or if I'm missing something key about asmap.

If the intent of asmap is to influence inter-node communications to be effectively much more decentralized and distributed in general (as opposed to just relying on /16 boundaries), then I think using IRR and RPKI data is perfect enough for that. If the intent is -also- to prevent or protect against eclipse or sybil attack (by way of route hijacking), then I think the best data to use is the actual routing table, even when it conflicts with IRR and RPKI.

This is because RPKI and IRR data are also intended to help prevent hijacking and provide some level of BGP route AAA, making the data itself intent focused vs what can end up in the tables. If I had to come up with an analogy -- IRR and RPKI data are like reading the laws that are on the books, but the routing table is like talking to the local police/authorities about what is happening in their jurisdictions.

IRR and RADB

We are not using RADB for the same reasons you laid out and in terms of other IRRs we only use the IRRs which are run by the RIRs. The reasoning is that they know their customer base (KYC FTW ;) ) and what each of them controls and they can filter the entries based on that, so the risk of wrong/outdated entries is minimized. Our primary source, however, is RPKI and we are getting less and less data from the IRRs as RPKI network adoption grows steadily. I am somewhat hopeful that we can kick out IRRs altogether as a source in the near future. (The last time I checked we got about 10% of the asmap entries out of IRRs. When I started the work on asmap I think it was more like 20%.) We are then using preprocessed data from Routeviews to backfill everything we don't have yet. Routeviews uses the same approach as RIPE RIS to collect their data so I see them as equivalent and due to the trust issues you mentioned yourself we believe there is no upside to processing the data ourself.

Both interesting and cool that so much of the asmap data is able to be derived from RPKI data.

The concern I have about RPKI, similar to above, is that the further down the network tiers you go (and further out to 2nd world and 3rd world regions), the less RPKI is used for actually validating route advertisements. Particularly with the significant rise in peering points everywhere. Every network geek in the world wants to run their own peering point (and many of them are) and those are free-flows of traffic between networks that usually won't turn on RPKI validation until they're forced to.

Side note: A concern I have with the bitcoin network today is that I believe (and working on proving) that it is highly susceptible to eclipse and sybil, particularly further away from tier1/2 networks. I believe the two easiest way of reducing this risk is getting BGP table data into asmap, and increasing the default outbound connection counts from 8-12 to 48-64.

What I'm hoping to provide is a means for anyone interested in bitcoin infrastructure to be able to run this and pull their own BGP table data from multiple networks. There are possibly thousands of tier2/3 network operators that will provide this kind of route feed ebgp peer, remotely, for a small fee monthly. Just have to know what is being asked.

That would be cool and I don't want to discourage you from trying this but I have to say that I am sceptical that this will get high numbers of adoption. Paying a small fee to several network operators is probably not a small fee anymore? I also would suspect that paying them implies some business relationship between the node runner and the network operator which would require some version of doxxing yourself and your setup without significant additional anonymity-preserving effort. If this infrastructure is run continuously (to achieve real-time) it would probably require a separate server from the node as well.

Yep you're absolutely right, running this is not going to get high levels of adoptions from node runners, BGP and bitcoin core might as well be on different planets.

What I might envision is ~5-10 organizations in different regions that run this setup and provide access to the routing data specifically for asmap by way of http (and things like github). Those organizations would be publicly known. They could be bitcoin related organizations, or they could just be infrastucture or network operators properly motivated to provide this data.

Rough flow:

BGP table (view from X/Y/Z network) -> eBGP peer -> route collector and extractor (FRR and python on a VM/container) -> asmap-friendly data files -> public object storage (http, gh, etc) -> nodes-who-care

Couple of ways this could be implemented:

1. Org runs VMs/Containers on cloud and gets BGP table from cloud provider (ex: Vultr, Hertzner, AWS, OCI, etc). Generally all of these costs are self contained in a cloud provider relationship, and can be done for about $20 per month.
2. Org runs VMs/Containers on their own infra and gets BGP table remotely (multihop eBGP neighbor) from paid relationships with a few networks. This has split costs between running instances (low), and then the cost of eBGP peers to grab this data. There are thousands of potential networks to get this data from, with a cost range of $1 to $100 per month. There are also options like bgp-bmp.
3. Org runs VMs/Containers on their own infra and gets BGP table from their own network infrastructure. If an org already has this infrastructure, the costs to run this are around the same as # 1.

I have scenario # 1 mocked up today (using Vultr), and will have # 3 mocked up in a few weeks.

Can you say a bit about how this relationship would work in practice: Would you maintain a public contact list of network operators that provide this service to node runners for a pre-negotiated price or something like that? How many network operators would you recommend for a normal node runner and how would you discourage people from connecting to only one and possibly all the same one (the cheapest/most convenient)?

Good questions. I think if we're talking about the scenario of ~10+ orgs doing this work (if that's a feasible scenario, kind of like the orgs/nodes hosting the node seed data), all of that would need some basic initial coordination in order to get the most diverse data. Getting a list of networks that would provide data could be as easy as asking for networks willing to via network operators mailing lists and orgs, like nanog.

I would also recommend that the user runs an RPKI validator and checks the routes they are receiving for their validity status before including them in the asmap. This is really the single best thing to do for data quality. I am curious how you plan to reconcile the differences in the table data received from multiple network operators? Do you configure a preference sequence or something?

Generally, the subnet:origin-ASN pairs in the global routing table will be consistent regardless of where you are or what networks you connect to. What differs is the AS path (ASN hops in between) and other metrics to influence routing decisions, but that data probably isn't very relevant for asmap. You will have larger blocks being advertised by carriers and provider ASNs, and then smaller subnets part of the larger blocks being advertised by other ASNs, but generally when an ASN is advertising a subnet, that pair -should- be the same everywhere.

Some exceptions:

1. An org is doing an ASN migration, in which case the same subnet could be advertised by multiple ASNs.
2. A misconfiguration between a carrier and a customer of the carrier
3. An actual route hijack

I think key point is -- seeing the exceptions is very useful

And, just to clarify, this means you don't see a point in comparing hashes of the final asmap with other peers because everyone will have a different result anyway right? (different network operator sources, different starting times etc.)

I would take the opposite approach, I would specifically compare all of the routing table data from as many sources as possible, and when an exception has occurred (differences in subnet:origin-ASN pair), reduce the trust of nodes in that subnet. If there's a "default" hash or bucket where unmatched subnets currently fall, that would be a good place for the exception networks to fall.

[jurraca]

Super interesting, thanks for the insight @joe-bp .

If the intent of asmap is to influence inter-node communications to be effectively much more decentralized and distributed in general (as opposed to just relying on /16 boundaries), then I think using IRR and RPKI data is perfect enough for that.

yes, that's the idea. Glad to hear it :)

If the intent is -also- to prevent or protect against eclipse or sybil attack (by way of route hijacking), then I think the best data to use is the actual routing table, even when it conflicts with IRR and RPKI.

I would say it's not an explicit (or immediate) goal of the project, but i've thought about it. The main difference is level of threat like you said, but also operational intent. Your idea would require detecting "live" anomalies to be useful, and I think most high-sensibility nodes would certainly be interested, but other more casual users wouldn't necessarily.

The concern I have about RPKI, similar to above, is that the further down the network tiers you go (and further out to 2nd world and 3rd world regions), the less RPKI is used for actually validating route advertisements. Particularly with the significant rise in peering points everywhere.

yea we need to be looking at this. Once we've got the ASmap generation a little more performant, I intend to look at the data more closely.

I think key point is -- seeing the exceptions is very useful

100%. I think we could learn a lot from this data. Like you said it's kind of the opposite of the current work, which is "get the best IP-to-AS map to users", but I would love to see the project grow to watch the routing table and provide infra for others to do the same.

Have you looked at real-time BGP stuff like Cloudflare Radar, or pybgpstream?

[fjahr]

Thanks, leaving some further comments but either way you have my support to explore this project. I am mostly curious if we can learn from you about some weaknesses in our current approach that we could address independently of your approach.

If the intent of asmap is to influence inter-node communications to be effectively much more decentralized and distributed in general (as opposed to just relying on /16 boundaries), then I think using IRR and RPKI data is perfect enough for that. If the intent is -also- to prevent or protect against eclipse or sybil attack (by way of route hijacking), then I think the best data to use is the actual routing table, even when it conflicts with IRR and RPKI.

This is because RPKI and IRR data are also intended to help prevent hijacking and provide some level of BGP route AAA, making the data itself intent focused vs what can end up in the tables. If I had to come up with an analogy -- IRR and RPKI data are like reading the laws that are on the books, but the routing table is like talking to the local police/authorities about what is happening in their jurisdictions.

I am curious how you will know that the actual routing table data is better for us than RPKI data when they conflict. I would suspect that that's a route leak or hijack more often than not when that is the case and that's also what my take-away was from speaking with people at RIPE meetups. Would you just always take the route table over RPKI or do you have some further validation in mind that makes sure the differing route table data is legit? Is this happening for BGP peers that have RPKI implemented or do you now speak about those without RPKI? Would be great if you have some documentation or statistics on that because I think it's the first time I am hearing it if this means peers that have RPKI implemented still put entries in their routing table that conflict with RPKI data.

The concern I have about RPKI, similar to above, is that the further down the network tiers you go (and further out to 2nd world and 3rd world regions), the less RPKI is used for actually validating route advertisements. Particularly with the significant rise in peering points everywhere. Every network geek in the world wants to run their own peering point (and many of them are) and those are free-flows of traffic between networks that usually won't turn on RPKI validation until they're forced to.

It's not a big deal for them afaik. They don't have to do the validation themselves. They just have to sign once every few years that they control their IP block with their ASN and load it up to a RPKI repository. They don't have to run any software, just them signing and keeping that up-to-date would be enough for us on the asmap side when using RPKI as a data source. So there really isn't a barrier to adoption but sure, many still won't do it because YOLO. But if you run bitcoin nodes I am really not sure why you wouldn't invest 30 minutes to increase your security against BGP hijacks (assuming their provider has adopted RPKI and provides them with some kind of interface to sign/configure, something like myafrinic provides but honestly I don't know how this shapes up with the rest of the world, see https://afrinic.net/support/rpki/create-add-rpki-roa).

Side note: A concern I have with the bitcoin network today is that I believe (and working on proving) that it is highly susceptible to eclipse and sybil, particularly further away from tier1/2 networks. I believe the two easiest way of reducing this risk is getting BGP table data into asmap, and increasing the default outbound connection counts from 8-12 to 48-64.

How big do you expect the differences to be between the routing table and the routeviews collected data we use now? Do you see a weakness in how routeviews collects data that would lead to it missing significant parts of the routing table?

I would take the opposite approach, I would specifically compare all of the routing table data from as many sources as possible, and when an exception has occurred (differences in subnet:origin-ASN pair), reduce the trust of nodes in that subnet. If there's a "default" hash or bucket where unmatched subnets currently fall, that would be a good place for the exception networks to fall.

The unmatched nodes are in a separate bucket, so to leverage that behavior the table entries where there is disagreement would have to be excluded from the resulting asmap file entirely. If they are to be included with "reduced trust" that concept would need to be implemented in the bitcoin node software peer management and I guess the asmap file format would also need to change in order to be able to hold such information.

[luisschwab]

This is interesting. I can get my hands on a BGP dump directly from a local ISP.