HTTP ERROR for queries in pybgpstream

[shahzebmustafa]

I am trying to get all paths from ${origin} AS to ${destination} AS.
I want to scan all collectors, to make sure that if a path exists, between two ASes, I always get that.

I'm running into two issues, one of them is an error, and one of them is a confusion if you can help me out with that.

The error:
When i don't limit my collectors, the following errors are printed (the script doesn't stop though):
Filter: origin 11492 and peer 7922

HTTP ERROR: Failure when receiving data from the peer (56)
HTTP ERROR: Transferred a partial file (18)

Filter: origin 11492 and peer 22773

HTTP ERROR: Couldn't resolve host name (6)
2020-07-22 12:50:49 62124: bs_transport_file.c:40: ERROR: Could not open http://archive.routeviews.org/route-views.wide/bgpdata/2019.06/UPDATES/updates.20190601.0800.bz2 for reading
2020-07-22 12:50:49 62124: bgpstream_transport.c:97: ERROR: Could not open resource (http://archive.routeviews.org/route-views.wide/bgpdata/2019.06/UPDATES/updates.20190601.0800.bz2)
2020-07-22 12:50:49 62124: bgpstream_reader.c:170: WARNING: Could not open (http://archive.routeviews.org/route-views.wide/bgpdata/2019.06/UPDATES/updates.20190601.0800.bz2). Attempt 1 of 5
HTTP ERROR: Failure when receiving data from the peer (56)
HTTP ERROR: Transferred a partial file (18)

Similar lines print on the terminal multiple times (same numbers in the brackets)

The Confusion:
Even after all the errors get printed, I do get the results, but quite often they show that no such paths exist. While this is always possible, but I just wanted to know if there's something I'm doing something wrong.
If you have any recommendations to make the code or results better and more reliable, I would be happy to hear!

One additional information that might help in optimization is that all ASNs I am concerned with, are from within the USA.

I am using pybgpstream, and here is my code:
Given a source ASN and a destination ASN, this function returns all the paths that pybgpstream provides.

def getASPaths(src, dest):

    paths = []
    filter = "origin {} and peer {}".format(str(src), str(dest))

    stream = pybgpstream.BGPStream(
        from_time="2019-06-01 07:50:00", until_time="2019-06-01 08:10:00",
        filter=filter,
    )

    for elem in stream:
        paths.append(elem.fields["as-path"])

    return list(set(paths))

[alistairking]

Hey,

I'm not sure exactly what is causing those errors, but judging from the output, bgpstream has correctly retried and (i assume) managed to consume the data.

As for your question about finding paths between ASes, you are asking BGPStream to give you paths that have src as the origin (this is probably fine since we should see "all" origin ASes), but that have dest as the peer. Note that RV and RIS do not peer with every AS, so unless your dest ASN is one of the few hundred ASes that RV and RIS peer with, you will not find any paths. Here's an example of the peer ASes for the route-views.sg collector: http://www.routeviews.org/peers/route-views.sg.routeviews.org.txt and a list of all RV peers (per collector): http://www.routeviews.org/peers/

Hope this helps.

[shahzebmustafa]

Thanks a lot @alistairking!
I believe the errors are because of some collectors, but I don't think there's a way to identify them, is it?

I was certainly doing the wrong thing searching for dest as peer.
I only wanted paths with specific origin and destination ASN.
This regex expression worked for me: "aspath ^{src}_ and aspath _{dst}$"
I'm not sure why, but this one did not work: "aspath ^{src}_*{dst}$"

[dacCR]

Is it related to this?
CAIDA/pybgpstream#52

[mbUSC]

I'm experiencing the same

[sunjiankun126]

I'm experiencing the same too