hub.bugout.link performance

[draeder]

Hi Chris:

I've been building a tracker server tester and as part of that I built my own tracker server. I wanted to share some observations that might help you understand the performance issues you've been experiencing with hub.bugout.link.

Short Summary
Your tracker server got added to a list of tracker servers that is used widely by users in China who use an app called BitComet. The users in China are very likely killing your server. This issue was opened by someone to have your tracker server added to trackerslist. You can open an issue to have your server removed, too, per the trackerslist readme.

I know this to be the reason for your server's performance issues by reviewing https://hub.bugout.link/stats and seeing that it matches my own server's stats after personally requesting my server be added to trackerslist, in particular the type of client showing up in the stats: BitComet.

Details
While testing my tracker server tester, I found that your server often responds with a server 400 bad request. I remember you mentioning that SlingCode introduced extra load to your server, and at the time, that was the explanation for the performance problems.

However, I am starting to see the same kinds of problems with my tracker server, and I don't have any major applications built that uses it yet.

My tracker server was built to run on Heroku since it has a free plan and also a low price plan. After getting things working, I opened an issue with trackerslist to have my tracker server added to the list -- I was curious to see how it would perform with many users. Once it was added, I noticed that my Heroku server was generating a lot of errors and the response time was averaging 30s(!). I've gotten 3.5 million requests to my server and the majority of them are from China users using BitComet.

Since I am using Cloudflare for the CNAME associated with my domain and the Heroku app link, I was able to identify the origin of the traffic with Cloudflare's web analytics. The majority of traffic was coming from China. So, I added a firewall rule to block all traffic from China. This solved the performance issue for about 24-48 hours or so, bringing the response time of my server back to single digit milliseconds. But today it's getting pegged again despite the firewall rule. I have a case open with Cloudflare support to explain why only ~ some ~ traffic from China is getting blocked and other traffic let through.

In the end, I would like to block certain types of clients, or only allow certain types of clients like WebTorrent. It seems that bittorrent-tracker could allow this, since the stats page lists the connected clients, but it's not clear from the documentation how to do that.

I hope you find this information useful.. 🔢

Thanks,
Dan

[chr15m]

@draeder thank you so much for this analysis, that is indeed useful information. Will have a bit of a think about how to handle this.

[draeder]

@chr15m I personally want users from a restricted country like China to be able to use my tracker server however they need to, and ideally I wouldn't block any clients.... But the demand from China is so high, it leads to either performance issues or costs. I would love to explore a solution that addresses both....

[chr15m]

@draeder i have designed but not tested a possible solution to this using a "proof-of-work auction" or hashcash auction. Basically you give your server a limited number of slots which it is able to support, and clients have to perform proof-of-work of a sufficient difficulty to connect. This ensures that the server is never overloaded, and also that clients are given a signal about which server they should connect to (lower PoW is better). I've started preliminary work on this, will let you know if/when I make progress. Let me know if you come up with anything yourself.

[chr15m]

@draeder another alternative I've been exploring is to tie cpu load / memory consumption to the PoW value. So as the load goes up and memory becomes scarce, the PoW clients must perform becomes more difficult, warding them off.

[draeder]

@chr15m If you need me to test anything you've created, I can add new servers to Heroku as necessary with little to no cost as long as those servers don't live too long. Just let me know.

What I've been seeing with Heroku though, is CPU and Memory doesn't really increase as the requests increase.. The issue is the number of errors increases as peers get connected together, and that seems to create the increase in time to respond to requests. Specifically, as peers connect, sockets are no longer needed, so the server responds with 503 server unavailable due to the websockets timeout. It's unnecessary use of CPU time to handle/respond to those timeouts.

In Heroku, there's no way to address that... but there must be something that can be done in Node.js.

[chr15m]

@draeder huh, that's interesting. So it sounds more like the sheer number of websockets is the issue. There could also be memory leaks etc. in the server itself. I think I remember @DiegoRBaquero saying their tracker was restarted every 24 hrs, so maybe I should do the same with hub.bugout.link. 🤔

[draeder]

@chr15m Well that's interesting, because I was talking with the developer of fake-bitttorrent-client about how to address timeout issues for my tracker server tester ... In that issue I found that limiting the number of sockets for the client request helped speed up responses. Looking at it from the other direction may be useful.

[draeder]

Also, there is a limit to dynamically assigned ports that can be opened for any computer using IPV4... https://stackoverflow.com/questions/113224/what-is-the-largest-tcp-ip-network-port-number-allowable-for-ipv4

[chr15m]

@draeder ah yes, and all of these issues point to the benefit of there being more trackers that each handle a smaller load individually.

[chr15m]

More and smaller tracker servers that are part of a mesh also means the whole system is more robust to single individual trackers going offline.

[draeder]

Right... so how to get them participating is the question? I made P2P Tracker so anyone could run a tracker server either locally or in Heroku.. The trouble is, who will run it? My tracker server tester is finding very few responsive trackers. Yours and mine are in the list of working trackers..... unless China is killing our servers.

Your tracker server mesh idea is important, but people need to run their own trackers, first -- then run those tracker servers within the mesh....

[chr15m]

how to get them participating is the question?

I think the only think you can do is put a thing out there and tell people about it. If it is valuable and you have explained the value, people will run it. If not, try again.

[hello-smile6]

Why not just remove the tracker from the index or lower its position in the index?