Page 1 of 1

Site slowdown - 3/16/2017; UPDATE 3/19

Posted: Thu Mar 16, 2017 6:11 pm
by rkawakami
Hoping this gets through...

Had problems logging in last night. Several "is it down" sites reported forum.thinkpads.com offline. Finally able to log in just now and I see over 900 Guest accounts browsing the forum. Yesterday it hit a new high; 1406 users were online Wed Mar 15, 2017 1:05 am (PDT). From a random sampling of the guest account activity, it looks like most of them are on the Index page (it took several minutes to get any kind of response from my clicks). This appears to be a lot more than normal but I understood that the new server should have been able to handle it. Will try to get a message to Bill and see what he knows.

Re: Site slowdown - 3/16/2017

Posted: Fri Mar 17, 2017 1:09 am
by BillMorrow
it is a hack attack of some kind.. here is what joe hayes has done, so far.. this is froim his email to me wednesday..
I have CSF listening for distributed attacks but these IPs are only connecting once and not over and over, so it's hard to stop. Additionally, I've attached a graph of the countries getting blocked so far. To get the server load down for now, I've blocked all connections from China, Brazil, and Russia. At this point I have to get back to studying. Hopefully your CSF settings will log and block these folks. We just have to wait and watch.
slowly we are knocking out the attackers but it is time consuming..

sorry for all this..

:(

Re: Site slowdown - 3/16/2017

Posted: Sat Mar 18, 2017 1:30 am
by BillMorrow
update from joe on the bot attack or whatever this is:
It's some sort of Slowloris attack that won't trip any filters (mod_security, mod_qos, CSF firewall, etc. etc.). I've setup Nginx as a reverse proxy in front of Apache, still nothing. The connections keep coming. At this point the server load isn't the problem anymore, it's only around 5 right now and you've got 8 processors - so plenty of POWER to keep things going but they're holding onto ports and tieing them up.

Not that it makes things better for you, but you probably aren't the target for the attack. The way this looks you're probably just interference to hide someone's tracks while they're attacking a much bigger target. You're just going to have to wait it out unless you can find someone better than me at mitigating this. There are at least 3,000 IP addresses coming at you, and the better we get at fighting it the more are hitting you.

Good night. Maybe they'll stop by tomorrow. It costs money to run these bot attacks, especially at this magnitude. It won't last forever, and I'm surprised it's been this long.
then i got this:
Ignore my last email. Laid down in bed and had another idea. 3 lines of code from my phone while lying in bed, and all 3000+ bots are GONE. Pretty sure I got those [censored].

Still getting an occasional phpbb error because of the Nginx connections while they're still attacking (it's doing the blocking). I might have fixed it but I'm not sure. Haven't seen the error in a while. Either way I'm going to bed now for real.

Re: Site slowdown - 3/16/2017

Posted: Sat Mar 18, 2017 5:50 am
by RealBlackStuff
It works semi-OK again, but when I e.g. try to post a reply, I keep getting No route found for "GET /posting.php" several times, until the proper connection "kicks in".

Re: Site slowdown - 3/16/2017

Posted: Sat Mar 18, 2017 12:12 pm
by BillMorrow
"IT" will get better..
joe did a great job doing what was needed to stop the attack..

Re: Site slowdown - 3/16/2017

Posted: Sun Mar 19, 2017 3:30 pm
by rkawakami
As of now, the gigantic number of guest users (> 1,000) that had apparently slowed down or cut off access to the forum has been eliminated. It's down to about 50, which is normal. However, it seems that there's a side effect - an error message that pops up from time to time that says "No route found for "GET /xxxxx.php"" or a badly-formatted page that seems to be missing some HTML/CSS code or page elements, like small graphic images. In most cases, simply refresh the browser (maybe several times) and you'll get what you want. However, be careful if your last action was to post a message. Refreshing, and saying "YES" to the pop-up asking you if you want to re-send the request, could lead to duplicate posts. In those cases, before clicking the YES button, see if your original request was accepted by opening a separate browser tab for the thread you were responding to.