Site slowdown - 3/16/2017; UPDATE 3/19

Post Reply
Message
Author
rkawakami
Admin
Admin
Posts: 10084
Joined: Sun Jun 04, 2006 1:26 am
Location: San Jose, CA 95120 USA
Contact:

Site slowdown - 3/16/2017; UPDATE 3/19

#1 Post by rkawakami » Thu Mar 16, 2017 6:11 pm

Hoping this gets through...

Had problems logging in last night. Several "is it down" sites reported forum.thinkpads.com offline. Finally able to log in just now and I see over 900 Guest accounts browsing the forum. Yesterday it hit a new high; 1406 users were online Wed Mar 15, 2017 1:05 am (PDT). From a random sampling of the guest account activity, it looks like most of them are on the Index page (it took several minutes to get any kind of response from my clicks). This appears to be a lot more than normal but I understood that the new server should have been able to handle it. Will try to get a message to Bill and see what he knows.
Ray Kawakami
X22 X24 X31 X41 X41T X60 X60s X61 X61s X200 X200s X300 X301 Z60m Z61t Z61p 560 560Z 600 600E 600X T21 T22 T23 T41 T60p T410 T420 T520 W500 W520 R50 A21p A22p A31 A31p
NOTE: All links to PC-Doctor software hosted by me are dead. Files removed 8/28/12 by manufacturer's demand.

BillMorrow
*Senior* Admin
*Senior* Admin
Posts: 7172
Joined: Tue Apr 13, 2004 9:40 pm
Location: San Francisco -> Florida -> Georgia
Contact:

Re: Site slowdown - 3/16/2017

#2 Post by BillMorrow » Fri Mar 17, 2017 1:09 am

it is a hack attack of some kind.. here is what joe hayes has done, so far.. this is froim his email to me wednesday..
I have CSF listening for distributed attacks but these IPs are only connecting once and not over and over, so it's hard to stop. Additionally, I've attached a graph of the countries getting blocked so far. To get the server load down for now, I've blocked all connections from China, Brazil, and Russia. At this point I have to get back to studying. Hopefully your CSF settings will log and block these folks. We just have to wait and watch.
slowly we are knocking out the attackers but it is time consuming..

sorry for all this..

:(
Bill Morrow, kept by parrots :parrot: & cockatoos
Sysop - forum.thinkpads.com

*
She was not what you would call refined,
She was not what you would call unrefined,
She was the type of person who kept a parrot.
~~~Mark Twain~~~

BillMorrow
*Senior* Admin
*Senior* Admin
Posts: 7172
Joined: Tue Apr 13, 2004 9:40 pm
Location: San Francisco -> Florida -> Georgia
Contact:

Re: Site slowdown - 3/16/2017

#3 Post by BillMorrow » Sat Mar 18, 2017 1:30 am

update from joe on the bot attack or whatever this is:
It's some sort of Slowloris attack that won't trip any filters (mod_security, mod_qos, CSF firewall, etc. etc.). I've setup Nginx as a reverse proxy in front of Apache, still nothing. The connections keep coming. At this point the server load isn't the problem anymore, it's only around 5 right now and you've got 8 processors - so plenty of POWER to keep things going but they're holding onto ports and tieing them up.

Not that it makes things better for you, but you probably aren't the target for the attack. The way this looks you're probably just interference to hide someone's tracks while they're attacking a much bigger target. You're just going to have to wait it out unless you can find someone better than me at mitigating this. There are at least 3,000 IP addresses coming at you, and the better we get at fighting it the more are hitting you.

Good night. Maybe they'll stop by tomorrow. It costs money to run these bot attacks, especially at this magnitude. It won't last forever, and I'm surprised it's been this long.
then i got this:
Ignore my last email. Laid down in bed and had another idea. 3 lines of code from my phone while lying in bed, and all 3000+ bots are GONE. Pretty sure I got those [censored].

Still getting an occasional phpbb error because of the Nginx connections while they're still attacking (it's doing the blocking). I might have fixed it but I'm not sure. Haven't seen the error in a while. Either way I'm going to bed now for real.
Bill Morrow, kept by parrots :parrot: & cockatoos
Sysop - forum.thinkpads.com

*
She was not what you would call refined,
She was not what you would call unrefined,
She was the type of person who kept a parrot.
~~~Mark Twain~~~

RealBlackStuff
Admin
Admin
Posts: 18051
Joined: Mon Sep 18, 2006 5:17 am
Location: Mt. Cobb, PA USA
Contact:

Re: Site slowdown - 3/16/2017

#4 Post by RealBlackStuff » Sat Mar 18, 2017 5:50 am

It works semi-OK again, but when I e.g. try to post a reply, I keep getting No route found for "GET /posting.php" several times, until the proper connection "kicks in".

BillMorrow
*Senior* Admin
*Senior* Admin
Posts: 7172
Joined: Tue Apr 13, 2004 9:40 pm
Location: San Francisco -> Florida -> Georgia
Contact:

Re: Site slowdown - 3/16/2017

#5 Post by BillMorrow » Sat Mar 18, 2017 12:12 pm

"IT" will get better..
joe did a great job doing what was needed to stop the attack..
Bill Morrow, kept by parrots :parrot: & cockatoos
Sysop - forum.thinkpads.com

*
She was not what you would call refined,
She was not what you would call unrefined,
She was the type of person who kept a parrot.
~~~Mark Twain~~~

rkawakami
Admin
Admin
Posts: 10084
Joined: Sun Jun 04, 2006 1:26 am
Location: San Jose, CA 95120 USA
Contact:

Re: Site slowdown - 3/16/2017

#6 Post by rkawakami » Sun Mar 19, 2017 3:30 pm

As of now, the gigantic number of guest users (> 1,000) that had apparently slowed down or cut off access to the forum has been eliminated. It's down to about 50, which is normal. However, it seems that there's a side effect - an error message that pops up from time to time that says "No route found for "GET /xxxxx.php"" or a badly-formatted page that seems to be missing some HTML/CSS code or page elements, like small graphic images. In most cases, simply refresh the browser (maybe several times) and you'll get what you want. However, be careful if your last action was to post a message. Refreshing, and saying "YES" to the pop-up asking you if you want to re-send the request, could lead to duplicate posts. In those cases, before clicking the YES button, see if your original request was accepted by opening a separate browser tab for the thread you were responding to.
Ray Kawakami
X22 X24 X31 X41 X41T X60 X60s X61 X61s X200 X200s X300 X301 Z60m Z61t Z61p 560 560Z 600 600E 600X T21 T22 T23 T41 T60p T410 T420 T520 W500 W520 R50 A21p A22p A31 A31p
NOTE: All links to PC-Doctor software hosted by me are dead. Files removed 8/28/12 by manufacturer's demand.

Post Reply

Return to “Pending & Ongoing Outages”

Who is online

Users browsing this forum: No registered users and 2 guests