**WE ARE BACK, This is what happened**

Post Reply
Message
Author
BillMorrow
*Senior* Admin
*Senior* Admin
Posts: 7177
Joined: Tue Apr 13, 2004 9:40 pm
Location: San Francisco -> Florida -> Georgia
Contact:

**WE ARE BACK, This is what happened**

#1 Post by BillMorrow » Sat Oct 31, 2015 11:56 am

The forum dedicated server was configuered thus: 500 gig storage, 4 gig memory, about 4 cpu's..
on or about monday the 26th of Oct. the Apache OS crashed, probably ran out of memory due, maybe, to a LOT of "bots" and other spurious "guests" just after that mornings backup..

All would have been well except it was a hard crash and required a physical reboot.. which means some admin in utah had to go find the server and hit the reset button..

that was finally done sometime in the afternoon..

OK, all seems well EXCEPT one forum moderator and about 15,000 of his posts just went "poof".. gone..

SO, after doing all i knew how to do (which is not much) i executed a restore from that mornings last backup..

got errors during the backup but the log file said it was still working..

except it did NOT work, it wiped out the forumcode.. clean, gone from the hard drive on the server..

CHAOS, DISMAY, HORROR..! Now what..? Sadly i had not done another full system backup and downloaded to my local NAS storage..

The fellow who helps me out of these situations is a med student who was in the middle of exams and thus not immediately available to sort things out..

about this time i realized i DID have the MySQL database so i downloaded the last system backup in a 66gig zip file, extracted the db file and uploaded that..
also replicated it in several places.. here, a mods site and a thumb drive here as well..

then we started to set up a new version of the forum code and while doing that and while talking to a support guy at hostmonster we both realized that all was NOT lost, that i DID in fact have it all zipped in the 66gig backup file..

soooo, i extraced that, here, locally and uploaded it back to the server and into the right place on the server drive..

now all that was left to do was link the restored code to the restored db file..

about this time, joe, the med student/IT rescue guy comes back from exams and does his magic.. updates the forum to the latest revesion and gets it all back INCLUDING the previously disappeared moderator and his 15k posts..

we ARE back with an additional 500 gig of storage..

the backup file is so big due to all the driver and hardware maintenance manuals i host on the server..
so, we need to decide if these are worth the extra $50/monthly for the 500gig additional storage or, maybe move them someplace else where they won't be backed up daily, weekly and monthly..
FWIW that 66gig backup file eats a LOT of disk space when there are a half dozen versions spinning around on that hard drive..

phew..lots of typing, its late, so i'll quit for now.. and edit this tomorrow if needed.. :)
Bill Morrow, kept by parrots :parrot: & cockatoos
Sysop - forum.thinkpads.com

*
She was not what you would call refined,
She was not what you would call unrefined,
She was the type of person who kept a parrot.
~~~Mark Twain~~~

Post Reply

Return to “Pending & Ongoing Outages”

Who is online

Users browsing this forum: No registered users and 2 guests