Corrupted Server, That Sucks...

[Tony]

I do apologize guys. I was working in my email program and all of a sudden I get a "no permission" error. I come to this site and get the same error. I go to Broke Auto Forums and the same message. No email, no sites, but I could still get into the server. I started out with restarting httpd (Apache), and it doesn't restart. It fails. I restart MariaDB, restarts perfectly. I try starting httpd again, nothing. So I decide to go in and look for the issues, starting with looking at the files for the sites.

I click on File Manager under the BrokeAuto url and it comes back "nothing found". Okay, that is a little weird. I back out and try RCH, same message. Okay, so I log in as root, and I log into the directory under the root directory and the /home folder is completely gone. /home is where ALL the files are for ALL of my websites, public and private. It is all GONE...

I had just eaten taco's and my heart sank to badly and the anxiety of losing the sites was so great that they almost came back up. I figured there was one more thing I could do and that is to just reboot the entire server. I fired up a command prompt and typed in "reboot" and a second later, The command window printed out the same "reboot" like an echo, but not a linux echo.... I waited 10 minutes and the server didn't come back up.

It was then that I put in a ticket with my server company to go and pull the cord on the back of the server and manually power cycle it. Less than 5 minutes later, I refreshed the server page and it was back up. I quickly went into the file directory and /home was back... I have never in my life had a server selectively delete file locations. Not the files, but the locations and cause these issues. I have asked my Linux buddy if he has seen this hoping we can find a fix to the issue so it doesn't happen again.

Sometimes, you just have to hate Linux. My uptime on the server was only 168 days. Not very long at all.

 

[Geena]

I have been trying to get in here all day, but I kinda figured you took it down for a bit for maintenance or something. I`m glad you were able to get it back up and running. That had to be a bad feeling to have that happen. But it`s back, and you can breath easy again.

 

[Tony]

It was a horrible feeling! If we are planning on extended maintenance I will always let you all know ahead of time. This one was instant while I was working on an email. I don't even remember what email it was lol. But yea, when I saw that /home was gone, holy hell that was not a good feeling.

 

[murankar]

In 20 years of linux I have never heard of that.

The only thing I can think of is the "ntfs" portion got hosed. Basically if you use something like ext4 then your file system is journaled . What I surmised happened is that the journal got hosed and needed refreshed. I don't think you can command line a refresh of the journal.

 

[RandyDSok]

NTFS is a journaling file system... still, anytime anything is being written to the disk it's at risk of corruption whenever that process gets interupted.

 

[Tony]

We were finally able to get in and do a SMART test on the drive (well, Ken did it. I have no idea about this stuff lmao) and there is definitely some errors on the drive. So we are working out how to get the files on the new drive since my host doesn't offer any kind of cloning service. So this may be a lengthy process, and this site will go down again, so be ready for it. I'm super busy for a while and Ken is also super busy so it is taking an abnormal amount of time to get this resolved. But it will get resolved. Please be patient while we work on this.

 

[murankar]

I was having a Brian fart and couldn't remember ext4 so i said ntfs.


Glad you found the issue. So you can't ghost the drive and replicate it. How long would a back up take and a reinstall of CentOs.

 

[Tony]

the swap of the drive is simple and pretty quick. It's getting all the info on the new drive since they do not do anything at my datacenter. They just make sure the server is running, that's it.

 

[Tony]

Hey everyone, as you can see the site is finally back up and running. A HUGE thanks goes to Ken on this one as he is the one that did EVERYTHING on the site to get it going. If my host would have just took 20 minutes and cloned the old drive, it would have taken about an hour total to get the site up and running. Instead, Ken had to do it the hard way. Dude is awesome!

He did make a post under my name in this section explaining what he had done briefly. Man, this was a nightmare! We have never had this kind of issue before. Now I'm thinking about having a second HDD installed as a clone just in case... We will see what that costs.

I want to thank you all for your patience!

 

[Geena]

I`m happy you got it all sorted, and I hope you never have to go through that again. It`s good to have RC-Help back online.

 

[D.O.G.]

I agree with Geena post. Good luck from here on in . Shout out to Ken.

 

[Tony]

yea that was a nightmare to say the least. Definitely not something that I (or Ken) want to go through again.

 

[RandyDSok]

There was some good that came of it...

... I did backup's on my two websites JUST IN CASE of this exact issue... I guess that's no help though ( for you or Ken )

 

[Tony]

If it is just a file backup, that is what we do every single day with a script Ken wrote. We had all the backups, we had all of the files from the old server (after he figured out how to get to them without killing the new drive...), but what we needed was a system image. I just talked to him about this and he is running single parody on his server, and quite honestly, I don't have a need for that much space or the money to pay for all of those drives or what they would charge me to set them up. So I'm just going to do a monthly system image backup. Unfortunately while it does the backup, it will have to put the site down for a while as it is doing the backup.

 

[RandyDSok]

The term is "parity" not parody....

I did a complete WP backup which does the files and mysql backup. Still not the same as what you'd need since the forum db is changed on each post, so as you mentioned you need the forum down while backing it up.

The backups that I do are actually redundant to the ones my server guy does and I believe that includes an offsite copy as well. In short, I'd only need them if the servers had to be completely reloaded AND the backups he does were also unuseable. If he had to put me back up on a fresh install, it'd only take me a few hours at most to resetup ( if that ) since my sites are relatively static ( no constant changes ).

 

[Tony]

I know how to spell it, I just know it drives you nuts hahahaha.