In a special edition, the 40th Friday Follies is a tribute to GoDaddy. Yes, this one will have you up in arms, so do mind the standard “nothing liquid near your mouth or screen” warning. From the stupidest inbox in the blogosphere, it is time for the M3 Friday Follies.
Just in case you were off the web for the entire week, let The M3 Blog bring you the broken news: GoDaddy was hacked. Monday, for longer than anyone cares to admit, GoDaddy’s DNS servers were down. For the non-techies in the M3 Readers, this is the engine which directs traffic across the Internet.
Think of the DNS as the school crossing guard making sure you are not delivered to the Vatican’s website when you are looking for free porn…and vice versa. Now, since hackers looking to steal whatever personal data you may be transmitting over the wireless, interfering with DNS is a quick way of hijacking you and sending you to some malware worm farm just looking to nest in your hard drive.
So, when the largest registrar of domains and hostess with the mostest has a DNS server hacked (representing their still claimed 0.001% down time), it takes about 40,000,000 eggs to cover that face.
The M3 Blog
What does all this have to do with you? M3 is hosted by GoDaddy (a decision I ponder no less than once a week). On top of the inherent risks which come with the (malfeasant) WordPress software, I was not willing to go with a host with a lesser track record. After all, what are the odds something would go wrong with the hosting? (For the math phobes… 1 in 100,000.)
Fortunately for the M3 Readers, the ones most inconvenienced by the outage were the spiders. My crawl rate dropped to less than 15 crawls per hour. (See my comment earlier about the Google spankies list.) It was my tip off something was amiss. Very few of you reported the outage on Monday. Thank you to everyone who did.
The most common remedy after one restores anything which has been hacked it to universally change the passwords (which are to be assumed compromised, if not stolen). Brilliant email number one:
The passwords on your databases do not meet the suggested minimum required strength to ensure security.”
Really? When I changed them two weeks ago, they were pegged at the top of the STRONG side of the meter. But since you dropped the ball, I shall play along.
Remedy #1: Passwords changed on all six databases.
Problem solved, right? Wrong. Problem complicated. Enter WordPress.
It did not take but a slight nudge for me to realize when M3 went down about 35 minutes later, I had forgotten to change some coding. Off to GoDaddy to change the code. (Remedy #2) Alas, no FTP service. And as a reward for my failed attempts, I got a RED banner.
Unknown error. Please reload page.”
Red: I am aware of that. Your FTP manager is down and producing errors. Page reloads do not save the changes.
GD: You have to use the correct password to change the config file, which is located in the FTP manager.
Red: Can I speak to someone without a hearing deficit?
GD: Hold on.
(horrible mobile phone version of The Entertainer. 12 renditions, interspersed with theme music from some spaghetti western…and not the kind I like.)
GD: My information expansion team (???) says the password you are using is incorrect and that is why your website will not load.
(Red takes deep breath. Wishes she was a dragon and could shoot fire through the telephone.)
Red: Yes, I cannot change it because I cannot access the FTP manager to change the password on the config file to reach the database.
GD: Oh. Can you hold? *beep* (More annoying on-hold noise…which I opt out of when I am on hold in the first place, but cannot opt out of when placed manually on hold. 8 more minutes.)
GD: No, the password is still not correct.
Red: Have a nice night. *beep* Oh, did I happen to mention I already got the “Tell us how we did!” email survey and completed it whilst I was on hold 22 minutes? Oh, well. I did. (0041)
After attempting to get a bit of yeah, that is what you have to do from the WordPress people, I answered eight other people’s installation questions and went back to GoDaddy, having gotten no support (6th forum ticket with them to never have one answered).
I get into the FTP manager and look for the file I need to change…which is now missing. Yes, missing. I locate a file which is pointing to a shell database. If nothing else, I can use this to get to the site and merge the shell database with the monstrosity housing M3, or alternatively, modify this one to point at the monstrosity.
Telephone call #2: [Redacted explanation of last call] Any chance you can explain to me why the file, which I attempted to edit earlier, is now moved into another directory and pointing at the wrong database? And if I change the database and password, will this fix the problem which is a direct result of you being hacked?
GD: Can you hold?
Red: Only if you turn off the music.
(3 minutes of silence)
GD: Well, your config file has the wrong password.
Red: (Takes deep breath. Resists urge to award Educational Shoe to the head.) Yes, I know that. I changed it while I was on hold. Can you have the info tech run it to be sure they match?
GD: Oh, I cannot do that.
Red: Yes, I know you cannot. You are a CSR and do not have access to FTP files of clients. Can you get an info tech to run the config?
GD: Can you hold?
(2 minutes of blissful silence)
GD: The password is incorrect.
Red: I did not type it wrong. I have typed it over 50 times in the last eight days as I changed the master passwords on some of my systems and cached them. I did not type it at all here. I copied and pasted it. Are you telling me my computer copied and pasted incorrectly?
GD: The password is incorrect.
Red: Amazingly, the password you claim is incorrect is letting me look at the db in the SQL as we speak. You mean that password is incorrect?
GD: Can you hold?
(3 minutes of wretched [word used liberally] music)
GD: The password in the config file is wrong. Copy the password from the config file and paste it into the SQL password change. That should resolve the issue.
Red: Thank you, I am going to bed.
(Have I mentioned this was nearly 5 hours later…0448 to be exact?)
A Brave New Morning
Now, having jumped through all the appropriate flaming hoops with mere singe marks on my tush, I am expecting to sit down with a cuppa and read comments.
Error establishing a database connection
(And no, by no means am I the only one who saw that (thank you for all the emails, text messages, telephone calls and Facebook messages.). Remedy #3)
Telephone Call #3: We do not support third party applications. WordPress is a third party application.
Red: Something is wrong with your FTP manager. That is my issue. It moved a file, and a config file at that, it forced me to duplicate a database, which stands at the ready should I lose my marbles and create another website, and now it is denying me access to my files again. This is not a WordPress issue. This is a GoDaddy issue. May I speak to someone who is not you?
GD: Can you hold? *beep* (Same annoying noise…4 minutes)
Red: Never mind. I got the file changed. I got there without your info tech being in the FTP, which was the problem all along. For future reference, there is a reason why employees do not have access to client files. You do not know what to do with them. Otherwise, you would be running your own websites and not angering people who do with your apparent lack of cranial convexity.
Have a nice day. (1152)
Begin Waiting Game
Now, GoDaddy is quick to tell you it can take up to 48 hours to propagate something to the Internet. This, for the record, is crap. It takes Google up to 48 hours to index you, but the first time someone pings your DNS, the antique, little cotton top holds up the stop sign and the traffic on the information highway gives you the right of way to the website you called up in your browser’s GPS. (The address bar.)
What took them initially 35 minutes to cache on their end, took me about 45 seconds to change. (1154)
What appears when I call up M3? The ignorant crossing guard puts me on a short bus and delivers me to the infernal Just another WordPress Blog “Hello, World!” version of M3, which I hope no one remembers from 05MAR12. Yes, I wanted tequila, not coffee.
Telephone Call #4: Are you blinking serious the only way to fix this is to reinstall WordPress? How does changing a password fracture a system and move files? (I already know this was done by a well-meaning ignoramus at GD because s/he failed to understand my filing system. As though it were any more ridiculous than their name generator, ugh.)
GD: Can you hold?
Red: No, I cannot. (1207)
Let the Downloads Begin
If you know nothing about moving a WordPress site, I hope you never have to learn it in person. The original move to this site was call for gallons of tequila.
Back then, The M3 Blog was a mere 34 MB move. As of noon today, I began downloading M3 to my local machine…the one without letters on the keys. It is a trifling 684 MB. The maximum a WP transfer can handle is 64 MB. (Bull, it struggled with 34.) Transferring more than half a gig was out of the question. (And just so you know, nine hours later, it is still downloading.)
Enter epiphany: Recreate config file in a different directory, point it at mega db, ignore all advice from GD or WP. Delete all other references to config. Fracture empty DB (gut its schema).
In less than 15 seconds, M3 was back online after having far too many hands touch and mishandle it.
This 15 second solution brought to you compliments of 14 hours of ineptitude and a four hour nap I absolutely needed.
Morale of the story: Even when you go with the company that sucks less than all the rest, they are still only as bright as the small appliance bulbs who are not educated to identify the problems, even if they have not the power to fix them.
I hope your week has been free of hackers, CSR and GoDaddy outages. From the stupidest inbox in the blogosphere, thank you for joining me for the 40th edition of the M3 Friday Follies. Better late than offline.
Who wants to help me poison pen the comments to the latest survey on telephone customer support? PS How many of you are seeing ads for hosting from a company mentioned in this post? Bwahaha!