This story is about a recent incident that happened to our domain https://hostmywebsite.online. Our domain got suspended by the domain registry, Radix.
And this suspension caused us to bring down almost every domain name hosted by us. ????
This post will explain what went wrong and the cause that leads to 23+ hours of downtime.
The incident was unfortunate and based on a false positive. ????
On 10/12/2018 at 3.00 AM, I got a call from my colleague. She told me that our domain name loading slower and more often drops in connectivity for a few hours. That means the domain loads sometimes and fails to load sometimes.
She checked the server resources and found nothing unusual. Though, she even tried to reboot the systems twice. Luckily our reboot process would take less than thirty seconds.
- DNS Is Not Resolving at All
- Finally, Got Connected With BigRock
- What Is Server Hold?
- Why Is My Domain Name Under serverHold Status?
- Why Was I Not Intimidated When Serverhold Was Applied to My Domain?
- How Can I Get the serverHold Removed?
- We Got Our Domain Unsuspended
- Was My Domain Phishing?
- How Should the Process Have Done?
- Has This First Time Happened in History?
- Issues Beyond Technical
DNS Is Not Resolving at All
In my initial investigation, I found that every DNS query returns an empty response. A record, NS, MX, everything. Just like the domain doesn’t exist.
And at this stage, I noticed that almost every domain hosted with us is unreachable. Our free drag and drop website builder tool are down. Both WHM and cPanel are not loading.
BigRock is currently the domain name registrar. I have logged in to my domain name registrar’s control panel. However, I couldn’t find anything unusual.
Cloudflare is the DNS Manager, so I then logged in to my Cloudflare Dashboard to check if they have any resolution issues. Everything seems perfect. And I became clueless.
Since this is bizarre behavior, I had to seek assistance from the cPanel support team. And after the cPanel support team’s investigation, they backed my assumption. The domain is not responding at all. And to fix this issue, I will have to contact the domain name registrar.
I logged back to BigRock’s control panel. I tried to contact their support department. Interestingly, I found that their support system will not accept any new support requests during their off-time.
Curious me tried to reach their Support Number, 0824-6614011. And the IVR system said their support availability time is 09.00 AM to 08.00 PM IST.
That means I will have to wait six more hours to let them know that I am facing this issue.
Finally, Got Connected With BigRock
I called BigRock and connected one of their knowledgeable agents at 9.30 AM. After his investigation, he said that Radix, the domain name Registry suspended my domain name. And the current status is serverHold.
What Is Server Hold?
Here is the definition of serverHold from Radix’s website.
Okay, but why?
Why Is My Domain Name Under serverHold Status?
Radix’s answer to that question is below.
Why Was I Not Intimidated When Serverhold Was Applied to My Domain?
This is the question I asked many times when I first see my domain’s status.
And Radix has an answer to the question on their website.
I again called BigRock to see if they received such emails from Radix. My bad luck. The second agent didn’t even know if my domain name is under suspension.
I explained to the agent the current situation. I asked if they have received any notification from the Registry about this suspension. The support agent said, No, they don’t have any clue.
I had to disconnect the call and try again after some time to get connected with another support agent.
My third attempt also not so helpful. But I got an email address to check further.
The agent said I would have to send an email to email@example.com. Ah, sending an email? I know sending an email will not going to be so helpful. But still, I sent an email to see what will happen.
How Can I Get the serverHold Removed?
There would be one form that we need to fill and submit to the Radix Abuse team. The form asks basic questions like Name, Email Address, and a Message.
I have submitted the form. Again, after one hour, I re-submit the same form because I was in a hurry.
From their website, I got their Abuse Desk’s email address, firstname.lastname@example.org. I was in a hurry, so I sent an email to this address.
Though I know that both Radix and BigRock are a typical Indian company, I will never get their response, at least in the first four hours.
But, I was under fire. I started getting calls to my mobile number from my early stage loyal customers who were directly connected to my mobile number. It’s a relationship with more than six years old.
I know most companies are proactive in responding to customer’s tweets, so we tweeted to Radix.
We Got Our Domain Unsuspended
After waiting more than Six hours, we finally received an update from Radix via Twitter at 3:34 PM.
And at 3:43 PM, an Email from Radix with the updates that I eager to hear. They unsuspended the domain name and disabled the serverHold.
Funny that we got another email from Radix at 4:30 PM that tells us how to get this solved.
And finally, at 8:49 PM from BigRock’s Abuse Mitigation Team, I received a useless email update. That email contains nothing but almost similar words I wrote to them.
I believe that the BigRock’s Abuse Mitigation Team replied to my email just for the sake of sending a reply. Also, within the same email, I saw that my support ticket’s status is Closed.
I am sure that, BigRock’s Abuse Mitigation Team did not even validate my issue. And after Ten Hours of waiting, I got this pretty useless and who-cares of kind of reply from them.
Anyway, the domain started resolving, and everything came back online very shortly.
Was My Domain Phishing?
I have many reasons to believe that Radix suspended our domain based on a false positive. We never had a phishing URL.
As a proactive protection measurement, we have Cloudflare, Network, and System-level Firewall.
Above all, we have ImunifyAV, the leading malware scanning solution installed on our servers. And based on our internal analysis, this tool is the best available defense tool in the market.
I scanned my website to see if somehow we are infected, and as expected, the scanning result found nothing.
In PhishTank, the only website flagged our domain as a Phishing site; the suspected URL is http://hostmywebsite.online/?q=http://hsbc.co.uk.banking-verification.com&cmd=login_submit&id=1.
To get a much more reputable and reliable source’s expert opinion, I have contacted Cloudlinux Support Team. Cloudlinux is the company that offers most of the best tools and software for shared hosting industries.
And they expressed their thoughts the same as I had.
Since we are using WHMCS, I contacted WHMCS’s Support team to verify the possibilities of such URL parameters and their executables.
WHMCS also confirmed that such URLs would be inexecutable.
I scanned the website with Sucuri and found that only this PhishTank blacklisted us. I don’t know what is the role of ESET here as the link seems a dead link.
Any Other Possibilities?
I haven’t studied how PhishTank’s submission process and their approval takes place. But I guess that might be some desperate competitor of us playing this card against us?
The submitter’s username antiphishyogi looks suspicious.
How Should the Process Have Done?
At Host My Website Online, we hate spammers, online fraudsters, phishing and malware websites, and other threats as much as everyone else hates.
Of course, I am not saying that my systems and networks are hackerproof. Such incidents happen, I agree.
But companies like BigRock or Radix should have treated the situation wisely.
Usually, if any threats are found on a server or a network, the Service Operator would send a strict warning to the involved parties. That email would mostly include specific URLs in question, sometimes ways to solve the problem, and a timeframe to fix it.
This timeframe would be between four hours to twenty-four hours.
But in our case, we have not received any warning from both BigRock or Radix. Even though Radix mentioned on their website that they would send an email to the sponsoring Registrar, BigRock says they don’t know.
And even if I have an infected domain name, once I cleared the threats and submitted the review, they should act fast. I am sure that nobody can accept taking Six hours to Ten hours and more to get this reviewed and fixed.
And sadly, in my case, only Radix can uplift the suspension.
Especially when I informed them about the severity of this issue, the action should be swift.
Has This First Time Happened in History?
Taking down a domain name by the registry is very rare, but it happens to everyone. And that happened to Zoho Corporation, a multi-million company that owns zoho.com. Two months ago, tierra.net suspended zoho.com, and that affected 40 million users worldwide.
Issues Beyond Technical
Yes, this issue is not only a technical headache. It is a pain to us when it comes to customer trust.
In this competitive market, we work very hard to acquire and support our customers. Our customer’s online presence is significant to them as well as for us.
For any reason, if their websites and other critical online services like emails are down, it will impact everyone involved.
It will directly impact our reputation, their business, and online activities. And their end-users also will get affected.
Even if we take every possible measure to prevent, unfortunately, domain names remain a single point of failure in the system.
I can understand that registries may have strict abuse policies, but they also should have quick problem-solving capabilities.
Have you experienced such outages? Let’s discuss that through the comments box below.