123-reg.com ‘The UK’s #1 domain registrar’, provides hosting services for 3.5 million domain names and 1.7 million websites, as well as email accounts, for 800,000 UK customers. According to the 123-reg website, these customers “are trusted with us everyday”, but this statement was disproved on 16th April at around 7 AM, as many customers lost all of their data, resulting in catastrophe for an array of website owners and small businesses.
This detrimental occurrence was elicited by a clean-up process on 123-reg’s VPS (Virtual Private Server) platform. A script was run to convey the number of machines active against the master database. For customers using the company’s live VPS product, an error on the script showed a ‘zero-records’ response, creating a ‘failure’ scenario – showing no VMs (Virtual Machines), and effectively losing customer data by deleting what was on the host. This was outlined in an email, sent to customers by 123-reg to notify them of this catastrophe, which continued by informing them that, “Our teams have been working long into the night to restore as much as we possibly can. We have also invested in external consultants to recover, in the best way possible.”
Richard Winslow (123-reg brand director), sent his own email to customers affected by the error; in an attempt to salvage any chance of redeeming the company’s reputation. He said: “I understand that some customers may have lost some confidence in the service that we offer. So, I want to explain what we have done to prevent this happening again. We have started an audit on all cron-jobs and scripts controlling the platform, and associated architecture, so that no script will have ability to delete images, only suspend.
For image deletion for those suspended over 28 days we will have a human eye to double check. A new platform will be available by the end of the year for customers which we will provide self-managed and automated snapshot backups, in addition to architecture technology to backup the whole platform, something that is not available on the current platform. I hope this goes some way to win back your confidence.”
The explanation of this chaotic happening by the company was extended by the official statement sent to theMirror, in response to their article detailing the events: “Following your article on 123-reg, we wanted to provide an update on this situation. We suffered a technical fault, which has affected a minority of 123-reg customers. The fault was limited to 67 servers out of 115,000 (across Europe) and it is important to note that only a selection of customers on the 67 servers affected has been impacted.”
The very beginning paragraph of the article appears to be a futile attempt at suppressing the magnitude of their mistake. It seems comical, how a clean-up process error is the cause of this calamity, and yet their attempt at ‘cleaning-up’ their reputation is just as disastrous. The statement continues:
“We are investigating the restoration of each VPS on a case-by-case basis and are working individually with customers to keep them informed of the website recovery process. We are working as quickly as we can to restore service to normal.
“Our VPS product is an unmanaged service and we always recommend that customers implement backups to safeguard against unexpected issues.
“Customers who had purchased 123-reg backups can be online now, as can those who are using another solution for website backup. If customers restore from their own backups, this will not overwrite our efforts.
“Additionally, customers who have restored from their own backups are now hosted on new servers. By using new servers, we will ensure that we do not overwrite the previous servers and impact the data recovery process. We are working with Kroll, the leading data recovery specialist to manage the process of restoration.
“We would like to extend our apologies to affected customers and assure them that we are doing everything we can to restore their data as quickly as possible. We are keeping customers informed as we restore their websites.”
So in summary, “It’s not as bad as it seems, it was only a small portion of customers that have been affected; if you paid more money to us for a backup, you’re fine. If you implemented a backup yourself, that’s also fine; if you haven’t implemented a backup, you should’ve. We’ve got people to help us because we’re out of our depth, we’ll keep you updated, and sorry, again.”
Furthermore, the initial paragraph suggesting that each case is being investigated individually; working closely with each customer, is dubious via the empirical evidence of customer reactions. One customer, quoted bytheregister.com is quoted as saying, “This is the worst customer service I have ever experienced. In short, I’ve been told to rebuild, but not told how to do so, nor if there is any chance at recovering data.”
If this level of dishonesty and betrayal of trust isn’t infuriating enough, see how another customer (again, quoted by theregister.com) disputes the operation in itself, claiming that the practice “is frowned upon in the hosting world… I discussed the issue with a few acquaintances who have experienced running servers in a commercial environment and all of them were shocked to hear that 123-reg ‘automate clean-up of servers’ and that they ‘don’t have backups’.”
The customer continued: “One of my biggest issues is that even if they are running automated clean-up scripts, why aren’t these scripts developed in such a way that they are ‘safe-to-fail’. In my professional opinion, this clearly falls into the realm of negligence.”
He added: “If 123 achieve anything short of a perfect restore, we’re jumping ship. Might as well put in the effort to restore with a reliable host (this isn’t the first time 123 has lost our data in the last few months).”
Another customer quoted by theregister.com, went as far as to reply to 123-reg’s email: “Surely somebody in your management has a grasp on this basic and crucial detail of your business? I guess not, so I suggest you open a corner shop, market stall or something else that doesn’t have the potential to cripple many other businesses.”
To get a broader idea of how other businesses have suffered from the repercussions of the error, here are some examples:
One customer told the Mirror: “I run the @GlastoWatch Twitter account and forum, and my VPS died just before the Glastonbury Festival ticket resale, which is one of the busiest days of the year for me. So obviously I wasn’t very happy.”
Another customer said the outage had taken several of his sites offline, telling El Reg: “It has also taken a lot of people’s email offline and the service is popular with small businesses which will really be feeling the full force of it.” He claimed 123-reg told him they were unable to recover his 50GB of files, as it has no backup of them.
If you have been affected by this fault, or need any assistance with your web development, contact us at Kandekore for expert advice, on: 0121 4000 171