Earnware Corporation has moved its entire technology infrastructure to a new state-of-the-art facility in San Diego, California the weekend of May 22-23.

It was a major decision for us to move from XO Communications to AIS (American Internet Services) as it would involve an extended downtime period for thousands of customers. The networking team was literally faced with moving several thousand pounds of high tech rack-mounted servers and networking gear in the middle of the night and put it all back together again---fully functional with the minimum possible downtime.

Engineers moved 7 racks (8 truckloads) of heavy high-tech gear on which the company was hosting its primary internet and telecommunication services.

For several weeks the new facility had been prepped with brand new cabinets, data switches, firewalls, computer-controlled power delivery units, and a massive amount of organized cabling to receive the relocated hardware.  Each day prior to the move, the networking team went over the extensive migration plan and an extended team of consultants and employees were organized to help on both sides of the move.

The primary reason that the move needed to be done all at once as opposed to in stages is a somewhat technical answer. In fact, the rest
of this post will be pretty technical, but I’ll try to make it as easy as possible to understand:

Earnware Corporation hosts over two billion files (30+ Terabytes) for its clients and users. These files are stored on massive NetApp file servers. These file servers cost as much as a decent four bedroom home in southern California. They have four redundant controller heads and hundreds of disk drives where the data is stored, backed-up in multiple places and served.

The way the system was designed, the databases are all stored on the file servers to maximize speed and redundancy. They communicate across fiber connections to a farm of database servers just a few feet away in the same datacenter. The only way this level of high performance gear can work properly is across fiber in the same room. So this cluster of equipment had to move together. Long story short... the only feasible option was to engineer one big move to happen as quickly as possible over evening & weekend hours.

We began powering down the system at 5:00 P.M. (PDT) on Friday evening. The physical move went very smoothly we were out of the old facility by approximately 10:00 P.M. with the last truckload. I was there the entire time as I had enough networking experience to be somewhat useful. If anything bad happened, I wanted to see it with my own eyes so I could convey to customers exactly what was happening. Fortunately, there were no major glitches.

We had the eight truckloads of equipment installed at the new facility by about midnight. We felt very good about the physical move and all was going according to plan. Next was cabling all the migrated equipment into the new power and networking gear. That was a massive undertaking and we could see our 3:00 A.M. projection was in jeopardy. We had all the critical gear wired up within a few hours and then we started to bring up the massive NetApp filers one by one. Three of the four filer heads came up normally but one of them was throwing errors. We literally burned a few hours reseating drives, cards and switching around cabling, then rebooting and testing about fifteen times. We wound up finding a bad fiber connection and it finally came up normally after the repair. We had the option to run in a lesser performance mode on back-ups, but the better option was to fix the problem which we did. But we were behind.

The sun was coming up by time we had all the filers communicating in the new network properly. The next step was to bring up the database and web servers. DNS propagation issues really bit us as there is no surefire way to get all the systems and various ISPs around the country to point to the new facility IP addresses all at once. Some ISP’s and systems around the country take as much as 24 hours to fully propagate DNS changes. Even inside the new network, DNS (Domain Name System/Server) issues were really slowing things down. We had taken all the usual precautions beforehand but it was not happening nearly as fast as it needed to.

Once we had most of the critical servers back online, it was like a massive traffic jam of data, domain name mismatches and backed up mail queues from changing IP addresses and being offline for several hours. Without going into detail, it was a major ordeal and there was no easy way to get it all caught up. We had no way to drill for this part of the move as we were not going to shut everything off for several hours to see what happened. Getting through it took longer than we had planned for.

It is now Sunday afternoon as I write this and we are still feeling the effects of the move as we find various clients with websites, computers and systems all over the Internet pointed to our old IP (Internet Protocol) addresses which previously hadn’t changed for ten years. The team has been almost non-stop since Friday night getting everything updated to the new location. I’ll compare it to moving your own home or business address and sending in a change of address to the post office. While most places get the change easily so many do not get it or record it. You have to call everyone and change the address manually in so many cases. That’s like what we went through with all these servers and sites we host. Thousands of domains and tens-of-thousands of websites along with the Business Center mail, contacts, calendar and campaigns all of which needed to be brought back online, tested and confirmed functional.

The mail queues, campaign queues, and systems are mostly caught up now and our list of open issues crossed off. It feels good despite the difficult ramifications of having to take the systems down for the move as long as was necessary. Will we ever do that again? Never say never, but it would take a major event to compel us to move again. We chose to host our equipment with American Internet Services in San Diego because they are radically secure and host some of the most important data, websites and companies in the world. We’re now inside their earthquake-tolerant building with back-up systems on top of back-up systems. It’s an incredible facility and long-term-minded major upgrade from the prior facilities at XO communications.

I’m so proud of our team for doing what it takes, working over forty hours in just two days… non-stop all night with virtually no sleep and under such high stakes circumstances. Good job team.

To our customers, I’m terribly sorry for the necessary downtime and thank you for sticking with us through this move. It is extremely inconvenient to not have access to your business tools, I know. We appreciate your business and trust in us going forward. We expect to have a few smaller kinks to work out; however, it should be mostly smooth sailing going forward.