It’s so cool to see something one designed (I did the original sketch and Everaldo did the final polished design) come to life as a cute little conference giveaway ..

Ok, we calling it a “Public Beta”, but it only means we are still missing a few bits, what’s there is solid and working fine !

I am incredibly proud with what we achieved there, going “Cloud” for a VPS setting was a risky new idea, but once that once it sunk in, we knew we had a winner, and above all, we knew we had cracked the trend of things to come.

Now I wish I could say it’s “DONE” .. but it’s not yet, still have loads of ideas and cool features to add, man this has been fun

A grand design .. better see some awards for it !!

Back in the day, say first half of this decade, one of the problems that afflicted sysadmins, such as myself at the time, was downtime.

We hated the fact that when a machine went down, somebody had to run all the way to the DC and check what the problem was, this was especially a problem when you where the one that had to do it, I remember having to drive to the DC at 2:00 AM to fix an hardware failure, just to drive back at 5:00 AM for a fault in the same machine.

Yes, with the years advances where made, for once DC’s started to hire more than just mere NocMonkey’s, and a hole culture of DC’s targeting the Web Hosts emerged, but still, to pay $125 for a reboot and a FSCK was something many of us where not comfortable with, or even could afford, luckily the advent of KVM over IP and IPMI made our lives easier, but still, on a more serious problem, it was still you out of bed at 4:00 AM on a cold winter night rushing to the DC while customers complained on every forum they could find.

Yes, some people will say that a Web Host should have 24×7 staff and own their own DC, or pay the DC to fix the problem, but thats just not the real world, many Hosts out there cannot afford the 24×7, and if they have to pay allot of Tech fee’s to the DC, they simply cant afford to offer those dime-a-dozen ubercheap hosting plans that everybody loves, but complains when things go wrong, heck I remember when I owned my own Small DC back in 2002-2005, in the beginning when lights went out at 7:00 PM the place was empty, we could not afford “on site 24×7″ when we opened it.

So as I glaze at the upcoming VPS.NET platform, and test the fail-over ability that allows a server to take over and boot the client’s virtual machines from a failed server, in a matter of seconds .. all I can think of is all those late nights almost 10 years ago, when I wished I had such a system, how good it would have felt to wake up to a system notice page about a failure that was self-fixed, turn around and go back to sleep …

We are gearing up for a series of tests that will define what sort of SAN we will be using on the VPS Cloud, the actual tests will be performed early next week, for now we are setting up the hardware for it, once we have done it, we will post the results here in the bog.

Here is the info on what we are planning …

Basically, our data backend will be a SAN consisting of a huge number of virtual disk targets. The way this works is as follows:

• Each physical storage server (NAS) breaks down its disk array into smaller logical volumes (using LVM), with 1 volume for each virtual machine

• Each of these volumes is then shared using ATAoE technology across a switched gigabit Ethernet SAN, allowing all the hypervisors to efficiently access any virtual machine’s disks as required

This method allows hypervisors to be fully redundant, allows new storage hosts to be easily added to the SAN, and in the near future will allow full redundancy of storage hosts.

Initially our plan was to use the scenario 1 .. each NAS would consist of 10TB of Raid 5 data, the backups would then reside on separate 10TB (also Raid 5) clusters, if a volume failed, it could be restored into any of the SAN nodes within seconds.

But we then decided we want even greater availability, by RAID-1 the individual NAS themselves, so if a NAS itself fails, a mirror takes over, no need to restore backups (of course, we will still keep the backup nodes, you can never have enough redundancy) .. this will be the case, but not for the January 31st release, as we are still working on the backend code to make this possible.

By keeping the individual nodes at 10TB each, we expected the I/O to be fine, but doubts started to arise lately, and the last thing we want is a 10TB node to become saturated, so a decision was made to do a test, 10TB Raid 5 nodes vs 5TB Raid 10 .. basically, it’s the same config for the nodes, it jut happens Raid 10 halves the available data for the same amount of HD’s than Raid 5 (but you Raid buffs already know that)

Of course, as a tech, I love Raid 10, heck, i’d even use it on the backup nodes if I could .. but the accounting department would be less than impressed `:), no matter how you look at it, Raid 10 doubles the cost of Raid 5, and the more it costs, the more one needs to prove that it’s justifiable …

So come back soon, and we will post our tests results once we have them.

With that in mind, we decided to create one for VPS.NET .. here is it’s story.

We had this idea of a nice design representing the VPS Cloud, an array of nodes (Hypervisors & Data) being worked on by little men, such a construction site .. that idea was fun, but quickly we realized that was the wrong message, a “construction site” might lead people to perceive the cloud as still “under construction” .. there fore, an “unfinished” product… not the message we wish to relay.

While we all know that we will always be working on the VPS Cloud, to create new features and improve on it’s specs, we wanted a more mature image, so instead of a construction site with little men, we went for a finished VPS Cloud type setup with little robots maintaining it.

The next task was, what kind of robot ?, since the VPS Cloud is an omnidirectional object, the robot will need to move in all 3 axis (X,Y,Z), so we did away with legs and gave him a propulsion system that allows him to hover and fly anywhere, second, since he needs to receive data in real time from the Cloud Control, he got an antenna, and last, due to the complexity of tasks and maintenance he must perform, a tool-belt and flexible arms were added …

This was the first sketch of the mascot on the white board (proud to say I did it myself.. hehe):

We where quite pleased with it, so the go ahead was given for Everaldo (outstanding Brazilian artist who is working here in the UK office with us on this and other projects)to clean it up and vectorize it .. and in no time, he had a final design that blew us all away:

What do you guys/gals think ? we quite love it, and the best part is we dont believe we are plagiarizing anybody else, also, the idea will be we will have multiple color and configurations of the robot for different tasks.

We are now trying to come up with a name for this one, as he will be the default mascot, so please, if you have any suggestions, do post them in the VPS.NET Blog

That was the kind of post that makes me real happy that I dont work on Sales .. otherwise I’d tell that individual to shove it, take their money and go look for hosting elsewhere.

Lets do the math shall we ?

We know the host is using Quad Core 2.4Ghz CPU’s, and they offer “equal share CPU” .. this means they are putting (2.4ghz / 200Mhz = 12) 12 VPS’s per server, that on itself makes me doubt the sanity of the host, if they sell 12 of their $8 /month plan, it means the server brings in only $98 /month .. for Quad Core .. thats insane.

Why is it insane ?, because these are the kind of people that later on are wondering around trying to sell their businesses at any price, due to the fact their profit margins are so slim, they realize they can make more money per month by selling fries at the Mc Donald’s Drive-Thru than Web Hosting, and second, I am a firm believer you must be a very profitable company at all times, you OWE it to your customers, otherwise they are not safe with you as you are just a bankruptcy in the making ready to go off at any time (ok, I know I am generalizing here, but look at the amount of bargain basement hosts going bely up the past 5-6 years and tell me I am wrong..).

And the delirious customer ?, well, he is a by-product of the bargain basement offering, simply put, $8 for a VPS ?, such product should no exist, it creates a false statement that this technology can be sold at a certain price bracket, and that is just not true, problem is, customers expect it, and then when they get it, they want more, and again expect to be given more .. period ..

Rant Over …..

We had done some tests on the VPS Cloud self healing measure by shutting down a Hypervisor’s services, and always, the VPS nodes residing there, would automatically boot up on a different one in the Cloud … always below 40 seconds (yes, 40 seconds)… but we decided that was no fun, it was time to go to the datacenter and pull some cables … yes .. I love my job

So there we are, myself and Paul (head of IT @ UK2), we located the Hypervisor we wanted to test, enabled monitors for the main NIC and the VPS’s IP’s … all systems go, the anticipation of success in the air, our palms sweating, giggling like schoolgirls .. and “click” … we pulled the cable …

We then started counting till the VPS’s where back up elsewhere in the VPS Cloud … 1,2,3 … 40 .. ok, anytime now … 60 … 90 ? .. wait a second … what is wrong here ?

3 minutes later, we looked at each other in horror … the unthinkable happened … the VPS Cloud self healing feature .. one of the cornerstones of our offer .. had failed !!!!

Luckily this is still our beta testing .. but what had happen ? it always worked when we turned the services down, why did it fail when we pulled a NIC cable instead ?, what was the difference ? .. and so it began todays “must know” task.

We started by looking at the logs, nothing strange there, actually, nothing there at all for the past few minutes .. thats when it hit .. there was indeed NOTHING in there, the logs show no downtime had ben detected for a Hypervisor.. we returned to the Admin CP, and sure enough .. it still detected that system as being up .. but how ?

Well, as so often is the case, you look into a few hundred lines of code, until you decide to instead look at the obvious … could the internal monitor daemon failed … and if it failed .. why where we not notified ? .. wait .. what monitors the monitor .. ? ..

Simple schoolboy error, we had all sorts of monitors, bells and whistles .. if anything in the VPS Cloud fails, it gets detected within 5 seconds or less .. a true example of monitoring excellence … but what we forgot was .. what if the monitor fails ?

This brings me back to the post topic, the importance of failure .. there was no difference on our test .. pulling the cable or shutting down the services all lead to the VPS Cloud monitor to kick in and do it’s job … it was just a coincidence that this time the monitor daemon had hang, had the failure not happen, this simply oversight could have cause trouble later on, so yes, failure is good .. as long as it happens during beta testing .

So we rewrote some of the daemon to be more robust (we found what caused it to fail and fixed it) and implemented extra monitoring procedures that now we monitor the monitor too

This emphasizes the importance of beta testing and fault simulation, so often we see companies go live with untested ground breaking products and have a miserable first quarter or two of constant failure and bug fixing, many times driving them to closure .. if not properly tested, it’ the small things that will get you in the end.

That is usually followed by “yes .. but where is the actual cloud ?” .. and then my job just got harder.

Thanks to many Marketing Departments the web world over, this is how people see a “Cloud” System

Just a harmless cloud, living on this magical hyperspace world, an intangible object of mystery and awe .. with unlimited resources and power .. accessible only to those mega rich corporation and the smartest of sysadmins.

Fortunately the reality is a bit more down to earth, a cloud looks a little bit more like this:

Yes, sorry to crush your dreams, but the vps cloud really is a mega collection of servers and network storage, man made physical things that can be touched by a human at ground level.

So what is this technology different and ground breaking ? what makes our VPS Cloud so great ?,
Simple .. instead of having 1000 individual servers, each housing 20 or 30 individual VPS’s who are limited by the server they reside on, we have one massive VPS cloud, with the combined power of 1000 servers that can be utilized to deploy VPS’s in seconds that are amazingly scalable, not bounded by a single server limitations and can be self healed in case of disaster in less than 1 minute.

In a regular VPS setup, you are offered this:

Basically you are purchasing a fixed % of a single server, lets look at a theoretical scenario, if the VPS provider allows 20 VPS’s per server you get:
* 5% Of the memory (Ram)
* 5% Of the HD Space
* 5% of the CPU
* X Amount of traffic

But that wont stop there, many VPS providers sell the ability to “Burst” on such environment .. and it’s sold as a “good” thing, except for the other people on the server, when you need to “Burst” .. the resources need to come off the other people’s 5% .. you might not care if you are the one bursting, but what if somebody else is bursting on your machine ? ..

Plus there is overselling, yes, it is possible on a VPS and widely done so by budget providers, once again, it will take away from your expected 5%.

Keep in mind my scenario is not a generalization, some VPS providers have a better offering than others, allot wont oversell, but in the end the limitations of this single server setup are the same for most all.

And last, if the server goes down, so go all the VPS’s inside it, a lengthy recovery process then follows.

The VPS.net Cloud works a little different, instead of a server we have a much more complex setup:

It’s a huge array of cabinets full of servers, al in unisync with each others, sharing a resource pool (alias, the VPS Cloud) and split into groups depending on their function, this my friends, is our VPS Cloud.

In this environment, the vps is deployed across the VPS Cloud, utilizing resources from different sources, it also allows to scale almost instantly (less than 30 seconds in most cases) to sizes larger than most providers dedicated server offerings, it also allows for the creme de la creme .. Self Healing in less than 1 minute.

Our major feature will also be, NO Over-selling .. once somebody buys those resources, they are taken away from the VPS Cloud availability pool, even Hard Disk space is not oversold .. you truly get what you pay for ..

How it’s done is a combination of technical ingenuity and our kick-ass back-end software, exclusive to VPS.net .. this is what really differentiates us from a regular vps provider ..

But the discussion into our backend software and it’s capabilities is one for another blog post, …. stay tuned more more details on days to come both here and at the VPS.net Blog

So as such, time has come to start a blog (located at http://vps.net/blog), to keep everybody informed on the latest and greatest features we are working on for the VPS Cloud, as such, during the upcoming days and weeks, we will post details on the new website, the Cloud features, the user interface, and much more.

VPS.net is a new brand by that great hosting outfit that I work for (yeah, we are great, get used to it), the UK2 Group, owners of 10TB, UK2.NET, Midphase, etc ..

So keep checking that blog page every day (or even several times a day, things are moving fast), we expect to release VPS.net LIVE on January 31st, but will be looking for beta testers in the upcoming few weeks

30 minutes later and no show, I started to get annoyed, told support to get a grip on things and since it was very late, I went to bed.

This morning I woke up early, about 6:00 AM to check it out, site was up, but no DB connectivity .. I have my DB on a separate server (thats the proper way a H-Sphere install is done) .. so I ping the DB server .. no luck, I then checked the IP of the webserver and .. i’ll be dammed, I was on a US server, not a UK server, I ping the MySQL server by name, and sure enough, it was there, but also in the US.

I changed my config files and removed the old US ip, the site was fully up .. except the site was 13 days old !!

Now, for 99% of people out there, this is not a biggie, but for a online shop with sometimes dozens of orders in one day .. this is a catastrophe .. over 100 order records are lost and all since we do inventory by the website stock level, a messed up inventory as well.

Upon further investigation, I found out that Relio’s (webhost) server provider in the UK seems to have decided not to pay their bandwidth bill, and the bandwidth provided cut them off, in essence rendering companies like Relio’s UK servers .. dead (notice, it was not Relio that did not pay their bill, but their server provider, UK Easily)

So I called the bandwidth provider (Uk Grid … all those UK names, they confused the hell out of me today), explain to them our situation and after some negotiations, they agreed to open up traffic to the server for 3 hours to allow me (and others) to access and backup their data out, that was commendable .. those guys where true stars in doing that, after I got the database back, I restored it and the website was up to date … gosh was I relieved.

Why was my site 13 days old to start with ? .. well .. simple, Relio backups my data every day to another server, and every week or two they backup that server to another server in the US .. ok, at first glance, not a bad setup .. but this experience taught me .. not good enough, I need more.

So, since Relio’s UK servers where dead in the water .. and no access to the backup server was possible, they had to do an emergency recovery from the US backups.

I will give Relio credit, to restore every UK account in the US servers, move DNS, remap ip’s, etc .. etc in less than 6 hours .. thats not bad, they did a good job, but it goes to show, no matter how good your first Tier is .. somebody up the ladder is bound to mess it up !!.

Thats the problem with the Web Hosting market these days, too many idiots working on it, we put out livelihoods in the hands of people who later turn around and mess it up for you .. and does not matter how big or small the company is, your are always at risk.

What is the solution ? well, nothing below 50K of hardware and a few K’s a month in fees, so sometimes .. you just have to pay up and shut up.

Thats what I am doing next, first I am going to hire a local hosting company (when I mean local, I mean 3 streets over), I feel the threat of a 300lbs+ guys walking into the office and breaking your nose is a good incentive to make sure they dont screw with you .. that can only happen if they are local, second, I am going VPS (Virtuozzo) .. my idea is to find a setup I can create a Virtuozzo Failover Cluster with a backup residing off-site.. third, I am setting up a mirror of all this on a VPS server in the US, with my DNS residing also offsite (easily re-routable) so if something happens to RIPE itself (The IP consortium of Europe) .. I can still be up, fourth … dont know fourth yet .. but it’s sure to pop up while we doing this.

Overboard, overblown, over-budget ? .. of course .. but piece of mind, i’ve been in this business for 10 years .. I have seen some huge companies working with a setup that would make anybody cringe, promises of redundancy are usually all hogwash, backups many times are simply not practical, yes, they will back your data, they just cant extract it back, customers are at huge risks all the time of loosing all their data, and for some, thats instant bankruptcy, I have try to give away control on how my data is stored and managed, but it’s proven once again, if I want it right, I have to do it myself.