Ok, we calling it a “Public Beta”, but it only means we are still missing a few bits, what’s there is solid and working fine !

I am incredibly proud with what we achieved there, going “Cloud” for a VPS setting was a risky new idea, but once that once it sunk in, we knew we had a winner, and above all, we knew we had cracked the trend of things to come.

Now I wish I could say it’s “DONE” .. but it’s not yet, still have loads of ideas and cool features to add, man this has been fun

A grand design .. better see some awards for it !!

We are gearing up for a series of tests that will define what sort of SAN we will be using on the VPS Cloud, the actual tests will be performed early next week, for now we are setting up the hardware for it, once we have done it, we will post the results here in the bog.

Here is the info on what we are planning …

Basically, our data backend will be a SAN consisting of a huge number of virtual disk targets. The way this works is as follows:

• Each physical storage server (NAS) breaks down its disk array into smaller logical volumes (using LVM), with 1 volume for each virtual machine

• Each of these volumes is then shared using ATAoE technology across a switched gigabit Ethernet SAN, allowing all the hypervisors to efficiently access any virtual machine’s disks as required

This method allows hypervisors to be fully redundant, allows new storage hosts to be easily added to the SAN, and in the near future will allow full redundancy of storage hosts.

Initially our plan was to use the scenario 1 .. each NAS would consist of 10TB of Raid 5 data, the backups would then reside on separate 10TB (also Raid 5) clusters, if a volume failed, it could be restored into any of the SAN nodes within seconds.

But we then decided we want even greater availability, by RAID-1 the individual NAS themselves, so if a NAS itself fails, a mirror takes over, no need to restore backups (of course, we will still keep the backup nodes, you can never have enough redundancy) .. this will be the case, but not for the January 31st release, as we are still working on the backend code to make this possible.

By keeping the individual nodes at 10TB each, we expected the I/O to be fine, but doubts started to arise lately, and the last thing we want is a 10TB node to become saturated, so a decision was made to do a test, 10TB Raid 5 nodes vs 5TB Raid 10 .. basically, it’s the same config for the nodes, it jut happens Raid 10 halves the available data for the same amount of HD’s than Raid 5 (but you Raid buffs already know that)

Of course, as a tech, I love Raid 10, heck, i’d even use it on the backup nodes if I could .. but the accounting department would be less than impressed `:), no matter how you look at it, Raid 10 doubles the cost of Raid 5, and the more it costs, the more one needs to prove that it’s justifiable …

So come back soon, and we will post our tests results once we have them.

With that in mind, we decided to create one for VPS.NET .. here is it’s story.

We had this idea of a nice design representing the VPS Cloud, an array of nodes (Hypervisors & Data) being worked on by little men, such a construction site .. that idea was fun, but quickly we realized that was the wrong message, a “construction site” might lead people to perceive the cloud as still “under construction” .. there fore, an “unfinished” product… not the message we wish to relay.

While we all know that we will always be working on the VPS Cloud, to create new features and improve on it’s specs, we wanted a more mature image, so instead of a construction site with little men, we went for a finished VPS Cloud type setup with little robots maintaining it.

The next task was, what kind of robot ?, since the VPS Cloud is an omnidirectional object, the robot will need to move in all 3 axis (X,Y,Z), so we did away with legs and gave him a propulsion system that allows him to hover and fly anywhere, second, since he needs to receive data in real time from the Cloud Control, he got an antenna, and last, due to the complexity of tasks and maintenance he must perform, a tool-belt and flexible arms were added …

This was the first sketch of the mascot on the white board (proud to say I did it myself.. hehe):

We where quite pleased with it, so the go ahead was given for Everaldo (outstanding Brazilian artist who is working here in the UK office with us on this and other projects)to clean it up and vectorize it .. and in no time, he had a final design that blew us all away:

What do you guys/gals think ? we quite love it, and the best part is we dont believe we are plagiarizing anybody else, also, the idea will be we will have multiple color and configurations of the robot for different tasks.

We are now trying to come up with a name for this one, as he will be the default mascot, so please, if you have any suggestions, do post them in the VPS.NET Blog

We had done some tests on the VPS Cloud self healing measure by shutting down a Hypervisor’s services, and always, the VPS nodes residing there, would automatically boot up on a different one in the Cloud … always below 40 seconds (yes, 40 seconds)… but we decided that was no fun, it was time to go to the datacenter and pull some cables … yes .. I love my job

So there we are, myself and Paul (head of IT @ UK2), we located the Hypervisor we wanted to test, enabled monitors for the main NIC and the VPS’s IP’s … all systems go, the anticipation of success in the air, our palms sweating, giggling like schoolgirls .. and “click” … we pulled the cable …

We then started counting till the VPS’s where back up elsewhere in the VPS Cloud … 1,2,3 … 40 .. ok, anytime now … 60 … 90 ? .. wait a second … what is wrong here ?

3 minutes later, we looked at each other in horror … the unthinkable happened … the VPS Cloud self healing feature .. one of the cornerstones of our offer .. had failed !!!!

Luckily this is still our beta testing .. but what had happen ? it always worked when we turned the services down, why did it fail when we pulled a NIC cable instead ?, what was the difference ? .. and so it began todays “must know” task.

We started by looking at the logs, nothing strange there, actually, nothing there at all for the past few minutes .. thats when it hit .. there was indeed NOTHING in there, the logs show no downtime had ben detected for a Hypervisor.. we returned to the Admin CP, and sure enough .. it still detected that system as being up .. but how ?

Well, as so often is the case, you look into a few hundred lines of code, until you decide to instead look at the obvious … could the internal monitor daemon failed … and if it failed .. why where we not notified ? .. wait .. what monitors the monitor .. ? ..

Simple schoolboy error, we had all sorts of monitors, bells and whistles .. if anything in the VPS Cloud fails, it gets detected within 5 seconds or less .. a true example of monitoring excellence … but what we forgot was .. what if the monitor fails ?

This brings me back to the post topic, the importance of failure .. there was no difference on our test .. pulling the cable or shutting down the services all lead to the VPS Cloud monitor to kick in and do it’s job … it was just a coincidence that this time the monitor daemon had hang, had the failure not happen, this simply oversight could have cause trouble later on, so yes, failure is good .. as long as it happens during beta testing .

So we rewrote some of the daemon to be more robust (we found what caused it to fail and fixed it) and implemented extra monitoring procedures that now we monitor the monitor too

This emphasizes the importance of beta testing and fault simulation, so often we see companies go live with untested ground breaking products and have a miserable first quarter or two of constant failure and bug fixing, many times driving them to closure .. if not properly tested, it’ the small things that will get you in the end.