Menu
NullMind's Lair
  • Home
  • Statue Collection
  • Jacket Collection
  • Contact
  • Archives
  • About
NullMind's Lair

I should have been a cowboy !

Posted on June 3, 2005 by NullMind

What a crappy night.

One of our client’s d-servers decided to act up at 3:00 AM, when running most commands it would error with a Input/Output Error message.

The initial consensus was that this was a Bad HD, but this exact server had the HD replaced on April 11, although not impossible, the prospect of a 2 month old HD failing already was hard to believe, but this was not the problem, the problem was .. the client had not done any backups.

Even so it’s not my responsibility to make sure customers have backups, somehow in the end I knew we would take the head for it, so I attempted to backup the one site this server hosts, and it failed, mid way thru the backup, the server simply .. CRASH, so we moved to repair mode, while the tech ran a lengthy FSCK, I started to plan the move to another disk, and started to complement the terrible day I would have ahead, this is a online pharmacy, which demands a uptime SLA allot higher than usual customer do.

The first FSCK for some reason failed (I now believe due to tech error), so we moved on to a second one, using a few extra options .. and thanks to the lucky start .. it seems to have done it.

I checked the logs, there were no EXT3 or HD related errors even before when it was giving the I/O error .. so I’m uncertain if the HD was bad, or if it was just a corrupt sector of data, were part of the kernel or a major OS file might have been.

Regardless, this has allowed me to now have a stable system, and my first command to it was “BACKUP”

I have let it run now, as the HD seems to be behaving, but if it fails again, with a backup on the side, I can restore it in an hour, vs trying to harvest data from a bad drive, which could have taken days.

I really should have chosen a less stressful profession, like dismantling live bombs… at least those guys get to sleep.

Technorati Tags: Computers, work

Related

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Carlos Rego – OnApp CVO & Co-Founder

Recent Comments

  • Carlos on French corvette ‘La Légère’
  • NullMind on My Magrette ‘Bronze’ … sort of
  • NullMind on Finally got the work desk fully setup, took forever for some things to arrive due to the delays on the postal system, but it’s finally all here !!
  • Fintan on Finally got the work desk fully setup, took forever for some things to arrive due to the delays on the postal system, but it’s finally all here !!
  • wm. spencer on My Magrette ‘Bronze’ … sort of

Null's Flickr Photos

Відпочинковий комплекс ШепільськаSete CidadesIslamic MuseumBoat in DohaVM-E with 50mm/0.95Double Rainbow
More Photos

Categories

  • Books / Movies (8)
  • Camping (1)
  • Cars (14)
  • Check Ins (69)
  • Computers / Internet (146)
  • Cooking Recipes (2)
  • Family (53)
  • Featured (4)
  • Flights (176)
  • General (107)
  • Models & Statues (42)
  • Photography (276)
  • RC (66)
  • Short Rants (14)
  • Stuff (6)
  • Toonz (24)
  • Trips / Places (36)
  • Uncategorized (24)
  • Vape (3)
  • Voicemails (1)
  • Watches (6)
  • Work (48)

Tag Cloug

1/4 scale 8IGHT-T 8T Azores BSL canon50mmf095 Car Check Ins Copenhagen CPH england Flickr IFTTT Instagram LCY LGW LHR LIS Lisbon london Lviv LWO Marvel PDL Plymouth Prowler Ponta Delgada (Azores) Prowler RC RC NUT sonya7r Statues STN Tattoo Trip Tripit UK2 unitedkingdom VIE Vienna VPS VPS.NET Warsaw WAW Work ¼ Scale

Next Azores Trip

Quick TripFebruary 29, 2020
Going Home to Visit
©2021 NullMind's Lair | Powered by WordPress & Superb Themes