SSD benchmark:


RAID 10 benchmark:

 

I guess I've been lucky. Aside from one OCZ SSD, I haven't had a total failure of a hard drive (with loss of data) in more than 10 years.

For a long time, I was a huge fan of Deskstars. Back when they were actually still IBM. Even during the era when they were suffering massive failures left and right, I never had one fail on me. One of 'em did develop a weird clicky noise, but it ran like that for several years. The 2TB drive in my media server is a Hitachi 7K2000, which I bought right when they first came out.

I had a Toshiba 1.8" drive start to go out in my Vaio a few years ago, but it was gradual. Started making a whining noise, but it ran for more than a week like that, giving me plenty of time to back it up and order a replacement.

I've had a couple of Seagates and WDs over the as few years as well. No problems with any of them.

In fact, the last drive which I can honestly recall suffering a total failure was a full-height, 5.25" SCSI drive made by Control Data Corporation, which I think was somewhere around 100MB. The spindle bearings failed.


Still, I've gotten pretty ---- about backups. Every machine I use on a regular basis either contains a dedicated internal backup drive or is linked to a dedicated NAS backup volume, and the machine performs a differential backup of itself every single night. This way I'm protected not just from physical faults, but also from data corruption which a RAID would be powerless to stop. (eg: if you hose the OS, or accidentally over-write a file, etc., the RAID will just faithfully make a duplicate copy of the bad data. With differential backups, I have access to 30 days worth of valid data, and can revert either individual files or the whole machine back to any point in the past month.)

 

$500K. That's extraordinarily expensive for SATA drives. I just left one of the top 3 SAN data storage companies in the world and that pricing is very high. Sounds like they had a big bunch of other stuff bundled in? I know, because I wrote the quotes...



Good boy. Among the many things I'm an expert in (not wrenching, duh), I'm an expert in data storage. Now the question I have for you is: are the backups offsite?

I back up my stuff onsite to my NAS and also send everything to the cloud. Whee.

 

My desktop has only local backup (on RAID10 though, but with shitty drives) since I only deal with code and EC2 credentials which fit fine on an encrypted volume of a flash drive. Everything else is dispensable. All of my VMs are in a cage with a proper KVM/QCOW2 over Gluster setup.

I'm running Supermicro 2U boxes with 6 hot swap bays and an Atom-based Mini-ITX server board. The whole setup of 3 boxes (3x replicas with RAID10 behind them) cost me right around $3,604 from NewEgg (drives were purchased bulk from eBay). That's only 1.5TB though (500GB SATA WD Black) but it's 400Mbps on writes and 650Mbps on reads. That's with absolutely no cache other then buffer on hard drives themselves.

If you reconfigure it slightly and use same 3TB drives and larger enclosures + RAID controller to handle more drives, it will cost ~$0.50-$0.60/GB.

That's $8.62/GB that they've paid by the way.

 

No offense, but I haven't seen an e-pen0r circle jerk this big since I stopped posting at the hardforums.

I feel like i'm back in 1999.

If yo dont mind, What do you guys do anyway? If I had to guess id say either you operate a colo, lease rack space in a colo or consult data management services.

 

DevOps consulting, infrastructure architecture, hybrid (private + public) infrastructure management, application lifecycle management (automated deployment, QA, etc), and a little bit of hosting on the side.

 

It depends on how you look at it.

The overnight differential backups are all on-site, to dedicated hard drives co-located with the PCs which they are backing up. Some in the same physical case, others in the same room via USB or the next room over via ethernet.

I do have one USB drive which holds a mirror of my primary home PC which I keep at the office, though I'm fairly undisciplined about updating it. Maybe once every three or four months I remember to bring it home and cycle it.

Of late, I've started using DropBox (cloud storage) for projects that I am actively working on. This has the obvious benefit of mirroring the data across all of my PCs including both laptops so that I can access it wherever I am, and the incidental benefit of creating de-facto backup copies of everything in five different places in real time. Dropbox, of course, does not guard against accidental data corruption, so I include my dropbox directory in the nightly differential backups of two of my PCs- one at home, one at the office.

 

So in a nutshell you make sure a companies production, development and test environments meet application requirements from a technical point of view, are always running properly and you maintain a process for deployment of applications/changes from one to the other doesn't affect the system as a whole?

 

Urban. EC2?

You use AWS? I find AWS very interesting because it puts infrastructure management in the hands of developers. I learned all about it when I interviewed with Amazon a few years back. Bezos is a visionary.

 

EC2 is OK. It's very limited on resources and also very expensive. This is why I manage hybrid infrastructures (your own metal is always cheaper).

Saml01 - not exactly. I mostly focus on auto-scaling and easy deployment to a large cluster. Most of my clients have distributed applications so there are LOTS of different components that are mini applications on their own. Real complexity comes in optimization. Think about it this way - ~10x c1.medium (app servers) + 5x m1.small (web) + 2x c1.large (queue broker) + other crap. It all adds up to a large sum of money and my job is to find all the things that can be optimized for lower disk I/O and network I/O. Once that is done, I go through application code (whatever language it is written in) and try finding obvious bottlenecks that can reduce latency even further.

 

That's pretty impressive. What do you do when you dont have access to the code? Not every company develops their own applications.

 

I mostly deal with start-ups that have just secured their first/second round of financing and are looking to scale their infrastructure

 

Makes sense.

Do they find you or do you find them?

 

Both. Helps to have lots of contacts in the field I also do presentations at local tech meetups once in a while so that is another way to pick up some work.

 

Cleaned up wiring a little