The application of Moore's Law may have led to incredible advances in computing, but the growth of storage capacity is even more impressive. Disk and tape storage density has doubled every year, driving out cost and bulk, a phenomenon sometimes called Kryder's law. $250 bought an amazing 20 GB of hard disk space in 1999, but this is nothing compared to the 2 TB that can be had for the same money today. Yet enterprise storage costs have not dropped. The real cost of an enterprise storage array has not fallen significantly in that same period. In fact, the total cost of maintaining storage has actually increased as a percentage of IT budgets.
Why isn't storage getting cheaper? This series of articles attempts to answer this question:
- Too Cheap to Manage
- Too Much to Manage
- Tiered Storage
- The Glass Floor
- Storage as a Service
Too Cheap to Manage
In the 1960's, the development of nuclear power promised to reduce the cost of electricity to zero. "Too cheap to meter*" meant the rapid switch to electricity throughout the home, exemplified by the widespread application of electric baseboard heating. We all know how this story turned out: Rapid increases in power usage overtaxed the ancient power grid, nuclear power was halted by environmental and NIMBY protests, and the energy supply remains one of the most important issues facing humanity
My first hard disk drive boasted a generous 20 MB of capacity. With the addition of compression, this lasted me from 1988 through about 1993, when I finally ran out of space and had to upgrade. My next computer included a 100 MB drive, and I was very pleased indeed to be able to take along all of my old files and still have plenty of capacity for my new work. This process repeated every few years: I remember the joy that 200 MB, 840 MB, 1.2 GB, and 10 GB brought me through the decade. Each time I upgraded, the new capacity seemed limitless. More than I would ever need. Too cheap to manage.
Disk drive capacity growth has been phenominal for the last 20 years.
(Public domain image from Wikimedia Commons)
"Too cheap to manage" was the cop-out for corporate IT. I remember installing my first 10 GB storage array. I remember EMC announcing the amazing 1 TB Symmetrix 3000. I remember migrating 20 TB through a temporary Fibre Channel SAN. I had stopped being amazed by storage capacity. Instead, I was amazed by the size of the data sets I was working with. We once used quotas to limit users to 5 MB of storage on the corporate file server, but quotas went the way of the dodo once multi-TB NAS filers were installed.
But data growth suddenly caught up with us. "Too cheap to manage" led to "too much (data) to manage."