Thursday, June 11, 2009

A lay persons guide to data centers

Thanks to Matt Wells from our marketing analytics team who forwarded me this link to a New York Times article on Data Center expansion.

It is very easy to blame (I have done it) the explosive growth of data centers on the general public expectations to response times, coupled with the explosion in digital information available on the Internet, causing massive (horizontal) growth in search engine foot prints and similar data retrieval systems. We need to find a better answer, if data continues to grow at its current rate and if we make an assumption that index size and the corresponding search engine compute resource needs scale linearly then we are going to be in an appalling situation soon with search engines alone contributing massively to climate change. The amount of digital information increased by 73 per cent in 2008 to an estimated 487 billion gigabytes, according to the latest figures from IDC. If you believe this article Google already has deployed over one Million severs and is growing at rates of 100,000 servers every three months.
In the end it is not going to mater what improvements we make in server and data center efficiency if the application design requires this massive horizontal scale out to match the data growth, I think the only answer will be a radical rethink in the way we search/retrieve information, maybe text indexes need to be thrown out replaced with an alternative algorithm? got any suggestions?

0 comments:

Post a Comment

Disclaimer