Computer scientists shatter 'terabyte barrier', set new world record

Washington, Aug 1 (ANI): Computer scientists at the University of California, San Diego, shattered "the terabyte barrier" - and a world record - when they sorted over one terabyte of data (1,000 gigabytes or 1 million megabytes) in merely 60 seconds.

During this 2010 "Sort Benchmark" competition - the "World Cup of data sorting" - the computer scientists from the UC San Diego Jacobs School of Engineering also tied a world record for fastest data sorting rate. They sorted one trillion data records in 172 minutes - and did so using just a quarter of the computing resources of the other record holder.

Companies looking for trends, efficiencies and other competitive advantages have turned to the kind of heavy duty data sorting that requires the hardware muscle typical of data centres. The Internet has also created many scenarios where data sorting is critical. Advertisements on Facebook pages, custom recommendations on Amazon, and up-to-the-second search results on Google all result from sorting data sets as large as multiple petabytes. A petabyte is 1,000 terabytes.

UC San Diego computer science professor Amin Vahdat, who led the project, said: "If a major corporation wants to run a query across all of their page views or products sold, that can require a sort across a multi-petabyte dataset and one that is growing by many gigabytes every day."

He added: "Companies are pushing the limit on how much data they can sort, and how fast. This is data analytics in real time."

Better sort technologies are needed, however. In data centres, sorting is often the most pressing bottleneck in many higher-level activities, noted Vahdat who directs the Center for Networked Systems (CNS) at UC San Diego.

The two new world records from UC San Diego are among the 2010 results released recently on http://sortbenchmark.org - a site run by the volunteer computer scientists from industry and academia who manage the competitions. The competitions provide benchmarks for data sorting and an interactive forum for researchers working to improve data sorting techniques. (ANI)