Small but powerful

The experimental Blue Gene gives NCAR computing research a leg up


by David Hosansky

Henry Tufo (left) and Richard Loft are principal investigators for Blue Gene. (Photo by Carlye Calvin.)

When people walk by the new Blue Gene/L supercomputer in NCAR’s Mesa Lab, the first thing they might notice is what’s not there. Other supercomputers consist of rows of cabinets extending through much of the lab’s main computer room. Blue Gene, however, resides mostly in just one average-sized cabinet.

That single cabinet, easily recognizable by its tilted profile, contains 2,048 processors with a top speed of 5.73 teraflops (trillion operations per second). It’s unofficially rated as the world’s 33rd fastest system, according to the most recent census from a benchmark set of equations known as Linpack that is used for judging the speed of computers.

“This kind of machine represents a breakthrough in many ways, and it is likely the forerunner of other machines to follow,” says NCAR principal investigator Richard Loft.

Blue Gene promises to usher in a new era of supercomputing that will feature far denser and more energy-efficient machines. The manufacturer, IBM, used the innovative approach of ganging together thousands of slower processors that demand relatively little electricity (the ML system consumes about 25 kilowatts compared to approximately 400 kilowatts for NCAR’s flagship supercomputer, Blue Sky) and can be packed more tightly because they produce less heat. Hence the nickname for the new machine: Frost.

Blue Gene’s energy efficiency makes it a good fit for the Mesa Lab, where electrical power is limited. “This machine is a highly parallel system with a very low power footprint for the amount of computational power it presents,” Loft explains.

Blue Gene is far more compact than NCAR’s main supercomputer for community access, Blue Sky, which has 50 cabinets that each contain 32 processors. The new system, by one key measure, is also faster. Although Blue Sky has a top speed of 8.3 teraflops, it achieves just 4.2 teraflops on the Linpack benchmark. Blue Gene, in contrast, runs Linpack equations at 4.6 teraflops. A much larger Blue Gene system at Lawrence Livermore National Laboratory recently set a new world record by running Linpack at more than 135 teraflops. When complete, the Livermore system will have 131,072 processors and a top speed of 367 teraflops.

For the moment, Blue Gene remains an experimental architecture with limited capabilities. Unlike Blue Sky, which has a sophisticated job scheduler that can easily manage thousands of user experiments with varying demands, the new machine lacks a queueing system at present. Instead, it is partitioned into a handful of pieces, and users must fit their experiments into one of the available pieces.

“Blue Sky is a production workhorse while Blue Gene is kind of an experimental whiz kid,” Loft says. “It’s an immature but important technology.”

NCAR’s Scientific Computing Division collaborated with researchers from the University of Colorado’s Boulder and Denver campuses to get NSF funding for Blue Gene. The machine was delivered on 15 March, and just eight days later it was operational and undergoing tests. IBM helped defray the costs of the system, in part because the company wants to get feedback about the supercomputer’s performance.

Targeted research

Researchers will use Blue Gene to test several applications and to test and debug new pieces of system software as they become available.

One of the most intriguing applications, from an atmospheric science perspective, is the superparameterization of convective processes in clouds. This emerging and computationally intensive technique incorporates two-dimensional cloud models within a three-dimensional regional or global model. The process can capture cloud properties on scales as fine as one to five kilometers, thereby helping scientists simulate, with far greater accuracy, the impacts of cloud dynamics on weather and climate.

Another application that CU researchers are testing involves a set of numerical methods known as multigrid solvers. Such methods provide accelerated solutions of the partial differential equations that govern dynamics. If they can be developed to run effectively on large numbers of processors, modelers could simulate additional detail about the atmosphere for the same computational cost. That’s where a highly “scalable” machine like Blue Gene comes in. When researchers use a scalable machine and method, the time needed for solving a problem remains constant as the size of the problem and the number of processors are increased.

Researchers will also use Blue Gene for certain types of other applications, such as wildfire and flight test simulations, according to CU computer scientist and Blue Gene PI Henry Tufo. “If you can gain enough confidence in your simulations, then they can take the lead and complement experimentation and theory as a means of investigation,” he says.

Down the road, Tufo hopes to apply even greater numbers of processors to critical problems in fluid dynamics. “What I’m really interested in is looking at fluid flow simulations that scale out to many thousands or millions of processors. We’ve been stuck in the 10,000-processor range in the United States for about a decade,” he explains. He adds that full three-dimensional modeling of seemingly simple processes—the turbulence produced by a golf ball in flight, for example—is still beyond the ability of current supercomputers.

Loft says he has rarely been this excited by a new technology. When IBM created Blue Gene, it was for specific research applications at a single Department of Energy lab. But the architecture demonstrated such great potential that IBM began retooling it for broader types of research. “What we have here is a computer that’s escaped from the lab,” he says.

Loft adds that Blue Gene may soon evolve into a more general-purpose architecture. “I suspect this machine is the harbinger of other machines that will be increasingly flexible and usable for supercomputing,” he says.

Also in this issue:

Examining the N factor

Big year in 2007: IPY & IGY

Firor-Jacobsen Fellowship

Blue Gene: Small but powerful

President's Corner

HIAPER arrives

New directors for SERE, ISSE

Faculty Fellowship Program

Community Calendar