|The IBM SP, acquired last year, is an important part of NCAR's supercomputing strategies. (Photos by Carlye Calvin.)|
The plan grew from the work of an NSF-appointed Code Assessment Panel that visited NCAR last summer. The panel, whose members were computer science specialists, was asked by NSF to review NCAR's key models from a software standpoint. Their report, issued in August 1999, stressed the need for change. As early as 1997, NCAR had begun a shift from vector machines toward distributed-memory, shared-multiprocessor (SMP) architectures with the introduction of a 64-processor Hewlett-Packard cluster, and a 128-processor SGI Origin 2000 went into use in 1998. A much larger SMP machine, an IBM SP, was installed in August 1999, just after the NSF review was complete. While acknowledging the promise of the new IBM, the review panel asserted that NCAR would need to modify its model development strategies in order to remain a leader in its field.
"There was a genuine interest on the part of the leadership here in responding [to the report] in a positive way," says John Michalakes, a visiting computer scientist from Argonne National Laboratory who's been working at NCAR for over a year. Robert Serafin (then the director of NCAR) asked Steven Hammond, manager of the Computational Science Section of NCAR's Scientific Computing Division (SCD), to chair a committee that would prepare a strategic plan for high-performance simulation. Timothy Killeen joined the process shortly after becoming NCAR's director- designate.
Instead of looking at each point in the modeling process separately, the committee studied the entire computing environment "end to end," says Hammond. "There are some fundamental changes [proposed] in the plan."
Killeen points out, "During the time that NCAR has existed and models were being developed, computer science has grown out of its infancy to be a mature science. There's great strength now in theory, practice, applications, quality of service, networking, bandwidth utilization, and also on the details of how software gets developed and can be made agile. So you could say that now is the time to do this."
The new emphasis on computation will bring a greater "level of formalism in our modeling activities, consistent with [how we develop] field programs or observational programs," says Hammond. "There haven't typically been design reviews for our software. A lot of things that are part of the systematic process of software development in the commercial sector would be very beneficial to software projects conducted here."
Equal to the scientific challenges ahead is the challenge of creating a social environment where the new interdisciplinary teams can thrive. "But we're good at putting together large teams with a shared vision," says Killeen. "A buy-in from the big community is a social organization feat that Maurice [Blackmon, director of NCAR's Climate and Global Dynamics Division] and his people have already accomplished" in developing the CCSM.
Another question is how to divide labor between the atmospheric scientists and computer scientists. One way is to confine each group's main concern to its own layer in the simulation code. Scientists, concerned primarily with algorithms for dynamics and physics, are able to work within one layer of the software hierarchy to code these in a standardized, platform-independent form. This leaves parallelism and other computational concerns to an implementation layer tailored to the machine at handprimarily the domain of the computational specialists. The Weather Research and Forecasting Model (WRF; see p. 6) is being built in this way, with a mediation layer in between.
Frameworks (reusable collections of code) are being explored as a way to streamline the creation of these model implementation layers. NCAR has submitted a three-year grant proposal to lead the development of an earth system model framework that could be used for multiple models, in collaboration with NASA, Argonne, the National Centers for Environmental Prediction, Los Alamos National Laboratory, the University of Michigan, the Massachusetts Institute of Technology, and NOAA's Geophysical Fluid Dynamics Laboratory.
NCAR staff had an opportunity to discuss these and other issues related to the new plan at a workshop on 31 August. Upcoming workshops are planned to involve university scientists and other community members.
Computer scientists will be more involved at NCAR than ever before. "We're working through a strategy of how to make that happen," says Killeen. His office is hiring a distinguished visiting computer scientist for an initial term of about one year to help with that process.
Killeen believes that the problems we'll be offering to computer
scientists are right up their alley. "You have this inhomogeneous
set of providers and software; different pieces of the coupled
model have very different requirements of memory; [there are
issues of] storage, computational efficiency, parallelizability,
swapping in and out of memory
Michalakes and NCAR scientist Joseph Klemp work together in exactly the kind of team that's called for in the NCAR plan: the WRF model development group. Michalakes says that for him, "The attraction of the WRF project is that there's a genuine partnership between the scientific members of the team and the software engineering members." Klemp also sees collaboration as its own reward, and he adds that "a sincere interest in the scientific goals of the organization" is likely to attract the right people. Both warn, however, that part of the reason their team works is that "those who wanted to get involved did, and the structure was imposed later," as Michalakes puts it. Efforts that are organized from the top down may meet more pitfalls, they note.
Many NCAR and UCAR groups are seeking grants to help in the transition. Proposals are due by December for NSF's Information Technology Research program, which is offering grants ranging from single-investigator projects (total budgets below $500,000 each) to group projects (up to $5 million). The topics include complex geophysical coding, data assimilation, collaboratories, and accessible visualization tools. Large institutions are eligible to submit a single proposal for a large ITR grant (total budgets up to $15 million). Killeen has asked SCD to lead the coordination of the development of such a proposal for NCAR. "It'll probably build upon the themes laid out in the HP strategic plan as well as other initiatives under way at NCAR." says SCD director Al Kellie. "Some early thoughts are that NCAR could really serve the geoscience community if we could achieve much better efficiencies for our applications on highly parallel, microprocessor-based systems. We need to crack some of the barriers that have been in the way of using these machines." He believes that one of the keys to crafting the proposal will be "to seek strong partnerships with universities and potentially other centers."
Killeen notes that NCAR will also need more flexibility within its core funding. "There definitely has to be a lot of permeability at the boundaries among the divisions, and there already is. We need structures that facilitate cross-divisional and cross-disciplinary interactions," such as those in place in NCAR's Environmental and Societal Impacts Group and Advanced Study Program.
Killeen believes that these changes come at an opportune time. "There is a special opportunity now to help define the national agenda in [our] areas [of expertise] and help define the connections between environmental science and information sciences. We have to think hard and get our story straight so we can demonstrate continued leadership."
Killeen has already established the overall theme of the upcoming strategic plan: NCAR as integrator. "Even a national center as well endowed in terms of people and materials as NCAR cannot handle it all, and shouldn't anyway," he says. "Our role is to be a player and often a leader in the development of new science, and I think that [role] requires NCAR to put together consortia and then to learn how to collaborate most efficiently with its partners."