Radar data at a new level

NOAA, Unidata, and universities team up to get high-quality radar output into researchers’ hands

by Bob Henson

It was a tantalizing situation for researchers. When NOAA finished its national network of Doppler radars (NEXRAD) in the mid-1990s, forecasters at the National Weather Service found themselves with a gold mine of data. Severe-storm warnings took a leap forward, and many other benefits ensued. But research scientists had to wait weeks to get requests for the most detailed NEXRAD data fulfilled, and the archive’s reliability was spotty at best.

Now, the same NEXRAD data used by NWS forecasters are available within minutes to researchers. This happy state of affairs is the outgrowth of a multiyear development that involved public, private, and academic sectors. On 13 April, NOAA announced a set of four high-capacity sites that will redistribute real-time data from more than 120 NEXRAD radars, all at the zippy speeds made possible by Internet2.

Three of the sites—Purdue University, the University of Oklahoma (OU), and the Education and Research Consortium of the Western Carolinas—will offer the NEXRAD Level II data to universities without restrictions and to the private sector on a cost-recovery basis. A fourth site, the NWS Telecommunication Operations Center, will also serve private users if enough demand arises.

The billion-dollar tornado outbreak of 3 May 1999 (top photo from the aftermath in Moore, Oklahoma) showed the value of real-time access to high-quality NEXRAD radar data through the Collaborative Radar Acquisition Field Test (CRAFT). Level II data on the storms across Oklahoma (below) fed into storm-scale models that projected the day’s events with unusual precision. The tape archive system for the Oklahoma City NEXRAD unit went down on 3 May. “Were it not for CRAFT, the Level II data would have been lost forever,” says OU professor Kelvin Droegemeier. (Photo by Bob Henson; radar image courtesy CRAFT.)

UCAR's Unidata program is helping the three university-based sites map out how best to route the enhanced data to academic users via Internet2. "It’s really exciting for us," says Linda Miller, external liaison for Unidata.

This marks the first time that NOAA will rely on Unidata's Local Data Manager (LDM) technology—long familiar to atmospheric science departments—to move its operational data, says Unidata director Mohan Ramamurthy. "This is a significant milestone for us and the culmination of many years of collaboration with OU and the NWS," says Ramamurthy. "It’s a great example of what can be achieved by creative partnership among academic, government, and private sector groups."

Unidata’s Linda Miller and Mohan Ramamurthy. (Photo by Carlye Calvin.)

CRAFTing a solution

In the early days of NEXRAD, a university researcher typically could gain detailed, real-time data only from the nearest radar, if at all, due to cost and logistics. Harry Edmon (University of Washington, or UW) was the first to feed NEXRAD output from a single radar into Unidata’s Internet-based LDM software as a means for handling the large data flows.

“I’d been dreaming about this step for some time,” says David Fulker, then director of Unidata, “because it demonstrated—in principle—how all universities could access all radars, overcoming the previous one-on-one limitation.” However, realization of that grand dream had to await increased Internet speeds and other developments.

At OU, Kelvin Droegemeier upped the ante, landing a grant from the Oklahoma State Regents for Higher Education to explore how multiple radars and multiple users could be addressed. "We started with inexpensive 56K lines to understand bandwidth requirements, with a view toward using bigger pipes if needed," says Droegemeier. The group also employed a radar-data interface developed by the National Severe Storms Laboratory.

Unidata and UW soon joined forces with OU to launch the Collaborative Radar Acquisition Field Test. CRAFT brought in several other universities, NOAA offices, and private firms over several years to see if Internet2 could serve a broad spectrum of clients and keep pace with data churned out by over a hundred radars operating simultaneously.

A key selling point, says Droegemeier, was to get not merely Level III data but the top-of-the-line Level II output. While both types feature the same geographical resolution, the Level III data include only the lowest four elevation angles of the radar beam. Level II includes the higher-elevation scans needed to fully profile intense thunderstorms, especially those close to a radar site.

"The kicker," says Droegemeier, is that the Level III data are "binned" or rounded off into categories (e.g., 5–10 or 10–15), while Level II data are several orders of magnitude more precise.

Those enhancements are key to storm-scale numerical prediction, which was the driving force at OU behind CRAFT, according to Droegemeier. They also make a big difference to scientists working on tough problems in hydrology and hydrometeorology.

"Having ready access to Level II data is fundamental to the research conducted by our group," says Witold Krajewski, a water resource specialist at the University of Iowa. "We need large sets of such data to develop and test algorithms for radar-based rainfall estimates and probabilistic forecasts."

Krajewski's experience with Level II data through CRAFT proves the adage of being careful what you wish for, since you might get it. "Using the Level II data requires a certain level of experience and expertise," he says. Krajewski's group and Unidata plan to collaborate on technology that could help non-experts use the Level II data more readily.

Scientists at the Naval Research Laboratory have been working with Level II data via CRAFT for over two years. "We’ve been very impressed with the method that CRAFT employs to move such a huge volume of data efficiently and reliably," says Paul Harasti, a UCAR project scientist at NRL. He calls the data "invaluable" for such projects as the lab’s real-time data fusion system and its Coupled Ocean/Atmosphere Mesoscale Prediction System.

From the private sector, Baron Services has been one of the most active players in CRAFT, says Miller. Baron started providing Level II data to TV stations in 2002. The near-instant transmission and multiple angles from Level II have allowed Baron to furnish weathercasters with a variety of dramatic 3-D portrayals of storms in action.

After real time, real archival

The improvements in access to radar data aren't limited to real time. Researchers mining the archives for Level II data from years past are having a much easier go of it these days at the National Climatic Data Center.

Before CRAFT, the center archived all Level II data on 8-millimeter tapes provided by the radar sites, says NCDC's Stephen Del Greco. More than a third of the data failed to make it from the radar into the archive. A typical user might have to wait weeks before NCDC could retrieve and deliver data from a single radar. For example, copying a 20-gigabyte data set from tape took over 200 hours of processing time. Today, the same amount of data can be retrieved in only about 10 minutes from NCDC's mass storage system, which includes the contents of all 77,000 original 8-mm tapes as well as the newer data from CRAFT.

"The amount of data being accessed directly through our Web site has gone up exponentially," says Del Greco. Last December, NCDC shipped more than a terabyte of Level II data to more than 200 clients. Over 95% of NEXRAD output now gets successfully archived. Meanwhile, OU sends real-time data to more than 30 users from more than 120 radars.

Before such access could happen on large scales, says Fulker, "The NWS had to be sure that our software was robust enough to embed in the radars themselves. They were rightly careful about placing university software in an operational context. It took a lot of testing and a lot of good work from Tim Crum [NOAA Radar Operations Center] and many people."

Helping OU, UW, and Unidata smooth the way through this and other concerns were Unidata's Policy and Users committees, along with NSF and the U.S. Weather Research Program. "We worked through the issues in a collaborative environment with the stakeholders," says Miller, "and now we've got something that works."

To learn more about Level II data access or LDM training , contact Linda Miller, 303-497-8646. Below are contacts at the four high-capacity sites for Level II access.

• OU: Kelvin Droegemeier, 405-325-0453
• Purdue: Matthew Huber 765-494-3258
• Western Carolinas: Max Lennon, 828-281-1954
• NWS: Julie Hayes 301-713-0864, ext. 120