Unidata, and universities team up to get high-quality radar
output into researchers’ hands
by Bob Henson
It was a tantalizing situation for researchers. When NOAA finished
its national network of Doppler radars (NEXRAD) in the mid-1990s,
forecasters at the National Weather Service found themselves with
a gold mine of data. Severe-storm warnings took a leap forward, and
many other benefits ensued. But research scientists had to wait weeks
to get requests for the most detailed NEXRAD data fulfilled, and
the archive’s reliability was spotty at best.
Now, the same NEXRAD data used by NWS forecasters are available within
minutes to researchers. This happy state of affairs is the outgrowth
of a multiyear development that involved public, private, and academic
sectors. On 13 April, NOAA announced a set of four high-capacity
sites that will redistribute real-time data from more than 120 NEXRAD
radars, all at the zippy speeds made possible by Internet2.
Three of the sites—Purdue University, the University of Oklahoma
(OU), and the Education and Research Consortium of the Western Carolinas—will
offer the NEXRAD Level II data to universities without restrictions
and to the private sector on a cost-recovery basis. A fourth site,
the NWS Telecommunication Operations Center, will also serve private
users if enough demand arises.
The billion-dollar tornado outbreak of 3 May
1999 (top photo from the aftermath in Moore, Oklahoma)
showed the value of real-time access to high-quality NEXRAD
radar data through the Collaborative Radar Acquisition Field
Test (CRAFT). Level II data on the storms across Oklahoma (below)
fed into storm-scale models that projected the day’s
events with unusual precision. The tape archive system for
the Oklahoma City NEXRAD unit went down on 3 May. “Were
it not for CRAFT, the Level II data would have been lost forever,” says
OU professor Kelvin Droegemeier. (Photo by Bob Henson; radar
image courtesy CRAFT.)
UCAR's Unidata program is helping the three university-based
sites map out how best to route the enhanced data to academic users
via Internet2. "It’s really exciting for us," says
Linda Miller, external liaison for Unidata.
This marks the first time that NOAA will rely on Unidata's
Local Data Manager (LDM) technology—long familiar to atmospheric
science departments—to move its operational data, says Unidata
director Mohan Ramamurthy. "This is a significant milestone
for us and the culmination of many years of collaboration with OU
and the NWS," says Ramamurthy. "It’s a great example
of what can be achieved by creative partnership among academic,
government, and private sector groups."
Unidata’s Linda Miller and Mohan Ramamurthy.
(Photo by Carlye Calvin.)
CRAFTing a solution
In the early days of NEXRAD, a university researcher typically could
gain detailed, real-time data only from the nearest radar, if at
all, due to cost and logistics. Harry Edmon (University of Washington,
or UW) was the first to feed NEXRAD output from a single radar into
Unidata’s Internet-based LDM software as a means for handling
“I’d been dreaming about this step for some time,” says
David Fulker, then director of Unidata, “because it demonstrated—in
principle—how all universities could access all radars, overcoming
the previous one-on-one limitation.” However, realization of
that grand dream had to await increased Internet speeds and other developments.
At OU, Kelvin Droegemeier upped the ante, landing a grant from the
Oklahoma State Regents for Higher Education to explore how multiple
radars and multiple users could be addressed. "We started with
inexpensive 56K lines to understand bandwidth requirements, with
a view toward
using bigger pipes if needed," says Droegemeier. The group
also employed a radar-data interface developed by the National Severe
Unidata and UW soon joined forces with OU to launch the Collaborative
Radar Acquisition Field Test. CRAFT brought in several other universities,
NOAA offices, and private firms over several years to see if Internet2
could serve a broad spectrum of clients and keep pace with data churned
out by over a hundred radars operating
A key selling point, says Droegemeier, was to get not merely Level
III data but the top-of-the-line Level II output. While both types
feature the same geographical resolution, the Level III data include
only the lowest four elevation angles of the radar beam. Level II
includes the higher-elevation scans needed to fully profile intense
thunderstorms, especially those close to a radar site.
"The kicker," says Droegemeier, is that the Level III data
are "binned" or rounded off into categories (e.g., 5–10
or 10–15), while Level II data are several orders of magnitude
Those enhancements are key to storm-scale numerical prediction, which
was the driving force at OU behind CRAFT, according to Droegemeier.
They also make a big difference to scientists working on tough problems
in hydrology and hydrometeorology.
"Having ready access to Level II
data is fundamental to the research conducted by our group," says
Witold Krajewski, a water resource specialist at the University of
Iowa. "We need large sets of such data to develop and test
algorithms for radar-based rainfall estimates and probabilistic forecasts."
Krajewski's experience with Level II data through CRAFT proves
the adage of being careful what you wish for, since you might get
it. "Using the Level II data requires a certain level of experience
and expertise," he says. Krajewski's group and Unidata
plan to collaborate on technology that could help non-experts use
the Level II data more readily.
Scientists at the Naval Research Laboratory have been working with
Level II data via CRAFT for over two years. "We’ve been
very impressed with the method that CRAFT employs to move such a
huge volume of data efficiently and reliably," says Paul
Harasti, a UCAR project scientist at NRL. He calls the data "invaluable"
for such projects as the lab’s real-time data fusion system
and its Coupled Ocean/Atmosphere Mesoscale Prediction System.
From the private sector, Baron Services has been one of the most
active players in CRAFT, says Miller. Baron started providing Level
II data to TV stations in 2002. The near-instant transmission and
multiple angles from Level II have allowed Baron to furnish weathercasters
with a variety of dramatic 3-D portrayals of storms in action.
After real time, real archival
The improvements in access to radar data aren't limited to
real time. Researchers mining the archives for Level II data from
years past are having a much easier go of it these days at the National
Climatic Data Center.
Before CRAFT, the center archived all Level II data on 8-millimeter
tapes provided by the radar sites, says NCDC's Stephen Del
Greco. More than a third of the data failed to make it from the radar
into the archive. A typical user might have to wait weeks before
NCDC could retrieve and deliver data from a single radar. For example,
copying a 20-gigabyte data set from tape took over 200 hours of processing
time. Today, the same amount of data can be retrieved in only about
10 minutes from NCDC's mass storage system, which includes
the contents of all 77,000 original 8-mm tapes as well as the newer
data from CRAFT.
"The amount of data being accessed directly through our Web site
has gone up exponentially," says Del Greco. Last December, NCDC
shipped more than a terabyte of Level II data to more than 200 clients.
Over 95% of NEXRAD output now gets successfully archived. Meanwhile,
OU sends real-time data to more than 30 users from more than
Before such access could happen on large scales, says Fulker, "The
NWS had to be sure that our software was robust enough to embed in
the radars themselves. They were rightly careful about placing university
software in an operational context. It took a lot of testing and
a lot of good work from Tim Crum [NOAA Radar Operations Center] and
Helping OU, UW, and Unidata smooth the way through this and other
concerns were Unidata's Policy and Users committees, along
with NSF and the U.S. Weather Research Program. "We worked
through the issues in a collaborative environment with the stakeholders,"
says Miller, "and now we've got something that works."
To learn more about Level II data access or LDM training , contact
Linda Miller, 303-497-8646.
Below are contacts at the four high-capacity sites for Level II access.
• OU: Kelvin Droegemeier, 405-325-0453
• Purdue: Matthew Huber 765-494-3258
• Western Carolinas: Max
• NWS: Julie Hayes 301-713-0864,