UCAR > Communications > UCAR Quarterly > Fall 2000 Search

Fall 2000

Real-time acquisition and archival of WSR-88D base data

by Kelvin K. Droegemeier

Kelvin Droegemeier.

Kelvin Droegemeier is a professor in the School of Meteorology at the University of Oklahoma and director of the university's Center for Analysis and Prediction of Storms. This article is a shortened version of one to appear in the Bulletin of the American Meteorological Society.

With the completed installation of of 120 National Weather Service (NWS) WSR-88D (NEXRAD) Doppler radars—and additional NEXRAD radars sponsored by the Department of Defense and Federal Aviation Administration also in place—the United States now has a unique observing system that provides nearly continuous single- Doppler radar coverage across the continental United States. The superb sensitivity and sophisticated processing algorithms of these radars, and advanced user training, have led to a substantial improvement in the identification and short-term warning of hazardous weather.

Although the NEXRAD radars were intended as real-time surveillance systems, scientists soon recognized their value for research, especially that of full-volume, full-precision base data (also known as Level II data; the number differentiates them from the unarchived raw Level I data and the Level III products known as NIDS). As an interim strategy for archiving the Level II data for researchers and other users, the NWS outfitted each radar with an 8-mm tape cartridge recording system.

Figure 1. Overlay of NEXRAD radars and Abilene universities. The average distance between a NEXRAD and the nearest Abilene university is only 56 miles (80 kilometers).

For many years, The National Climatic Data Center (NCDC) has been making base data available, via these tapes, to the national atmospheric science community. Unfortunately, this process is extremely human-resource intensive (six steps are required to process a single tape), costly (the retrieval of a single data set covering several weeks can cost thousands of dollars), slow (obtaining a single data set can take several weeks), and unreliable (the national data archival rate for NWS radars is only 65%, due in large part to the use of tape recording systems that were not designed for continuous use in the field).

To provide real-time base data for evaluation in storm-scale numerical weather prediction, and to begin addressing the problem of archiving base data over the long haul at the NCDC, the Center for Analysis and Prediction of Storms (CAPS) at the University of Oklahoma joined forces in 1998 with UCAR, the University of Washington, the National Severe Storms Laboratory (NSSL), and the WSR-88D Operational Support Facility to establish the Collaborative Radar Acquisition Field Test (CRAFT). Funded initially by a grant from the Oklahoma State Regents for Higher Education, CRAFT is an experiment in the real-time compression and Internet-based transmission of NEXRAD base data from multiple radars. The initial test bed of six radars has been delivering data continuously for over a year with virtually no outages. These radars are located at Oklahoma City and Tulsa, Oklahoma; Fort Smith, Arkansas; and Fort Worth, Lubbock, and Amarillo, Texas.

CRAFT leverages two important infrastructures to achieve low-cost, reliable transmission of base data in real time. The first is the Unidata Local Data Manager (LDM) software, created by UCAR, which runs on standard PCs or workstations. LDM is used by many universities and by several elements of the NWS to acquire meteorological data. It has the ability not only to transmit data onto the Internet from a given node, but to pass on data from other nodes as well.

A unique aspect of CRAFT is the addition to LDM of the second of these infrastructures: an off-the-shelf, loss-less data compression algorithm, BZIP2. This algorithm compresses the base data in real time down to an average of 1/12th their original size for transmission over low-bandwidth phone lines. In light of the fact that a single 5- or 6-minute volume scan never exceeds about 15 megabytes, the data compression achieved is more than adequate for a 56 kilobit/sec phone line, even in the most extreme situations (e.g., a hurricane). Indeed, the aggregate compressed base data rate for the entire national WSR-88D network is only 30- 40 megabits per second, so bandwidth is not an issue. Data decompression is performed in real time at the recipient end. In the event of communications failure, the LDM personal computer at the radar site will store and then retransmit up to four days' worth of data. This amount can be increased by increasing the capacity of the local disk.

In June, the NCDC began receiving compressed base data in real time from the 6 CRAFT radars via the commodity Internet. Recently another 6 radars were added, and now all 12 are sending data to the NCDC, where the data are directly and automatically archived on the long-term mass storage system.

Recently, CAPS, NSSL, and the NCDC were awarded a NOAA Environmental Services Data and Information Management (ESDIM) grant to expand the successful CRAFT concept for eventual application to the entire NEXRAD network. This new effort, known as CRAFT-2, takes advantage of two major national networking infrastructures, Internet2 and Abilene. Internet2 is a consortium of nearly 200 universities involved in developing new tools and applications for the Next Generation Internet, and Abilene is a high-capacity network backbone that supports these efforts.

An overlay of Internet2 universities and the NEXRAD network reveals that the average distance from any radar to the nearest Internet2 node is 56 miles (80 kilometers; see Figure 1). If a 56 kilobit/sec phone line can be run from each radar to the nearest Internet2 node via collaborative arrangements with regional networks, the CRAFT concept can be immediately reproduced, at relatively low cost, nationwide. Once the base data arrive at an Internet2 site, they can be transferred to the high-speed Abilene backbone where they can be made available to all users. To access data from a particular radar, users will simply enter the appropriate radar LDM/IP address into their local LDM system (see Figure 2).

Selected sites on Abilene, such as high-priority NOAA facilities and universities, will serve as transfer points for the entire data stream, while "satellite" nodes not linked directly will obtain base data via the commodity Internet for as many radars as available bandwidth allows. As networking capabilities in the United States continue to expand, bandwidth limitations are likely to vanish entirely.

Besides the 6 radars recently added to the original group, CAPS and the NSSL plan to add about 24 additional radars to the Internet2/Abilene infrastructure as part of CRAFT-2. The broad geographic distribution will provide an acid test of overall reliability, network efficiency, and real-time ingest at NCDC (and, eventually, NCEP). As part of this effort, NSSL and the University of Oklahoma will improve the radar data-compression algorithms to accommodate the larger data sets associated with dual polarization and more dense scanning strategies. We will also work toward implementing the LDM-based data compression and transmission capabilities in the new NEXRAD Open Systems architecture.

Real-time base data will be of rather limited value if not accompanied by suitable analysis tools. Consequently, we are starting to explore the application of data mining techniques to base data, and the creation of synthetic climatologies and other metadata sets by running storm-feature identification algorithms on the data as they arrive from the radar. We may also create one or two sites that would maintain two or three years' worth of base data online for immediate perusal and download.

In this issue... Other issues of UCAR Quarterly

Edited by Carol Rasmussen, carolr@ucar.edu
Prepared for the Web by Jacque Marshall
Last revised: Wed Dec 13 17:24:16 MST 2000