UCAR > Communications > Staff Notes > September 1996 Search

NCAR at night:
A random walk through the Mesa Lab

After the sun goes down and most of us go home, the Mesa Lab gets quiet, but it's by no means quiescent. Animals pad through the tall grass. Security guards make their silent rounds. Computer operators keep watchful eyes on the Crays. And scientists pore over their work in savored solitude. On a Thursday evening last month, SN photographer Carlye Calvin joined me to take a first-hand look at the nocturnal side of NCAR. •Bob Henson

Lightning crackles above Boulder during a recent nighttime foray outside the Mesa Lab. (Photos by Carlye Calvin.)

9:44 p.m.

"Boy, it's dark up here," says Carlye as we hop out of her car. Indeed, it's dark--more so than you might expect. Lamps stand like sentries in the parking lot, with only a few cars and patches of dusk in between them. To the west, a gulf of blackness swallows the Flatirons. There've been thunderstorms off and on through the evening, and now a ragged, broken cloud deck sits overhead, softening the scene. The gentle drone of crickets is overlaid on the ever-so-faint hum of Boulder traffic. And a puff of cool air tells us that summer's days are numbered.

I throw on a long-sleeved shirt over my T-shirt as we stroll toward the Mesa Lab.

It Happened One Night

"There were many nights I worked late down in room 51, which is the darkest of three rooms in that office area. One night the power went off and I was the only person in that part of the basement. Fortunately, some time back I had brought a flashlight and candles to work just in case of this kind of event. I turned on the flashlight to find the candles and continued to work by candlelight for the remainder of that evening."
Karen Friedman,
SCD technical communications coordinator

9:50 p.m.

There's no denying the power of I.M. Pei's creation at any time of day, but it takes on new drama at night. Yellow light radiating from below brings out the beauty of the textured sandstone walls. The cleanness of the design registers even more strongly against the night sky than it does when the Flatirons and their rocky detail loom overhead.

As Carlye and I approach the building, we see only a few brightened windows hinting of people at work. One illuminated corner room, sandwiched between walls, stands out near the top of the A tower. Hoping to capture the contrast on film, Carlye sets up her camera and tripod in the circle drive and adjusts the exposure.

Just as she's ready to take the shot, the light blinks out. [Subsequent expletives deleted.]

10:03 p.m.

Soon enough, we're out of the darkness and swimming in a sea of lemon yellow--the refurbished cab of the A-tower elevator. We exit at the fourth floor, where we'd noticed several lights on from outside.

Strolling down the CGD hallways, we soon come upon our first subject. Even though we're brazenly dropping in on him, Gokhan (pronounced Yohkan) Danabasoglu cordially accommodates our visit.

He hails from the ancient Turkish city of Eskisehir, where "almost nothing is considered to be historic if it's not 300 or 400 years old." Gokhan came to NCAR in 1992, after completing his doctorate at the University of Colorado at Boulder. He's been doing ocean modeling with Bill Large, Jim McWilliams, Peter Gent, and Scott Doney in the CGD oceanography group. Gokhan is among the longer long-term visitors.

Although he's the only person in sight, Gokhan isn't the prototypical night owl we'd expected. "This is the first time I've been here at night," he protests. "Usually I come in about six a.m." In fact, he's planning to come back in at 6:30 the next morning. Tonight, he's on deadline, producing plots and transparencies for an upcoming meeting of the Pacific Working Group for the World Ocean Circulation Experiment.

I assumed Gokhan was taking advantage of the late-night lull in NCAR computing demand to crank out calculations. But he quickly disabused me of that notion. The after-dinner hours aren't always the best for number crunching, it seems. "A lot of people are still running jobs on the Crays late at night, and the network traffic is mostly heavy. In the early morning hours, however, the load is lighter." As it turns out, Gokhan is also doing modeling on a computer at the University of California, Los Angeles--a reminder that distributed computing goes in both directions.

Before we duck out of the fourth floor, we find another oceanographer burning 10:00 p.m. oil down the hall from Gokhan. Carlye and I clamber into Frank Bryan's office and find him gracious but a bit distracted. "I'm actually going on vacation next week," says Frank, "so I'm trying to get some last-minute stuff done." It's definitely August at NCAR. We bid Frank a prompt adieu and move on.

Left to right: Juan Carlos de Jesus Hernandez, Gilverto Chavez, Eleuterio Martinez, and Fabian Magno de Jesus.

10:24 p.m.

Working our way downward through the A tower, we find only shadows and light. Where is everyone? We decide to take a stroll southward and downward, toward the second basement. On our way to the foot of the B-tower stairwell, we hark to a cheerful and unexpected sound: whistling.

The whistler is Juan Carlos de Jesus Hernandez, one of the janitorial staff who have worked at NCAR through a contract with MasterKlean since 1995. He and three of his colleagues take a few moments out from sweeping, buffing, and trash collecting to talk with us. Two other janitors are making rounds elsewhere in the building.

It Happened One Night

"Sometimes you go into an alternate space/time. The clock starts accelerating and your focus grows increasingly concentrated till you slip into a zone where hours pass without notice and your productivity increases by orders of magnitude. Everything is hushed and dark, like an observation deck on the Starship Enterprise, and you can listen to "Music from the Hearts of Space" and "Tubular Bells." The halls echo like an empty cathedral, and the supercomputing room looks like a scene from the TV program Babylon 5.
Lynda Lester,
SCD writer/editor

Juan, the most proficient of the group in English, is the Mesa Lab supervisor and the unofficial spokesperson for this interview. "I've been working for this company for about a year. I've been at NCAR for eight months," he says.

The janitorial shifts at NCAR run from 5:00 p.m to 1:00 a.m. "In the morning, I work at Hardee's during breakfast and lunch, from about 7:00 a.m. to 3:00 or 4:00 p.m. There I do prep for salads and sandwiches.

"All of us work at least two jobs," Juan adds.

I try to picture these six people cooking or cleaning elsewhere through the day, then putting the entire Mesa Lab in order for the next day, then grabbing perhaps four or five hours of sleep before the cycle repeats itself. I'm impressed that Juan and his colleagues manage to exude such equanimity. His was the only whistling I heard this evening.

10:43 p.m.

We stop in the tree plaza and take in the scene. A rectangle of city-lit stratocumulus floats overhead, framed by the building. A light, mild wind carries the crickets' drone through the gaps between the towers.

It only takes a few steps at the Mesa Lab this time of night to find yourself alone, and propelled by the beauty of your surroundings into thought and reflection. I think and reflect while Carlye shoots.

This deer is one of many nonhumans who explore the mesa between sunset and sunrise.

10:51 p.m.

"If you want to know about wildlife on the mesa, just stick around here a couple of hours." So says Don Cash, one of the NCAR security guards. He and partner Ken Sprague are parked in a patrol car on the circle drive, having just finished their routine 10:00 p.m. circuit of the ML parking lot. By this time, anyone without NCAR business is supposed to be off the mesa. It's the job of Don and Ken to chase off the stragglers.

"The idea is not to give people a problem," says Don.

"We do it politely," Ken chimes in.

Don recalls a Sunday morning not long ago. "It was about 4:30 or 5:00 a.m. We had a bunch of guys who--well, let's put it this way: they had no pupils. They were climbing the walls."



Well, I reasoned, Boulder is a mecca for rock climbers. And Woody Allen did climb down the ML walls in 1973, filming Sleeper.

Most of the creatures that Don, Ken, and their coworkers encounter in the wee hours are nonhuman. "We've got two red foxes," says Don, "who'll come to you if they know you. All winter long I'd see this fox and I'd sit down and talk to him. I came in one night and heard a rattlesnake in the brush. It turned out to be that fox playing with the rattlesnake. The fox saw me and sat down. I told him, 'Bad timing! Bad timing!' "

It Happened One Night

"What I remember best is the camaraderie I've shared with the various people I worked with. We hung together, sometimes even when we were angry at each other, because we felt working this shift required something special and we had what it took."
Susan Albertson,
night-shift computer operator since 1977

For the past several years, the NCAR security staff has been subcontracted through American Patrol & Guard (APG), whose regional office is based in Loveland. A total of ten men and women work rotating shifts 24 hours a day, seven days a week--even on Christmas Day.

The security guards do more than watch for trouble at night. On weekends, for example, they serve as on-call tour guides for daytime visitors who have questions that range beyond the self-guided tour materials. On any given evening, they may be summoned after dark to escort staff or official visitors from the ML lobby to the parking lot. (This service is available to anyone at the Mesa Lab by calling or stopping by the receptionist's desk, ext. 1140. Advance notice of at least a few minutes is appreciated.)

On a typical weeknight like this one, security concerns are minor and the visitors few. However, all bets are off during full moons, big thunderstorms, and meteor showers. "That's when we get all kinds of people," says Don. He didn't see a single meteor during the Perseid shower of early August, even though he was on the mesa all night working a graveyard shift.

"I was too busy chasing people off the mesa."

Don Cash and Ken Sprague return from a patrol.

11:05 p.m.

If Don and Ken are two of the lead actors in the Mesa Lab spotlight this evening, Walt Mendenhall is the director. Walt's easygoing manner is familiar to nighttime denizens of the Mesa Lab. Under one auspice or another, he's performed security duties here for a good while. "You can say 'over ten years,' " he offers.

Walt's post is the receptionist's desk. His job as dispatch officer is to serve as the liaison between the security staff at ML, those at the Foothills Lab, and those who rove among the various UCAR-managed sites: ML, FL, UCAR North, UNAVCO, Jeffco, Marshall, and Sam's warehouse.

At Walt's disposal is a small bank of television monitors that keeps an electronic eye on the various sites. Walt can zoom in or scan the horizon at any of the sites when needed, including the ML parking lot.

There's seldom a question of who is entering or leaving a site, because the standard access cards now in use at most sites provide instant identification that Walt can access. Carlye and I ask Walt to scroll back on his site-history monitor so we can find our ML cybertrail for the evening. It's all there: each door we entered and the time we entered it. For a second I thought I saw the outline of a genial Big Brother in the phosphorescent glow.

Carlye asks if she can take Walt's photo. He balks at first--"I'm shy"--then finally relents.

Walt Mendenhall and his bevy of site-scanning monitors.

11:25 p.m.

Noise! Light! We've just entered the SCD machine room, and my senses feel overloaded by the harmonic whir of fans and the brilliance of industrial-strength fluorescent bulbs. Actually, it's not all that loud or that bright, but compared to the rest of the building, this feels like Times Square.

Jeff Cowan holds the quintessential NCAR night job: machine-room operator. Jeff's compatriot is fellow operator Greg Berman. The two have a lively, easygoing rapport marked by occasional faux sucker punches. "Jeff studies Tae Kwon Do and I can bench-press 300 pounds," says Greg, "so it'd be a pretty fair fight."

Jeff is one of the four-by-ten staff, meaning he works four ten-hour shifts per week, normally from 10:00 p.m. until 8:00 a.m. He's been at NCAR since March. Greg is a part-timer who's worked nighttime hours in the machine room since 1989. The schedule agrees with him. "My wife works days part-time and I'm here on the average 20 hours a week at night. I feel like I've got the best of all worlds." (In an endeavor completely separate from his NCAR job, Greg, who studied meteorology at Texas A&M University, writes forecasts and weather summaries for several local newspapers.)

It Happened One Night

"It was one night in 1985, around ~0430, and I had just arrived to get ready for some system test time on the CRAYs. (My recollection is that we were upgrading to Cray Operating System 1.13.) I was walking up from the bottom end of the parking lot. It was dark (it seems that there were fewer sidewalk, parking lot, and building lights then than now). Against the lit stairwell of the building, I noticed the silhouette of an animal sitting on a branch of the tree at the half-circle in the sidewalk.

"At first I thought, 'It's probably just a feral cat, but a large one.' As I got closer, it occurred to me, 'That's one heck of a big cat,' and I noticed longish, pointed ears. Just before I reached where the walks cross, the cat jumped from the tree and ran towards the Fleischmann Building through the tall grass. As it jumped, I saw a bobbed tail against the lights of the walkway. Unfortunately, it was too dark to make out much else. I decided it was either a bobtailed, 30-pound city cat or a bobcat waiting for a meteorologist meal--and all that came along was a 'Cray-on.' "

Tom Engel,
Cray site analyst at NCAR from 1983 to 1988, now an SCD software engineer

Despite the perpetual cool of the climate-controlled night, the pace in this room occasionally heats up. At any time, one of the dozens of monitor screens positioned around the room can turn magenta, a sign that one or another system needs attention. A blood-red screen denotes even more urgency. Other cues are audio; for instance, the two largest Crays emit a chirping alarm if they experience cooling problems.

The midnight hour is one of the operators' busier times, says Greg. "We'll run through all the vitals on each system. We do this at the start of each shift. Even though everything might look OK, we want to make sure that there aren't any problems lurking somewhere." The operator's domain includes seven mainframe computers (six Crays and an IBM), more than a dozen distributed systems, and thousands of cartridges and disks storing well over 75 terabytes of data.

Power outages are among the specters these operators face. Jeff was on hand during one of this summer's widespread blackouts. "I saw the lights flash two or three times, then the whole room went dark." A battery-based uninterruptable power supply (UPS) kicks in at these points to keep the room safe for the temperature-sensitive computers. The UPS backup provides a 15-minute window for graceful shutdown of the computers should the power glitch be prolonged.

Jeff Cowan and Greg Berman.
Nighttime work has its definite disadvantages for the operators. They are among the few direct employees of UCAR who seldom spend time at work during regular business hours. "There are a lot of meetings and seminars that I don't get to attend," says Jeff. "There's a kind of disembodiment from the rest of the NCAR world." Bulletin boards, e-mail, and newsletters help the operators to maintain a connection with the nine-to-fivers, and there are flextime arrangements that allow the operators to attend daytime seminars on work time as their sleep patterns allow.

On the plus side of the ledger, the machine room's pace is generally low-key while the rest of the world sleeps. "There are definitely nights that are just hell nights," says Jeff, "but on the whole, the job is probably 80% routine, 15% not so routine, and 5% sheer panic."

After a pause, he reconsiders. "Make that 19% not so routine and 1% sheer panic."

Greg adds the ultimate spin control: "It's not like we're panicking--it's more like the room is panicking." •

In this issue...

Other issues of Staff Notes


Prepared by Jacque Marshall, jacque@ucar.edu, 303-497-8616