I throw on a long-sleeved shirt over my T-shirt as we stroll toward the Mesa Lab.
"There were many nights I worked late down in room 51, which is the darkest of three rooms in that office area. One night the power went off and I was the only person in that part of the basement. Fortunately, some time back I had brought a flashlight and candles to work just in case of this kind of event. I turned on the flashlight to find the candles and continued to work by candlelight for the remainder of that evening."
It Happened One Night
SCD technical communications coordinator
As Carlye and I approach the building, we see only a few brightened windows hinting of people at work. One illuminated corner room, sandwiched between walls, stands out near the top of the A tower. Hoping to capture the contrast on film, Carlye sets up her camera and tripod in the circle drive and adjusts the exposure.
Just as she's ready to take the shot, the light blinks out. [Subsequent expletives deleted.]
Strolling down the CGD hallways, we soon come upon our first subject. Even though we're brazenly dropping in on him, Gokhan (pronounced Yohkan) Danabasoglu cordially accommodates our visit.
He hails from the ancient Turkish city of Eskisehir, where "almost nothing is considered to be historic if it's not 300 or 400 years old." Gokhan came to NCAR in 1992, after completing his doctorate at the University of Colorado at Boulder. He's been doing ocean modeling with Bill Large, Jim McWilliams, Peter Gent, and Scott Doney in the CGD oceanography group. Gokhan is among the longer long-term visitors.
Although he's the only person in sight, Gokhan isn't the prototypical night owl we'd expected. "This is the first time I've been here at night," he protests. "Usually I come in about six a.m." In fact, he's planning to come back in at 6:30 the next morning. Tonight, he's on deadline, producing plots and transparencies for an upcoming meeting of the Pacific Working Group for the World Ocean Circulation Experiment.
I assumed Gokhan was taking advantage of the late-night lull in NCAR computing demand to crank out calculations. But he quickly disabused me of that notion. The after-dinner hours aren't always the best for number crunching, it seems. "A lot of people are still running jobs on the Crays late at night, and the network traffic is mostly heavy. In the early morning hours, however, the load is lighter." As it turns out, Gokhan is also doing modeling on a computer at the University of California, Los Angeles--a reminder that distributed computing goes in both directions.
Before we duck out of the fourth floor, we find another oceanographer burning 10:00 p.m. oil down the hall from Gokhan. Carlye and I clamber into Frank Bryan's office and find him gracious but a bit distracted. "I'm actually going on vacation next week," says Frank, "so I'm trying to get some last-minute stuff done." It's definitely August at NCAR. We bid Frank a prompt adieu and move on.
The whistler is Juan Carlos de Jesus Hernandez, one of the janitorial staff who have worked at NCAR through a contract with MasterKlean since 1995. He and three of his colleagues take a few moments out from sweeping, buffing, and trash collecting to talk with us. Two other janitors are making rounds elsewhere in the building.
Juan, the most proficient of the group in English, is the Mesa Lab supervisor and the unofficial spokesperson for this interview. "I've been working for this company for about a year. I've been at NCAR for eight months," he says.
"Sometimes you go into an alternate space/time. The clock starts accelerating and your focus grows increasingly concentrated till you slip into a zone where hours pass without notice and your productivity increases by orders of magnitude. Everything is hushed and dark, like an observation deck on the Starship Enterprise, and you can listen to "Music from the Hearts of Space" and "Tubular Bells." The halls echo like an empty cathedral, and the supercomputing room looks like a scene from the TV program Babylon 5.
It Happened One Night
The janitorial shifts at NCAR run from 5:00 p.m to 1:00 a.m. "In the morning, I work at Hardee's during breakfast and lunch, from about 7:00 a.m. to 3:00 or 4:00 p.m. There I do prep for salads and sandwiches.
"All of us work at least two jobs," Juan adds.
I try to picture these six people cooking or cleaning elsewhere through the day, then putting the entire Mesa Lab in order for the next day, then grabbing perhaps four or five hours of sleep before the cycle repeats itself. I'm impressed that Juan and his colleagues manage to exude such equanimity. His was the only whistling I heard this evening.
It only takes a few steps at the Mesa Lab this time of night to find yourself alone, and propelled by the beauty of your surroundings into thought and reflection. I think and reflect while Carlye shoots.
"The idea is not to give people a problem," says Don.
"We do it politely," Ken chimes in.
Don recalls a Sunday morning not long ago. "It was about 4:30 or 5:00 a.m. We had a bunch of guys who--well, let's put it this way: they had no pupils. They were climbing the walls."
Well, I reasoned, Boulder is a mecca for rock climbers. And Woody Allen did climb down the ML walls in 1973, filming Sleeper.
Most of the creatures that Don, Ken, and their coworkers encounter in the wee hours are nonhuman. "We've got two red foxes," says Don, "who'll come to you if they know you. All winter long I'd see this fox and I'd sit down and talk to him. I came in one night and heard a rattlesnake in the brush. It turned out to be that fox playing with the rattlesnake. The fox saw me and sat down. I told him, 'Bad timing! Bad timing!' "
For the past several years, the NCAR security staff has been subcontracted through American Patrol & Guard (APG), whose regional office is based in Loveland. A total of ten men and women work rotating shifts 24 hours a day, seven days a week--even on Christmas Day.
"What I remember best is the camaraderie I've shared with the various people I worked with. We hung together, sometimes even when we were angry at each other, because we felt working this shift required something special and we had what it took."
It Happened One Night
night-shift computer operator since 1977
The security guards do more than watch for trouble at night. On weekends, for example, they serve as on-call tour guides for daytime visitors who have questions that range beyond the self-guided tour materials. On any given evening, they may be summoned after dark to escort staff or official visitors from the ML lobby to the parking lot. (This service is available to anyone at the Mesa Lab by calling or stopping by the receptionist's desk, ext. 1140. Advance notice of at least a few minutes is appreciated.)
On a typical weeknight like this one, security concerns are minor and the visitors few. However, all bets are off during full moons, big thunderstorms, and meteor showers. "That's when we get all kinds of people," says Don. He didn't see a single meteor during the Perseid shower of early August, even though he was on the mesa all night working a graveyard shift.
"I was too busy chasing people off the mesa."
Walt's post is the receptionist's desk. His job as dispatch officer is to serve as the liaison between the security staff at ML, those at the Foothills Lab, and those who rove among the various UCAR-managed sites: ML, FL, UCAR North, UNAVCO, Jeffco, Marshall, and Sam's warehouse.
At Walt's disposal is a small bank of television monitors that keeps an electronic eye on the various sites. Walt can zoom in or scan the horizon at any of the sites when needed, including the ML parking lot.
There's seldom a question of who is entering or leaving a site, because the standard access cards now in use at most sites provide instant identification that Walt can access. Carlye and I ask Walt to scroll back on his site-history monitor so we can find our ML cybertrail for the evening. It's all there: each door we entered and the time we entered it. For a second I thought I saw the outline of a genial Big Brother in the phosphorescent glow.
Carlye asks if she can take Walt's photo. He balks at first--"I'm shy"--then finally relents.
Jeff Cowan holds the quintessential NCAR night job: machine-room operator. Jeff's compatriot is fellow operator Greg Berman. The two have a lively, easygoing rapport marked by occasional faux sucker punches. "Jeff studies Tae Kwon Do and I can bench-press 300 pounds," says Greg, "so it'd be a pretty fair fight."
Jeff is one of the four-by-ten staff, meaning he works four ten-hour shifts per week, normally from 10:00 p.m. until 8:00 a.m. He's been at NCAR since March. Greg is a part-timer who's worked nighttime hours in the machine room since 1989. The schedule agrees with him. "My wife works days part-time and I'm here on the average 20 hours a week at night. I feel like I've got the best of all worlds." (In an endeavor completely separate from his NCAR job, Greg, who studied meteorology at Texas A&M University, writes forecasts and weather summaries for several local newspapers.)
Despite the perpetual cool of the climate-controlled night, the pace in this room occasionally heats up. At any time, one of the dozens of monitor screens positioned around the room can turn magenta, a sign that one or another system needs attention. A blood-red screen denotes even more urgency. Other cues are audio; for instance, the two largest Crays emit a chirping alarm if they experience cooling problems.
"It was one night in 1985, around ~0430, and I had just arrived to get ready for some system test time on the CRAYs. (My recollection is that we were upgrading to Cray Operating System 1.13.) I was walking up from the bottom end of the parking lot. It was dark (it seems that there were fewer sidewalk, parking lot, and building lights then than now). Against the lit stairwell of the building, I noticed the silhouette of an animal sitting on a branch of the tree at the half-circle in the sidewalk.
It Happened One Night
"At first I thought, 'It's probably just a feral cat, but a large one.' As I got closer, it occurred to me, 'That's one heck of a big cat,' and I noticed longish, pointed ears. Just before I reached where the walks cross, the cat jumped from the tree and ran towards the Fleischmann Building through the tall grass. As it jumped, I saw a bobbed tail against the lights of the walkway. Unfortunately, it was too dark to make out much else. I decided it was either a bobtailed, 30-pound city cat or a bobcat waiting for a meteorologist meal--and all that came along was a 'Cray-on.' "
Cray site analyst at NCAR from 1983 to 1988, now an SCD software engineer
The midnight hour is one of the operators' busier times, says Greg. "We'll run through all the vitals on each system. We do this at the start of each shift. Even though everything might look OK, we want to make sure that there aren't any problems lurking somewhere." The operator's domain includes seven mainframe computers (six Crays and an IBM), more than a dozen distributed systems, and thousands of cartridges and disks storing well over 75 terabytes of data.
Power outages are among the specters these operators face. Jeff was on hand during one of this summer's widespread blackouts. "I saw the lights flash two or three times, then the whole room went dark." A battery-based uninterruptable power supply (UPS) kicks in at these points to keep the room safe for the temperature-sensitive computers. The UPS backup provides a 15-minute window for graceful shutdown of the computers should the power glitch be prolonged.
On the plus side of the ledger, the machine room's pace is generally low-key while the rest of the world sleeps. "There are definitely nights that are just hell nights," says Jeff, "but on the whole, the job is probably 80% routine, 15% not so routine, and 5% sheer panic."
After a pause, he reconsiders. "Make that 19% not so routine and 1% sheer panic."
Greg adds the ultimate spin control: "It's not like we're panicking--it's more like the room is panicking."
Other issues of Staff Notes