President's Corner

The allocation analysis:   A "roughly right" look at UCAR's level of effort for science and the community

"It is better to be roughly right than precisely irrelevant."
--John Lumsden, director of MetService, New Zealand's weather service

UCAR carries out many scientific, technological and educational activities through NCAR and UOP. I am sometimes asked such questions as:

What does UCAR do with its resources?

How much do our programs interact with and support the community?

What fraction of our effort is collaborative with universities?

What fraction goes to technology transfer or to education and outreach?  

The answers to these questions constitute part of a set of metrics used to evaluate UCAR's total portfolio of activities. In early 2004 I initiated a process to estimate, in a "roughly right" way, the level of effort the institution devoted to the four broad categories of education and outreach, community facilities, community support, and research and development. A second part of the analysis considered the level of effort going to visitors, technology transfer, and collaborations. This article summarizes the exercise and results, which have been reviewed by and discussed with the University Relations Committee and the UCAR Board of Trustees.

The two allocation matrices

I asked every NCAR division, each UOP program, and the Office of Education and Outreach (EO) to provide a "roughly right" estimate of the fraction of their total effort in each category for fiscal year 2003 (Table 1, below). This was not a detailed, audit-worthy cost-accounting exercise; I just wanted honest, good-faith estimates. We interpret "level of effort" to mean not only expenditures but also nonfiscal factors, such as volunteer (unpaid) work, in-kind contributions, provision of space for visitors, or tutoring of students and postdocs outside working hours.

I originally asked the program directors and administrators to estimate each category to the nearest 5%. However, early on, we decided to allow entries to the nearest 1% because many programs spend a small amount of effort in some of the categories (such as education and outreach). I did not expect or encourage a level of effort to be reported in every category; in fact, most divisions and programs reported zeros in most of the categories. I also realized that there would often be significant overlap between some of these categories; in such cases, I asked for an estimate of the level of activity in the predominant category.

The second part of the analysis (Table 2, below) addressed the fractional level of effort devoted to a separate set of variables: visitors to each program, technology transfer activities, and collaborations within and outside of UCAR. The first two categories are independent and nonadditive: programs were asked to estimate what fraction of their program involves visitors and what fraction is related to technology transfer. The subsections of the third category, collaborations, are additive and were designed to total 100%. The subcategory "collaboration within UCAR" means collaborations with people in any part of UCAR, including NCAR. "None" accounts for individual efforts such as independent research, single-authored publications, or book writing.

After I asked division and program directors (working closely with their administrators) to complete these matrices in draft form, the directors met with me in March, when they had a chance to explain their techniques, ask questions, and challenge each others' estimates.  The goal was to achieve some degree of uniformity in how activities were counted and to avoid any large errors or misunderstandings. Significant changes resulted, and a second round of drafts was followed by discussion at an April meeting. This time, the changes were minimal, indicating convergence (or folks getting tired of the process!).

Complete details of the level of effort analysis by NCAR division and UOP program are available as a .pdf file. (In the first table of this file, we estimate the amount spent on the different categories by multiplying the percent of effort in each category by the total fiscal-year 2004 expenditures for each division and program.)

The results

Table 1 summarizes the institutionwide allocation of effort in four broad categories for NCAR, UOP, and EO, as well as for UCAR as a whole. According to these estimates, about 10% of UCAR activities are primarily education and outreach, split evenly between external and internal efforts. The largest fraction of UCAR's total activities is in community facilities (51%); within this category, observational facilities and community models represent the largest activities. Other types of community support total 11%, while the research and development category account for 26% (fractions do not add up to 100% because of rounding).

The second part of the exercise (Table 2) indicates that about 14% of UCAR activities are devoted to visitors and 20% to technology transfer.   Again, these two categories overlap with the four categories of Table 1.  

Table 2 also shows that more than half (54%) of all UCAR activities are collaborative with the community, while a significantly smaller fraction (22%) are collaborative with other divisions and programs within UCAR. This was a somewhat surprising result to me; it suggests that individuals working within a discipline or certain type of activity tend to collaborate with others in the same discipline or activity outside of UCAR rather than with people in other disciplines or types of activity within UCAR.   A relatively small fraction (25%) of UCAR activities were regarded as noncollaborative. As in Table 1, the totals within the collaboration category may not add up to 100% because of rounding.

Some thoughts

At the start of the exercise I told our division and program directors that there were no right answers and so they should not try to game the process. Thus, I hesitate to make any judgments about whether the above balances are the appropriate ones. Different people will draw different conclusions about what these numbers should be for a national center. Still, there could have been some clearly wrong results. For example, a national center should have many interactions with and support the university community, so if the facilities, community support and collaboration fractions had been small, this would have been a concern. We learned that NCAR has significantly higher fractions of its efforts in community facilities and in research and development, while UOP has higher fractions in education and outreach and in community support. Such results might be expected from the nature of the programs in each institution.

All in all, I think the above analysis provides a useful starting point in discussing what the balances should be and in establishing a baseline to estimate possible trends over time. In keeping with Lumsden's quote above (a paraphrase from economist Maynard Keynes), I hope this analysis is "roughly right," but I am confident it is neither precise nor irrelevant.

Complete level-of-analysis effort (PDF)
Below are some highlights from the PDF file.

Table 1: Allocation matrix for education and outreach, community facilities and support, and research and development

Each subcategory links to a list of examples below this table.

Categories & Subcategories NCAR UOP EO All UCAR

Education & Outreach

6%
24% 95% 10%

Products         

1% 18% 45% 5%

Other

5% 6% 50% 5%

Community Facilities

59% 31% 0% 51%

Computational

7% 0% 0% 5%

Software

3% 7% 0% 4%

Datasets

7% 12% 0% 8%

Observations

20% 10% 0% 17%

Models

22% 2% 0% 17%

Community Support

3% 42% 0% 11%

Visitors

0% 17% 0% 4%

Workshops

1% 13% 0% 3%

Field support

2% 12% 0% 4%

Research and Development

33% 3% 5% 26%

TOTAL (actual total may not equal 100% due to rounding)

100% 100% 100% 100%

 

Table 1A: What's included in Table 1 categories

The examples listed below are intended to be illustrative rather than all-inclusive.

Categories What's included Examples

Education and Outreach

Products         

 

Products used by universities and others outside the Boulder area

Facilities and workshops based in Boulder but used by the community

COMET modules, websites such as Windows to the Universe, slides, presentations, books, brochures, educational models

COMET classroom education, ASP summer colloquia, NCAR Undergraduate Leadership Workshop

> return to Table 1

Other

Other education and outreach activities, all or mostly in the Boulder area

Exhibits, lectures, seminars, tours of NCAR

Educational activities involving community models, observing facilities, or data sets are included below under Community Facilities

> return to Table 1

Community Facilities

Computational
Scientific Computing Division computer resources  

bluesky, MSS

> return to Table 1

Software

Software apart from community models

NCAR Command Language, NetCDF, software libraries, data analysis software

> return to Table 1

Datasets

Archived data sets and real-time data distribution

JOSS, ATD, SCD, GPS, TOPSE, Unidata, SuomiNet

> return to Table 1

Observations

Community instruments and platforms, including development, maintenance, hardware and software, and operations

Radars, aircraft, SuomiNet, HIRDLS, MOPITT, HIAPER

Operational costs associated with the exclusive support of UCAR scientists are included below under R&D.

> return to Table 1

Models
Staff time and the value of computer time for community models, including associated analysis, data assimilation software, and support

WRF, CCSM, MM5, MOZART, ESMF, TIME-GCM, WACCM, TUV

Effort involving the use of models by UCAR scientists doing their own research is included under R&D (below)

> return to Table 1

Community Support
Visitors

Community workshops other than those held in support of NCAR or UOP programs 

Numerous meetings whose logistics are supported by JOSS and VSP; the annual CEDAR workshop supported by HAO.

> return to Table 1

Workshops

Programs that support visitors at institutions and/or locations other than NCAR and UOP

Field support
Field project support

Research and Development

R&D activities other than those already covered above


Table 2: Allocation matrix for visitors, technology transfer, and collaborations within and outside UCAR

  NCAR UOP EO All UCAR

Visitors

15% 8% 40% 14%

Technology Transfer

19% 25% 0% 20%

Collaborations

73% 84% 50% 76%

Within UCAR

23% 16% 40% 22%

Outside UCAR

50% 68% 10% 54%

None

27% 16% 50% 25%

TOTAL for Collaborations
(may not equal 100% due to rounding)

100% 100% 100% 100%
     

Also in this issue...

And then there was 3.0
What's new in CCSM 3.0

A lidar that's easy on the eyes
Forecasting for the Pentagon

Saving a department: the Arizona story

Tomorrow's fiber-optic network today: National LambdaRail

A new trajectory for COMET

Taking a closer look at present (and past) weather

Web Watch
More than a makeover: UCAR's new umbrella site
Choice images by the megabyte: UCAR's Digital Image Library

Initiatives in Brief - Biogeosciences

Governance Update - UCAR trustees go to Washington

Science Bit - What makes a model hurricane head east?

UCAR Community Calendar