President's Corner

The allocation analysis:   A "roughly right" look at UCAR's level of effort for science and the community

"It is better to be roughly right than precisely irrelevant."
--John Lumsden, director of MetService, New Zealand's weather service

UCAR carries out many scientific, technological and educational activities through NCAR and UOP. I am sometimes asked such questions as:

What does UCAR do with its resources?

How much do our programs interact with and support the community?

What fraction of our effort is collaborative with universities?

What fraction goes to technology transfer or to education and outreach?  

The answers to these questions constitute part of a set of metrics used to evaluate UCAR's total portfolio of activities. In early 2004 I initiated a process to estimate, in a "roughly right" way, the level of effort the institution devoted to the four broad categories of education and outreach, community facilities, community support, and research and development. A second part of the analysis considered the level of effort going to visitors, technology transfer, and collaborations. This article summarizes the exercise and results, which have been reviewed by and discussed with the University Relations Committee and the UCAR Board of Trustees.

The two allocation matrices

I asked every NCAR division, each UOP program, and the Office of Education and Outreach (EO) to provide a "roughly right" estimate of the fraction of their total effort in each category for fiscal year 2003 (Table 1, below). This was not a detailed, audit-worthy cost-accounting exercise; I just wanted honest, good-faith estimates. We interpret "level of effort" to mean not only expenditures but also nonfiscal factors, such as volunteer (unpaid) work, in-kind contributions, provision of space for visitors, or tutoring of students and postdocs outside working hours.

I originally asked the program directors and administrators to estimate each category to the nearest 5%. However, early on, we decided to allow entries to the nearest 1% because many programs spend a small amount of effort in some of the categories (such as education and outreach). I did not expect or encourage a level of effort to be reported in every category; in fact, most divisions and programs reported zeros in most of the categories. I also realized that there would often be significant overlap between some of these categories; in such cases, I asked for an estimate of the level of activity in the predominant category.

The second part of the analysis (Table 2, below) addressed the fractional level of effort devoted to a separate set of variables: visitors to each program, technology transfer activities, and collaborations within and outside of UCAR. The first two categories are independent and nonadditive: programs were asked to estimate what fraction of their program involves visitors and what fraction is related to technology transfer. The subsections of the third category, collaborations, are additive and were designed to total 100%. The subcategory "collaboration within UCAR" means collaborations with people in any part of UCAR, including NCAR. "None" accounts for individual efforts such as independent research, single-authored publications, or book writing.

After I asked division and program directors (working closely with their administrators) to complete these matrices in draft form, the directors met with me in March, when they had a chance to explain their techniques, ask questions, and challenge each others' estimates.  The goal was to achieve some degree of uniformity in how activities were counted and to avoid any large errors or misunderstandings. Significant changes resulted, and a second round of drafts was followed by discussion at an April meeting. This time, the changes were minimal, indicating convergence (or folks getting tired of the process!).

Complete details of the level of effort analysis by NCAR division and UOP program are available as a .pdf file. (In the first table of this file, we estimate the amount spent on the different categories by multiplying the percent of effort in each category by the total fiscal-year 2004 expenditures for each division and program.)

The results

Table 1 summarizes the institutionwide allocation of effort in four broad categories for NCAR, UOP, and EO, as well as for UCAR as a whole. According to these estimates, about 10% of UCAR activities are primarily education and outreach, split evenly between external and internal efforts. The largest fraction of UCAR's total activities is in community facilities (51%); within this category, observational facilities and community models represent the largest activities. Other types of community support total 11%, while the research and development category account for 26% (fractions do not add up to 100% because of rounding).

The second part of the exercise (Table 2) indicates that about 14% of UCAR activities are devoted to visitors and 20% to technology transfer.   Again, these two categories overlap with the four categories of Table 1.  

Table 2 also shows that more than half (54%) of all UCAR activities are collaborative with the community, while a significantly smaller fraction (22%) are collaborative with other divisions and programs within UCAR. This was a somewhat surprising result to me; it suggests that individuals working within a discipline or certain type of activity tend to collaborate with others in the same discipline or activity outside of UCAR rather than with people in other disciplines or types of activity within UCAR.   A relatively small fraction (25%) of UCAR activities were regarded as noncollaborative. As in Table 1, the totals within the collaboration category may not add up to 100% because of rounding.

Some thoughts

At the start of the exercise I told our division and program directors that there were no right answers and so they should not try to game the process. Thus, I hesitate to make any judgments about whether the above balances are the appropriate ones. Different people will draw different conclusions about what these numbers should be for a national center. Still, there could have been some clearly wrong results. For example, a national center should have many interactions with and support the university community, so if the facilities, community support and collaboration fractions had been small, this would have been a concern. We learned that NCAR has significantly higher fractions of its efforts in community facilities and in research and development, while UOP has higher fractions in education and outreach and in community support. Such results might be expected from the nature of the programs in each institution.

All in all, I think the above analysis provides a useful starting point in discussing what the balances should be and in establishing a baseline to estimate possible trends over time. In keeping with Lumsden's quote above (a paraphrase from economist Maynard Keynes), I hope this analysis is "roughly right," but I am confident it is neither precise nor irrelevant.

Complete level-of-analysis effort (PDF)
Below are some highlights from the PDF file.

Table 1: Allocation matrix for education and outreach, community facilities and support, and research and development

Each subcategory links to a list of examples below this table.

Categories & Subcategories	NCAR	UOP	EO	All UCAR
Education & Outreach	6%	24%	95%	10%
Products	1%	18%	45%	5%
Other	5%	6%	50%	5%
Community Facilities	59%	31%	0%	51%
Computational	7%	0%	0%	5%
Software	3%	7%	0%	4%
Datasets	7%	12%	0%	8%
Observations	20%	10%	0%	17%
Models	22%	2%	0%	17%
Community Support	3%	42%	0%	11%
Visitors	0%	17%	0%	4%
Workshops	1%	13%	0%	3%
Field support	2%	12%	0%	4%
Research and Development	33%	3%	5%	26%
TOTAL (actual total may not equal 100% due to rounding)	100%	100%	100%	100%

 

Table 1A: What's included in Table 1 categories

The examples listed below are intended to be illustrative rather than all-inclusive.

Categories	What's included	Examples
Education and Outreach
Products	Products used by universities and others outside the Boulder area Facilities and workshops based in Boulder but used by the community	COMET modules, websites such as Windows to the Universe, slides, presentations, books, brochures, educational models COMET classroom education, ASP summer colloquia, NCAR Undergraduate Leadership Workshop > return to Table 1
Other	Other education and outreach activities, all or mostly in the Boulder area	Exhibits, lectures, seminars, tours of NCAR Educational activities involving community models, observing facilities, or data sets are included below under Community Facilities > return to Table 1
Community Facilities
Computational	Scientific Computing Division computer resources	bluesky, MSS > return to Table 1
Software	Software apart from community models	NCAR Command Language, NetCDF, software libraries, data analysis software > return to Table 1
Datasets	Archived data sets and real-time data distribution	JOSS, ATD, SCD, GPS, TOPSE, Unidata, SuomiNet > return to Table 1
Observations	Community instruments and platforms, including development, maintenance, hardware and software, and operations	Radars, aircraft, SuomiNet, HIRDLS, MOPITT, HIAPER Operational costs associated with the exclusive support of UCAR scientists are included below under R&D. > return to Table 1
Models	Staff time and the value of computer time for community models, including associated analysis, data assimilation software, and support	WRF, CCSM, MM5, MOZART, ESMF, TIME-GCM, WACCM, TUV Effort involving the use of models by UCAR scientists doing their own research is included under R&D (below) > return to Table 1
Community Support
Visitors	Community workshops other than those held in support of NCAR or UOP programs	Numerous meetings whose logistics are supported by JOSS and VSP; the annual CEDAR workshop supported by HAO. > return to Table 1
Workshops	Programs that support visitors at institutions and/or locations other than NCAR and UOP	VSP   > return to Table 1
Field support	Field project support	ATD, JOSS > return to Table 1
Research and Development R&D activities other than those already covered above

Table 2: Allocation matrix for visitors, technology transfer, and collaborations within and outside UCAR

	NCAR	UOP	EO	All UCAR
Visitors	15%	8%	40%	14%
Technology Transfer	19%	25%	0%	20%
Collaborations	73%	84%	50%	76%
Within UCAR	23%	16%	40%	22%
Outside UCAR	50%	68%	10%	54%
None	27%	16%	50%	25%
TOTAL for Collaborations (may not equal 100% due to rounding)	100%	100%	100%	100%