Ranking the researchers

We live in a society where "winning is not the only thing, it is everything."

We live in a society where "winning is not the only thing, it is everything." Being ranked #1 seems to be just as important to a university physics department as it is to an NFL football team. But that's really no surprise—top-ranked university departments generally attract the best students, lure the most learned faculty, and most important, acquire the most research funding. Thus, it's easy to see how proposed assessments of research-doctorate programs grab the attention of department chairmen and professors alike.

Naturally, there is great interest among the academic community in the upcoming National Research Council (NRC) assessment of research-doctorate programs expected to occur in 2003-2004. This will be the third such NRC assessment: earlier surveys took place in 1982 and 1993 (see "Research-Doctorate Programs in the United States: Continuity and Change," http://books.nap.edu/html/researchdoc/summary.html). In the

UK, the higher-education funding organizations engage in a similar survey, called the "Research Assessment Exercise" (RAE) which takes place every four or five years (see "What is the RAE 2001?" http://www.rae.ac.uk/AboutUs/default.html). And in a more commercial vein, the magazine US News & World Report periodically issues rankings of graduate programs (for a 1999 ranking of US physics Ph.D. programs see http://www.usnews.com/usnews/edu/beyond/gradrank/gbphysic.htm). Other countries, notably Japan, are also investigating the assessment of research programs.

Rating the reputations

Not surprisingly, academics tend to sniff at the US News rankings by saying that the magazine doesn't accurately tally the opinions of real peers. Yet, US News claims that the magazine's survey results closely mimic the findings of the NRC. For the US News survey, graduate school deans were asked to rate the reputation of all the schools in their respective disciplines on a scale of 1 ("marginal") to 5 ("distinguished"). The physics Ph. D. programs at two schools (CalTech and Stanford) achieved a perfect score of 5.0, closely followed by four schools tieing with a 4.9 score for third place (Harvard, MIT, Princeton, and UC Berkeley). Intuitively, the US News rankings seem to be pretty much on target, I'd say.

In the UK, the RAE has also borne the brunt of much academic criticism, particularly from professors who argue that the survey does not take into account excellence in postgraduate teaching. However, partly in response to such arguments, the UK Higher Education Funding Council also conducts an extensive study of teaching skills, called the Teaching Quality Assessment (TQA). But no matter what the outcome of the TQA, it is the research-oriented 2001 RAE that will largely determine the distribution of around $8 billion of research funds to UK universities. Thus the highest-rated institutions get the most funding and so the system is self-perpetuating, say some critics.

Grading the teaching

The NRC surveys have also been criticized as relying too heavily on research reputations, always a very subjective attribute. Excellence in teaching has not been assessed in earlier NRC surveys nor does it appear to figure in the plans for the next study. Some critics argue that more attention should be paid to the opinions of students and recent graduates of postgraduate programs. Surveys of graduates have tended to provide results that differ considerably from surveys based on estimation of academic reputation. Clearly, teaching is a vital part of a research university and should be an integral element of the upcoming NRC assessment. High-quality teaching should be just as important a goal for any university department as excellence in research. Surely a study that is projected to cost $5 million can assess the quality of both research and teaching.

Of course, hard-pressed students in demanding doctoral programs may have a jaundiced view of their departments. Some professors would argue that a stressful postgraduate education would most likely lead to a successful research career. Such students may later change their unfavorable views of the teaching they received. Nonetheless, the NRC should turn its attention to measures of teaching excellence such as time to graduate, dropout rates, students' rankings of professors, and the starting salaries of graduates. Even subjective attributes, such as the quality of mentoring and advising, should be part of the NRC teaching assessment.

If the NRC does assess teaching excellence, there will most likely be some surprises in the next survey. That could be valuable for students and researchers alike.

Jeffrey Bairstow
Group Editorial Director
[email protected]

More in Research