Home » Miscellany » Assessing the Worth of Assessment

Assessing the Worth of Assessment

       In a lifetime as a self-appointed “essayist” I have published over a thousand articles. There are all kinds of ambiguities about that figure including revisions and syndications, but it a rough indication. Mostly they were part of continuing series and the vast majority fell “still-born from the press” to use David Hume’s rather chilling phrase. That is, they may have been read and appreciated by somebody, but there was no evidence available to me that that was the case. But occasionally there was something that put itself in for what might be called the J’Accuse award, meaning that it got a little bit down the track towards Zola’s famous article on the Dreyfus Affair which “everybody” in France read.

One of the most prominent of these was an article I wrote on what was then called the Research Assessment Exercise which was published in the Daily Telegraph in the spring of 1993. In the usual way I had no idea when it was going to appear or what it’s title would be, given that if sub-editors are employed your own title will never be used however good it is. I actually glimpsed it on the way to an airport. I couldn’t complain about its prominence as it took up a whole page including a picture of me with my feet on the desk. There was a deliberate paradox imbedded in the presentation insofar as the title was, “Sorry, I Only Teach” whereas a pile of books by me was on the desk with the author’s name clearly visible, though they weren’t all academic books.

It did occur to me that this might get me noticed, but I barely gave it another thought. I was making one of a series of visits to the State University of Tbilisi with the object of turning the Department of Scientific Communism into a politics department (though they said “political science”). Georgia was in a terrible state: the OECD figure told of a record-breaking 83% fall in the standard of living and law and order had almost completely broken down. Suffice it to say that when you are hiding under a bed listening to the gun battle going on outside and eating the Mars bars you had intended to distribute to colleagues’ children then you don’t give a lot of thought to what people might be saying about an article you wrote a couple of thousand miles away.

And there was plenty of reaction. Old fashioned mail was still the order of the day and when I got back I found close to a hundred items. If you tossed them up and let them fall into categories there was a substantial minority containing words and  phrases such as “reactionary”, “condoning failure” and “cannot be seen to represent the university”, but the majority were along the lines of, “courageous”, “honest”, “needed saying” and so on. The bit about representing the university was relevant because I was due to become acting chairman of my department. It was clear that some of my colleagues would be disturbed by having a chairman who opposed research assessment so I quickly resigned and had to forego that particular source of pleasure. Others reactions were more complex, such as those senior officials of the university who muttered, “Actually, I agree with every word you say, but I couldn’t possibly . . .etc”.

I called the RAE a “shyster’s charter” and I see no reason to change that judgment. The knowledge that your work is going to be publicly ranked is about the most distorting thing that one could invent in terms of wasting people’s time. After the RAE academics spent more time getting work out more quickly than they should have done and more work was done which was no good to anyone. Less time and effort was spent on teaching and more on networking and arranging to be cited. I remember very early on someone remarking that if you introduce the measurement of performance people will simply become very good at being measured and that proved accurately prophetic. After retiring from formal academic life as early as possible I remember asking a distinguished colleague how he was spending his time and he quickly told me (having had to make the calculation already for bureaucratic reasons) that he spent some 55% of his time in assessing his own and other people’s performance. My real sticking point was the effect it was having on teaching, including the replacement of tried and trusted methods of teaching by “innovatory” methods which invariably required less effort.

There were originally two potentially valid reasons for research assessment, only one of which turned out to be a genuinely good reason. The first was indicated in the predecessor concept of the “Research Selectivity Exercise”: if there are only the resources for (say) four state-of-the-art medical physics research laboratories then there should be competition to decide where those four should be and who should lead them. But this doesn’t remotely apply to literary theorists or theologians or a hundred other disciplines in the humanities. They need resources on only a miniscule scale. I used to enjoy the posthumous fame of Antonio Gramsci, pointing out that he produced a far better theory of political power while sitting in gaol with a pencil and a notebook than anyone with a research grant had ever produced.

So how does it look a generation later? Much has changed including names, concepts and procedures: the key concept of “impact” has complemented the monolithic dominance of the idea of pure research quality. The Stern Report, the recommendations of which will probably frame the 2020 Research Excellence Framework, acknowledges that teaching is important, that academic work can be valuable in many different ways and that not all academic staff should be research-assessed. For a time, when the RAE was introduced all of these eminently sensible propositions seemed to be denied or ignored officially. One estimate has it that ratings in 2020 will be less than 65% decided by any sort of publication. In hypothetical terms of self-interest I might have been rather pleased by the Stern report: I thought teaching was the most important thing, I had inter-disciplinary interests and I wanted to write  broadly as well as academically.

But in terms of the good of universities I remain a root and branch opponent of all forms of unnecessary assessment of academic work. (Necessary here is if a government department wants to spend £50 million on researching the options for energy policy the best experts should compete for the job; unnecessary is pretty much anything else.) In one sense this is not a fundamentalist objection because I don’t believe there is anything immoral about research assessment, but it is consequentialist and starts with what seems to me to be an immediate utilitarian perception that the offical assessment of the value of academic work is bound to do far more harm than good.

When I first heard about research assessment I thought it was barmy for the obvious reason that one’s assessment would entirely depend on whether one’s friends or enemies were doing the assessing. I thought of reviews of my own books which varied from “state of the art” to “complete rubbish”. It would depend entirely on the panel, wouldn’t it? And if it depended entirely on the panel then the whole process would be laced with bitterness and corruption, wouldn’t it? There are no underlying agreed criteria for judging intellectual  activity and the proper attitude to people you think are wrong and no use is cheerful and friendly contempt, not mutual respect. All subsequent experiences seem to have confirmed these suspicions. I have often said that the natural sciences would, of course, be different because there could be something like objective standards only to be told by natural scientists that the problem was just as acute in their fields.

The most fundamental objection to research assessment, however, is the sheer waste of human resources, of time and effort, involved. There is all the energies of highly intelligent men and women that go into trying to win a zero-sum game which is quite unnecessary. Then there is the efforts of two hundred thousand academic staff into producing research which, in most cases, is going to be read by almost nobody and have zero impact on the world which would be a better place if they concentrated on teaching – or, for that matter, if they looked after their children better or went fishing more often. The amounts of money being quoted as being distributed by research assessment – the sum is normally placed as being in the low billions – are, frankly, trivial by the standards of this appaling waste of human resources. In effect we imitated the Soviet Union at roughly the time of its demise by establishing a set of production targets for goods that nobody wants.When it comes to ideas it is only a tiny sliver of the very best that matter and, to quote Hume again, the incentive of “literary fame” is quite enough to motivate such production.

And at the risk of sounding simplistic as a person who left universities as an employee more than a dozen years ago, but who frequently returns, the whole thing seemed to make everybody unhappy. Arguably, of course, there may be many things that make academic staff less happy than they used to be or less happy than there predecessors were, but surely the stress arising out of research assessment is one of the most important. Over the years very good work has come out of calm backwaters whether they were monasteries or rural English parishes or relaxed universities. But there is a peculiar contemporary (and often American) equation which sees stress as work and work as stress. It’s as if somebody didn’t want us to be happy – and they got their way. Specifically, there was that more or less tolerable chap, Dr. Jekyll, who had the office next door. He morphed into evil Mr. Hyde who defined himself in terms of “management” and “academic leadership”. Now based in a much bigger office he devoted himself to “upping the department’s game” as if he were some sort of fantasy football manager. He certainly didn’t want people to be happy.

If I’m right about what a bad thing research assessment is then I should attempt an explanation of why governments make such bad decisions. The answer to the question surely lies with public choice theory which tells us that there is no nice sovereign or legislator making wise decisions to benefit us over time, but only self-interested persons asking themselves, “What should I be seen to be doing?” And versions of Mr. Hyde jumping on the bandwagon.

Lincoln Allison

(A version of this article was published in Times Higher Education in October 2017.)