Critical Knowledge Forum: Academic Responsibility, Agency and Evaluation

Registration website

How rankings are an even deeper threat to the academy than you thought possible.

Much of the discourse around Academic Freedom is quite sterile, reflecting a concern with protecting status quo behaviors and authority than supporting critique and interrogation. It may be far more interesting to consider instead the scope of Academic Responsibility. What restrictions do we willingly take on ourselves as members of the club that makes up the academy? And why? Why those restrictions in particular? From a cultural perspective, there are perhaps three central requirements that make up the core of academic identity: the requirement to write, the requirement to accept and respond to critique of our work from peers, and an expectation that we will be evaluated as scholars. The latter two are distinct, one relating to the vetting of outputs joining the formal scholarly record (acceptability), and the second relating to the importance of the scholar and their body of work (quality or excellence).

In concrete terms we can see the boundary work for these cultural identity commitments in debates such as those around whether the contributions of “technicians” (whether laboratory, computational or information/archival) are to be considered “academic”; whether preprints are to be considered to be “real” academic outputs; and who deserved the grant more. The last of these is the focus of this presentation. Most of our scholarly evaluation systems – metrics, rankings, quality measures – fail to meet even the basic methodological standards we would expect in any of the relevant disciplines, ranging from statistics to policy studies, by way of data and measurement science. Yet these issues are rarely communicated, and these failings rarely critiqued. When they are it usually within specialist communities, largely ignored by those who provide, deploy and experience these evaluations.

I will argue that if we, as academics, are to make a claim to a form of special authority as a result of the restrictions we accept on ourselves, then we must be internally consistent in the design and testing of the systems that impose those restrictions upon us. If we are unable to communicate, critique and evaluate the quality of the measurement of ourselves, and to show how it is done then we build an edifice without foundations. Academic freedom, if it is to mean anything at all, cannot be founded on appeals to the authority of old buildings, but must be built on clearly communicated evidence, supported in turn by clearly communicated, critiqued, and evaluated assessments of the quality of that evidence and how it is institutionalized. Our current systems restrict academic freedoms by limiting the range of choices we can make, defining where we publish, and the narrow means by which is evaluated. Those current systems are directly responsible for the continuing failure to address equity, diversity, and inclusion issues in the academy, and in the continuing failure of the academy to effectively address EDI in society.