Daily Maverick readers interested in the national review of doctoral degrees may well be wondering what to make of two recent conflicting judgements on the quality of review processes being followed by the Council on Higher Education (CHE).

Concerns about the pending National Review of Doctoral Degrees were raised by me in a Daily Maverick Op-Ed (“The Council on Higher Education’s review of PhD degrees reveals worrying trends”), published on 10 February 2020. With reference to the three CHE documents central to the review, this Op-Ed concludes that the doctoral review is on course for an outcome that is likely to undermine its own noble intentions of assuring and developing quality.

In a “Right of Reply” contribution, Professor Narend Baijnath, CEO of the CHE, offers a point-by-point counter to my critique (“The Council on Higher Education’s review of doctoral qualifications: It’s the attributes that count”, 13 February 2020). This response concludes by assuring the public that “the council is confident that the forthcoming national review will be of high academic quality”. This claim rests mainly on the three key CHE national review documents having been “developed and endorsed, in the first instance, not by the CHE but by an independent group of academic experts”.

So, following the CHE’s response, should we no longer be concerned about the quality of the review documentation after all? The account that follows is based on two areas of major concern.

First, the purposefulness and usefulness of documents cannot be assured only by their respectable origins “in the first instance”. We can know if review documentation is fit for purpose only by interrogating it from the perspective of institutions and peer reviewers whose judgements will be directed and framed by it.

From this perspective, the National Review Manual and the Institutional Self-Evaluation Report (SER) Template do not pass muster. It is the standard that underpins everything: it specifies the key concepts that constitute the essence of what a doctorate is. The function of the manual and the SER template is to enable and facilitate a review process based on the standard. This entails a responsibility to guide universities in respect of the nature of evidence required under the prescribed template headings. 

Sadly, the manual and the SER template do not saddle themselves with this responsibility. They present the same headings as the standard. But by limiting their scope to the technicalities of formatting and review processes, the manual and the SER template offer no support to universities in their crucial task of packaging valid and meaningful representation of their doctorates – vis-à-vis the standard’s concepts – into the prescribed, standardised format.

Review processes are meant to be evidence-based. In the doctoral review, universities are left to fall back on their own interpretations, perhaps even best-guesses, regarding the nature of evidence required under the SER template headings.

In short, the manual and the SER template do a good job of supporting the CHE’s bureaucratic line of reporting; they do a poor job of supporting the academic project.

Second, the CHE’s “Right of Reply” contribution shows that its grasp of its own review documents is somewhat shaky.

These two uncomfortable judgements are made clearer in the two illustrative cases presented below. These cases represent the first and third of the four most important analytical sections in the SER template (the four are: Preamble and rationale; Purpose of the doctorate; Graduate attributes; Context and conditions for supervision and assessment). Here we see how CHE suppositions can differ from perfectly reasonable possible interpretations universities may make when deciding on the nature of evidence required under respective SER template headings.

Case 1. ‘The Preamble and Rationale of the Qualification Standard for Doctoral Degrees’

The op-ed critique drew attention to a human capital orientation in the standard’s “Preamble and rationale”. In this regard, there seemed to be a link with the role of the National Research Foundation (NRF), the proposer of the national review in the first place. From the manual, we know that the NRF needed assurance that the doctorate was an instrument fit for its own mandate for Human Capital Development (HCD).

In imputing a human capital orientation, the op-ed critique was certainly offering a judgement that is arguable. It’s an interpretive judgement. However, in the absence of clarity from the manual and the SER template, it is a plausible interpretation. After all, most readers of the “Preamble and rationale”, will probably find that it resonates more readily with the development of human capital than it does with the development of human beings, such as in Plato’s Academy where the search for truth was directed at the development of humankind as rational and moral beings in harmonious societies (see, for example, Murphy).

In its response, the CHE refutes a human capital orientation in the Qualification Standard. More importantly, it removes the ground for debate about whether there is, or whether there is not such an orientation: “This section of the standard is intentionally descriptive. It does not form part of the normative section of the standard.”

Well, one could be forgiven for thinking it does. While we can’t be too sure of what the CHE means by “normative” here, the “Preamble and rationale” is incontestably an integral component of the Qualification Standard. After its introduction, the standard proper begins thus:

“STANDARD FOR A DOCTORAL DEGREE

1. PREAMBLE AND RATIONALE”

Consistent with this, the first part of the SER Template to which universities must respond, begins: “With reference to the Preamble and Rationale in the Qualification Standard for Doctoral Degrees, briefly (a) describe …. (b) explain … (c) describe and evaluate …” the institution’s doctoral offerings.

The role of the NRF is also relevant to one’s interpretation of the “Preamble and rationale”. In its rebuttal of the op-ed, the CHE states that: “It is true that the NRF has a vested interest in the outcome, in terms of its own mandate, that includes value for money. The NRF was adamant from the start that the primary focus should be on the quality and ultimate value of research production, not on any extraneous factor such as quantity of produce or orchestrated utility.

“Furthermore, the NRF left the development of the standard, manual and SER template entirely in the hands of the CHE.”

This account is at odds with what the CHE told universities when advising them of the commencement of the national review in June 2019: “The necessary review instruments have been developed and approved with the support and close involvement of the National Research Foundation” (writer’s emphasis).

If one accepts the latter account of the NRF’s role, one will be more readily inclined to infer that the CHE expects doctorates to be mindful of Human Capital Development. Will writers of SERs feel that it would be strategically wise to foreground examples of doctoral studies that fall into the ambit of HCD – thereby endowing themselves with “quality”? We’ll have to wait to find out.

Case 2. Graduate Attributes

The op-ed critique pointed out a disjunction between the standard and the SER template in respect of how institutions are expected to demonstrate compliance with the standard’s specification of Graduate Attributes. The former specifies nine areas of knowledge and skills, each beginning with an achievement-based statement such as: “The graduate demonstrates …”, or, “shows evidence of …”. On the other hand, the SER template says: “Describe and evaluate how your institution prepares candidates to attain the graduate attributes…”.

Outcomes are related to but analytically distinct from the means by which they are achieved.

In justifying its position, the CHE now writes: “It is correct that the former [the standard] focuses on what the graduate can do, and the latter [the SER template] on institutional policies and procedures for ensuring the attainment of the attributes. … It was agreed that institutional policies and procedures would serve as proxy for the quality of individual graduate achievement.”

Rather than clarifying what is expected, this statement, in fact, muddies the waters still further. Most academics are likely to read “how your institution prepares candidates”, as an invitation/instruction to describe supervisory guidance and support as offered by academics, and as experienced by students in practice, not in policy and procedures. In any event, an account of policy and procedures is clearly what is expected in the next section that follows graduate attributes. This section is headed “Contexts and conditions for supervision and assessment of a doctoral qualification”. Here, in covering student progress from admission through to final assessment, the watchword in almost every one of the many bulleted points in this section is “Policies on…”.

If universities were now in a hypothetical position of being able to take account of the CHE’s “right of reply” in Daily Maverick, they would seem to have three options for representing achievement of graduate attributes: 1) as evidence-based outcomes (in line with the standards); 2) as supervision practices and support (in line with a common-sense understanding of the SER template); and 3) as enabling policies (in line with the CHE’s “Right of Reply” clarification).

Knowledge of the latter option would have greatly simplified the task of universities. It’s relatively easy to assemble a collection of policies and procedures. But it’s now too late for that because the SER template provides no hint of policy and procedures being a proxy for the achievement of graduate attributes. 

The word “policy” does not appear anywhere in the SER template’s account of graduate attributes. Indeed, the SER template suggests that the nature of the task – whatever that might actually be – is much more expansive than the mere assembling of policy documents. After requiring universities t0 describe and evaluate how they prepare candidates to attain the specified attributes, the SER template cautions: “This is a high-level question requiring a high-level response supported by adequate evidence.”

It’s perhaps a stroke of good fortune that review documentation makes no mention of CHE “agreement” (with whom?) that “institutional policies and procedures would serve as proxy for the quality of individual graduate achievement”. At best, such a proxy is a precarious measure of genuine quality. Research studies finding disjunction between policy and practice are legion.

In light of the additional layer of muddle introduced by the CHE’s account of graduate attributes, it’s ironic that the title of its “Right of Reply” carries the subtitle: “It’s the attributes that count”.

This present overview of the CHE’s rebuttal of a critique of its doctoral review documentation leads to one clear conclusion. In respect of the doctoral review, even though the CHE recruits independent academic experts, there is a compelling need for the CHE to be accountable to an external authority rather than self-referentially to itself.

We need to find answers to the question that prompted the 10 February 2020 critique in Daily Maverick: “Who will guard the guards themselves”? While this is not the place to discuss possibilities, a good start would be to look for ways of recalibrating the balance of power to manage peer review of doctoral programmes in favour of academics. DM

Ken Harley is Emeritus Professor, University of KwaZulu-Natal. He is a former Head of the School of Education and Interim Dean at the University of Natal/KwaZulu-Natal. He was editor of the Journal of Education for 12 years; manager of the CHE’s National Teacher Education Review; Team Leader of the EU-funded HIV Pilot Project in Teacher Education; and as a consultant, he has conducted 11 evaluations of funded Open Education Resource (OER) projects, mainly in East Africa and Ghana. He is co-editor of a COL/UNESCO book on OER.