 |
|
| Issue 53, November 2012 |  |
|
| Intermittent Transparency: Findings from a Freedom of Information Request into the Excellence in Research for Australia Journal Ranking Process | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| By Anna Poletti and Simon Cooper | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
© all rights reserved. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This article examines the results of a Freedom of Information (FOI) request into the journal ranking process that took place as part of the Excellence in Research for Australia (ERA) project between 2008-2010. We discuss the background to the request and the information that was received. We analyse the data, as well as the gaps in the data, relating this to the rankings process with respect to 13 journals in the Humanities and Creative Arts (HCA) cluster. We note some anomalies in the rankings of specific journals and relate this to the information about the ranking process. We conclude by commenting on the degree to which the FOI request sheds light on what was essentially a secretive process, and whether academics ought to have confidence in exercises that attempt to fix quality on such a large scale. We suggest that the FOI data demonstrates that attempts to produce and fix a quantitative measure of quality in the Humanities publishing cannot adequately account for the diversity of opinion or approaches that constitute the field.1
1. The Freedom of Information Request Most of the members of the scholarly community interested in Biography assumed this final ranking was a mistake, the result of a data entry or typographical error. However, once it became clear that no discussion would be entered into, we had to try to come to terms with how this mistake would impact life writing scholars in Australia, from senior researchers to early career researchers. In the interest of transparency, we acknowledge that the Freedom of Information request was partly motivated by self-interest in regards to the question of Biography’s ranking. One of the authors of this article had been published twice in the journal at the time the final rankings were released. However, we knew that Biography was not the only field-defining and long running international journal that had inexplicably been given a low rank. Screen, the film studies journal credited by many with being foundational to the formation of cinema studies as an academic discipline, was ranked B. To anyone with even a passing interest in the discipline this is not reflective of the status of the journal in the field. We believed that this policy of not discussing individual finalised rankings was understandable given the volume of journals the ARC had ranked, but given the importance of the rankings both in influencing future funding decisions, and their implementation in a range of bureaucratic regimes in the University sector, we nevertheless felt that such a policy needed to be challenged. Interestingly, it was this second ‘unintended use’ of the rankings—in areas such as performance management and decisions regarding academic appointments—that was cited by Senator Carr as a key reason for abandoning the ranking process (Rowbotham). The documents we asked for
Here then was the first piece of evidence, on ARC letterhead, of the size and complexity of the undertaking the ARC had set itself. The sheer volume of documents was impracticable for an FOI request or, we suspected, for a Government department to process with accuracy. This response to our first request did help us understand the types of documents available however, and so we decided to narrow our request. The volume of documents available led us to decide on requesting as much documentation as we knew existed regarding the journal rankings process in relation to 13 journals. We nominated in our revised Freedom of Information request six journals—ARIEL, a/b: Auto/Biography Studies, Biography: An Interdisciplinary Quarterly, Life Writing, Mosaic and Prose Studies—that constitute the main journals where literary studies research into life writing is published. We chose these six journals in the hope of getting some sense of how the journal rankings worked across a field of research (in this case, the field of life writing research). We also nominated pairings of journals in other fields: media studies (Media International Australia and Southern Review), feminist theory (Australian Feminist Studies and Outskirts), and cinema studies (Screen and Screening the Past). Finally we requested the documentation relating to Overland, a broad-based literary magazine that also deals with wider themes in the humanities. (Similar publications such as Griffith REVIEW, Meanjin and HEAT had been taken out in the drafting process). We were interested in how this publication’s self-description as a ‘magazine’, albeit one that contained comprehensive and scholarly material, might be dealt with in the ranking process. Throughout the process, the ARC’s in-house solicitor was helpful in assisting us understand the need to refine our request, and without their assistance (which is legislated in the Freedom of Information Act) we would not have got far at all. We also received some advice from a friend with legal training and an experienced journalist. Once our revised request was submitted, the ARC advised that the cost of processing our request would be several thousands of dollars. If we were unwilling to pay this charge we could make a submission to the ARC requesting the charge be waived because the information we were seeking was in the public interest. We made such a submission, and the application to have the charge waived was successful. The information released We received the following types of documents on the nominated journals:
The following information and documents were not released:
Reason given: ‘Provision of the information in the document would disclose the deliberations of ARC staff in relation to the candidature of people to participate as assessors in ERA ranking. Various public interest factors for and against disclosure of the document were weighed up in order to determine whether or not disclosure would be contrary to the public interest. The ARC contends that release of this material would disclose information in the nature of the deliberations and this disclosure would prejudice the effectiveness of the ERA Ranking process. Disclosure of such material was determined, on balance, not to be in the public interest and is considered exempt under s47C’. Unlike other types of information defined under the ‘Conditional Exemption’ descriptions of the Federal Freedom of Information Act, deliberative processes (47C) does not include any requirement for the agency to prove harm can result from the release of the information.2
Reason given: ‘The documents over which exemption is claimed details the steps undertaken by the ARC and its assessors in the ERA ranking process. This information is part of the ERA assessment of the journal. Release of this information would be contrary to the public interest. It is not in the public interest to release details of the specific steps taken, and the process undertaken in relation to the ranking of a Journal. … The ARC contends that release of this material would prejudice the effectiveness of the ERA Ranking process. Disclosure of such materials was determined, on balance, not to be in the public interest and is considered example under s47E’. Section 47E of the Federal Freedom of Information act permits agencies to not release information ‘where disclosure would, or could, reasonably be expected to, prejudice or have a substantial adverse effect on certain listed agency operations’.3 In this case, we infer that the ARC excludes some information on the basis that it may prejudice the ARC’s ability to undertake the ERA ranking process as instructed by the Minister. The use of these exemptions raises interesting questions given that, in this instance, a government agency is seeking to instrumentalize the intellectual practices of the research community in order to create methods for evaluating the work done by the community. In academic culture, giving others access to deliberative processes is fundamental to research practice. The research community’s ability to understand and have confidence in the ARC’s process could be limited by the ARC’s protection of its deliberative processes given its status as a government agency. This culture clash is not new, as access to deliberative processes regarding the awarding of research grants has long been protected. However, the extension of the ARC’s scope in defining research quality to include a qualitative measure of all outlets for the reporting of academic research does heighten the potential for tension between the sector and the agency regarding transparency.4
2. What the data might tell us Changes in rank across the 3 stages
of ranking: notable shifts5
Table 1: Shifts in journal rankings While some of the journals remained relatively stable in terms of their rank there were some remarkable shifts, notably Biography, Mosaic, Screen, Screening the Past and Outskirts. In the transition between the second and final stages where public submissions were taken, the most dramatic shift occurred with Biography, which dropped from the highest ranking to the lowest. For those in the academic community the effects of such changes are regarded as highly consequential. It quickly became clear that C ranking could spell the end of a journal, no matter how long-running or influential; for example, the public demise of the Australian journal People and Place (Lane). The rankings had a rapid impact upon academics, who began to make decisions about where to publish according to the journal’s rank. Thus to drop from an A* to a C in the case of Biography was potentially devastating. Such shifts created a sense of unease in the academic community. How could journals move from being regarded as ‘leaders in their field’ to the lowest quality or vice-versa in a single drafting process? We hoped that the FOI process might shed some light on how the rankings were made and altered in each phase. One factor that could contribute to the significant shift in rankings is the limits put on the number of journals which could achieve the top tier ranks. As Genoni and Haddow explain:
This raises the larger question around any attempt to fix quality on a mass scale, namely the idea of quality as a consequence of the artificial production of scarcity. The determining relationship between scarcity and quality should be interrogated in discussion of ranking processes. In not releasing information regarding its deliberative processes, the ARC leaves the sector to speculate about how the necessity for scarcity is applied in each FoR category.
3. Analysing the Data 3.1 Public Submissions
Table 2: Summary of public submissions & subsequent decisions
Is there any association between the public
submissions and the change in ranking between
the first and second drafts? In the case of two journals there was little or no correlation between the submissions and the ranking. Southern Review moved from an A to a B. This is the only overt shift downward in the ranking amongst the journals at this stage. It occurs despite 14 submissions recommending an A ranking and only 1 submission recommending a C. Screening the Past rose from a C to an A*. 3 submissions recommended an A*, 2 an A, 3 a B and 2 a C. This mixture does not reflect the change in ranking. There were some anomalies worth noting. Australian Feminist Studies moved from an A* ranking to having no ranking at all. The journal received a substantial number of submissions recommending an A* ranking but we are not given any information in the FOI documents about why the journal received no ranking in the 2nd draft stage. Outskirts was not listed at this stage in the rankings and thus had no submissions. (Interestingly this journal receives an A in the final draft.) In the absence of any other information about process or decision-making criteria we are left to speculate on the significance of the public submission process on the second round of rankings. If public submissions did have an impact (and there seems to be a correlative effect in some cases) we do not know if it was the number of submissions, the quality and content of the submission, or the author(s) of the submission that carried force. The only other evidence for decision making at this period comes from an email dated 20th February 2009 from ‘ARC-ERA journals’ to a list of recipients (redacted).6 The email is to ‘Professors’ who are members of the ‘Studies in Creative Arts and Writing Round Table’. The email mentions a list of journals that reflected the ‘suggested changes’ made by the members. We do not know anything about how the changes were made, or what guidelines were used. The email asks members to declare any conflict of interest with the journals on the list and reflects ‘I appreciate that we were doing this on the day, but I want to ensure that our records were correct’. This suggests that between stage one and two of the rankings at least some of the changes were made in a group forum and perhaps in the space of a single day. 3.2 ‘Details of Decision’ 3.3 Reviewing process instructions
We can assume from the number of these emails to reviewers, the identical instructions for the spreadsheet and the fact that it is emphasized that the ‘ARC will only be uploading these comments’ that the spreadsheet was thought to be significant in the ranking process. The FOI documents contained 25 of these spreadsheets and the table below summarizes the input of the ARC reviewers with respect to this spreadsheet. 3.4 Reviewing process: quantifiable comments on specific journals
Table 3: Numbers of reviewer comments on rankings
The first thing that stands out about each reviewer’s relation to the spreadsheet is the significant lack of comment. Seven of the journals (Biography, a/b Studies, Life Writing, Mosaic, Prose Studies, Screen and Ariel) receive no comment, indeed no input at all from the ARC reviewers. In relation to the other journals there is a stark disproportion between reviewer comment and the number of reviewers who chose not to comment. In terms of numerical data, and remembering that ARC reviewers were specifically instructed to complete the spreadsheet and that the spreadsheet would be uploaded into the ARC database (giving this a certain evidentiary force), it is significant that there is very little input by the ARC reviewers into the spreadsheet at all. Readers might speculate upon the motivations behind such minimal intervention: were reviewers reluctant to intervene at all, or were they more comfortable in other contexts (roundtables, verbal discussion etc)? Finally there is nothing in the FOI documents that would explain the dramatic shifts in ranking from the second to final stage of the process. For instance Biography, which moved from an A* to a C (the maximum range downward) or Mosaic which moved from an A to a C; in both cases there is no documented intervention from the ARC reviewers at all. Yet something caused both journals to be dramatically downgraded. Outskirts is again an anomaly. Its current ranking is given as A yet elsewhere (in publically released drafts) it is not ranked at all. 3.5 Specific Comments by Reviewers
Most of the reviewers’ comments work as assertions (‘a good journal’ or ‘parochial’) rather than justifying a change of rank through evidence. It is instructive to compare this to the categories that structured the public submissions where contributors were invited to cite more specific indicators of quality. From what evidence we have here, it seemed that ARC reviewers were not required to provide the same justification. What also seems clear is that the decision to alter journal rankings took place outside of what documented evidence we have. Despite the attempts by the ARC to create a template where reviewer decisions could be lodged, along with comments or reasons, the available evidence indicates the ranking process remained opaque and an ‘insider’ secret. A tension is created between the activities of the ARC as a government body, which can protect the efficacy of its programs by not disclosing key information about its decision making processes, and the practice of scholars as reviewers of quality in an environment where they are accountable to their peers. Such outcomes do not create confidence in a process designed to create faith through transparency. Beyond this we learn little about how ‘quality’ is determined by those involved in the ranking process. The dearth of actual comments, along with the lack of specificity revealed in the small number of actual comments we have access to, results in the notion of quality remaining shrouded in conjecture.
4. Summary of Findings
While the FOI request has provided some information with respect to the ranking process, any attempt to gain a reliable and comprehensive understanding of how and why journals were ranked across the three stages remains a frustrating experience. For an exercise aimed at producing ‘transparency’ the results are disappointingly opaque. The kinds of anomalies that prompted the initial request, such as the ranking of Biography, remain as mysterious as ever. There remain substantial gaps in the information received and where we do receive specific indications of how quality is measured (reviewer comments etc) they remain vague and ad hoc in nature. Perhaps this is reflective of the peer-review process in general, where most academics have firsthand experience of conflicting views on what constitutes quality work (Cooper and Poletti). We think, however, this raises larger questions regarding the decision to attempt such an exercise in the first place. The FOI data demonstrates that attempts to produce and fix a quantitative measure of quality in Humanities publishing cannot adequately account for the diversity of opinion and approaches that constitutes the field. The large volume of documents required in an exercise such as this, alongside the large amount of human resources in collating, judging and ranking submissions and evidence, suggests that the task set for the ARC—identifying and ranking every peer reviewed publication in the English-speaking world—can only be a partial and at times necessarily inaccurate attempt to fix the elusive issue of ‘quality’. Unlike the process of peer-review, where reviewer feedback and argument is evident in the decision process, or metrics, where at least errors and flaws in methodology can be exposed, we are still largely unsure as to what determined the ranks of individual journals in the ERA exercise. With the journal rankings now officially abandoned, there is plenty of anecdotal evidence that the lists of ranked journals are still being used as measures of performance by institutions. If this is the case the situation is made even worse by the fact that appeals to change a ranking can no longer occur. Finally, it is evident from this study that any insight as to how the rankings were arrived at remains elusive to academics and members of the public.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Anna Poletti is a lecturer in Literary Studies and co-director of the Centre for The Book, Monash University. She is the author of Intimate Ephemera: Reading Young Lives in Australian Zine Culture (MUP) and co-editor of the forthcoming collection Identity Technologies (U Wisconsin Press). Simon Cooper teaches in communications at Monash University and is an editor of Arena Journal. His publications include Technoculture & Critical Theory: In the Service of the Machine? (Routledge) and Scholars and Entrepreneurs: The Universities in Crisis (Arena Publications). |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Acknowledgements
Notes 2 See 6.59, in ‘Conditional Exemptions’ http://www.oaic.gov.au/publications/guidelines/guidelines-s93a-foi-act_part6_conditional-exemptions.html#_Toc286928230 3 ‘Certain operations of Agencies’ http://www.oaic.gov.au/publications/guidelines/guidelines-s93a-foi-act_part6_conditional-exemptions.html#_Toc286928230 4 This is reflected in the submission by the Australian Academy of Humanities to the ERA Consultation in March 2011: ’Given what is at stake, there must be transparency not only around journal rankings and other metrics, but also about how the final scores are put together‘ (5). 5 Definition of ranks:
6 p.33 of the FOI documents 7 There was no public knowledge of Outskirts' rank at stage 2. Works Cited Carr, Kim. ‘New era in research will cut the red tape.’ The Australian, 15 July 2009. http://www.theaustralian.com.au/news/new-era-in-research-will-cut-the-red-tape/story-e6frg6n6-1225750118068 Accessed 30 Oct 2012. Cooper, Simon, and Anna Poletti. ‘The New ERA of Journal Ranking: The Consequences of Australia’s Fraught Encounter with “Quality”.’ Australian Universities’ Review 53.1 (2011): 57. Genoni, Paul, and Gaby Haddow. ‘ERA and the Ranking of Australian Humanities Journals.’ Australian Humanities Review 46 (May 2009). http://www.australianhumanitiesreview.org/archive/Issue-May-2009/genoni&haddow.htm Accessed 30 Oct 2012. Lane, Bernard. ‘Global Focus Stills Vital Local Voice.’ The Australian, 2 March 2011. http://www.theaustralian.com.au/higher-education/global-focus-stills-vital-local-voice/story-e6frgcjx-1226014345226 Accessed 30 Oct 2012. Redden, Guy. ‘From RAE to ERA: Research Evaluation at Work in the Corporate University.’ Australian Humanities Review 45 (Nov 2008). http://www.australianhumanitiesreview.org/archive/Issue-November-2008/redden.html Accessed 30 Oct 2012. Rowbotham, Jill. ‘End of an ERA: Journal Rankings Dropped.’ The Australian Higher Education Supplement,30 May 2011. http://www.theaustralian.com.au/higher-education/end-of-an-era-journal-rankings-dropped/story-e6frgcjx-1226065864847 Accessed 30 Oct 2012. Young, S., D. Peetz and M. Marais. ‘The Impact of Journal Ranking Fetishism on Australian Policy-Related Research: A Case Study.’ Australian Universities’ Review 53.2 (2011): 77-87.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Please feel free to contribute to this discourse. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||