theEzone : Quality

Very good or exceptional?

robyn.owens — Mon, 14 May 2007 01:35:00 GMT

Our examiners are asked to comment on the quality of the thesis, and give a rating on a 6-point score ranging from 1 = Exceptional through to 6 = Flawed. A score of 2 is Excellent ("strongly competitive at international levels, with fewer than 20% of candidates in this band") and 3 is Very Good ("An interesting, sound and compelling thesis, with approximately 30% of candidates in this band").

Here's how one examiner justified the given grading:

"To place the thesis in the range of theses I have examined, as requested by the University of WA, I rate it as a very good thesis rather than an exceptional one. The concept is simple and the experimental design, once agreed to, could be carried out by most competent researchers. The work is important for the operational field to which it speaks but the techniques are all standard and are applied in a standard way. The analysis is also competent, providing new information rather than new insights. That said, the sheer amount of work is impressive and its applied value beyond question."

That's what I want!

robyn.owens — Wed, 09 May 2007 13:44:00 GMT

Three examiners, all commenting on the same thesis:

"I recommend this thesis be passed with no requirement for any correction or amendment and that the candidate be awarded the degree of Doctor of Philosophy. The thesis as such is a substantial and original contribution to the knowledge in the field, the candidate shows excellent familiarity with the relevant literature (with an estimated number of references of more than 1,000), the techniques adapted are challenging and up to date, and the results are suitably set out and accompanied by a close to perfect exposition. Therefore, I do not hesitate to score this thesis as exceptional, at the forefront of international PhD theses in the field."

"This is a thesis rich in cutting edge technology, technical sophistication and original, publishable (indeed already published) findings."

"This is an excellent thesis comprising an impressive body of work. The thesis is extremely well written and organized. ... The introduction, in itself, demonstrates excellent scholarship and illustrates a detailed and critical understanding of the field. ... The candidate assertively emphasized the main contributions of his work while pointing out the issues that remain to be addressed. Overall, this is an excellent thesis that compiles a substantial amount of work and undoubtedly contributes to the advancement of this field."

Oh dear!

robyn.owens — Tue, 08 May 2007 05:05:00 GMT

"The thesis presented for examination is poorly prepared as evidenced by errors in figures, incorrect labels on tables and figures, no chapter titles in the table of contents, references to appendices that are not included and a significant section of chapter 5 missing. I question whether the co-supervisor has seen the thesis given that his name is incorrect. Given these omissions, it is difficult to examine the thesis and it certainly requires resubmission."

Quality Research: What is it and how is it recognized?

robyn.owens — Fri, 30 Jun 2006 08:16:00 GMT

Here are three quotes, from different examiners of different theses, all commenting on the quality of a research publication.

“This PhD work is a major achievement, as evidenced by the fact that the discovery/identification of the XXX compound was accepted for publication in Science.”

“Mr XXX’s thesis is scholarly, critical and contains an extremely thorough review of the literature of the field. It presents his own substantial contributions to his field. His results have been published in high quality journals, and he is now well known internationally as an expert within his field.”

“The work presented in the thesis contributes a substantial body of knowledge to the field as evidenced by an impressive 6 publications in good quality journals including five first author publications with a further two papers in preparation. Overall, an outstanding effort.”

The nature of quality now dominates academic discussion. Recent academic fashion has turned to the ranking of universities and academics. Vice-Chancellors now follow closely the various university ranking schemes and hotly dispute the variables that are used to determine overall place. These variables include a number of “hard” data, such as the proportion of international students in the student body, or the number of Nobel Prize winners on staff, and often softer perceptual rankings, such as how Vice-Chancellors perceive each others’ universities. Arguably, the most influential of these ranking systems is the Shanghai Jiao-Tong University Academic Ranking System http://ed.sjtu.edu.cn/ranking.htm . See also the Times Rankings http://www.thes.co.uk/worldrankings/ and the Melbourne Institute Rankings http://www.australian-universities.com/rankings/

Inevitably, ranking universities involves ranking the academics within them and the quality of the research and teaching they do.

The UK introduced their Research Assessment Exercise in 1986 and is coming up to their sixth round of assessments in 2008 http://www.rae.ac.uk/. The RAE is an exercise undertaken every 5 years on behalf of the four UK higher education funding councils to evaluate the quality of research undertaken by British higher education institutions. RAE submissions from each subject area are given a rank by a subject specialist peer review panel. The rankings are used to inform the allocation of quality weighted research funding each higher education institution receives from the national funding council.

Australia is set to introduce its own Research Quality Framework (RQF) along similar lines in 2008, with the idea that Federal Funding of Universities for research activities will be determined by these quality rankings.

There are two key variables that are commonly used to help determine the quality of an individual’s research: citations, and the impact factor of the journals in which publications appear.

A citation is logged against your name every time some other researcher cites one of your papers in their paper. Of course, not every obscure citation is captured: a major source of information and data around citations is made available through ISI - Thomson Scientific http://www.isinet.com/ who maintain up-to-date databases of citations through a master list of journals across many disciplines. You can find out who is citing your work by enquiring with the Library. Alternatively, Google Scholar (http://scholar.google.com/ ) provides an alternative free online service, but this may not be so useful in certain disciplines that have low web exposure. Google uses the Page Rank algorithm (see http://en.wikipedia.org/wiki/PageRank) to provide a quality ranking of journals.

The Impact Factor of a journal is a measure of its importance. It is calculated each year by the Institute for Scientific Information for those journals which it tracks, and results are published in the Journal of Citation Reports (JCR). Impact Factors have a huge, but controversial, influence on the way published scientific research in particular is perceived and evaluated.

The impact factor is a measure of the frequency with which the "average article" in a journal has been cited in a particular year or period. The annual JCR impact factor is a ratio between citations and recent citable items published. Thus, the impact factor of a journal is calculated by dividing the number of current year citations to the source items published in that journal during the previous two years as detailed below (see http://scientific.thomson.com/free/essays/journalcitationreports/impactfactor/ )

The impact factor calculation is based on a three-year period, and can be thought of as the average number of times published papers are cited up to two years after publication. For example, the 2003 impact factor for a journal would be calculated as follows:

A = the number of times articles published in 2001-2 were cited in indexed journals during 2003;

B = the number of articles, reviews, proceedings or notes published in 2001-2;

2003 Impact Factor = A/B

ISI excludes certain article types (such as news items, correspondence, and errata) from the denominator. New journals, which are indexed from their first published issue, will receive an Impact Factor after the completion of two years' indexing; in this case, the citations to the year prior to Volume 1, and the number of articles published in the year prior to Volume 1 are known zero values. Journals that are indexed starting with a volume other than the first volume will not have an Impact Factor published until three complete data-years are known.

There is a lot of debate about the usefulness of impact factors, but they still have a huge influence in discussions and evaluations of the quality of published work and the reputation of a journal.

Nature and Science are generally regarded as the most prestigious journals in science. Nature has an ISI Impact Factor of 30.98 and a Page Rank of 16.78. Science has an Impact Factor of 29.78 and a Page Rank of 16.38.

The average impact factor for various disciplines can vary widely. The practice of measuring citations and impact factors is probably best established in the medical and biological sciences, and least well established in the arts and humanities.

Publication appreciated

robyn.owens — Wed, 28 Jun 2006 01:33:00 GMT

"The author, with her supervisory team, must be congratulated on the success in getting the empirical work published as the PhD has progressed. Thus the work has received favourable peer-review and is already deemed by this process to make a significant and new contribution to the body of knowledge. This, of course, helps make my job somewhat easier and is akin to the process of attaining a PhD in Scandinavia. The adoption of this new practice with PhD students is to be applauded and promoted more widely."