Quality Research: What is it and how is it recognized?
Here are three quotes, from different examiners of different
theses, all commenting on the quality of a research publication.
“This PhD work is a major achievement, as evidenced by the
fact that the discovery/identification of the XXX compound was accepted for
publication in Science.”
“Mr XXX’s thesis is scholarly, critical and contains an
extremely thorough review of the literature of the field. It presents his own
substantial contributions to his field. His results have been published in high
quality journals, and he is now well known internationally as an expert within
his field.”
“The work presented in the thesis contributes a substantial
body of knowledge to the field as evidenced by an impressive 6 publications in
good quality journals including five first author publications with a further
two papers in preparation. Overall, an outstanding effort.”
The nature of quality now dominates academic discussion. Recent
academic fashion has turned to the ranking of universities and academics.
Vice-Chancellors now follow closely the various university ranking schemes and
hotly dispute the variables that are used to determine overall place. These
variables include a number of “hard” data, such as the proportion of
international students in the student body, or the number of Nobel Prize
winners on staff, and often softer perceptual rankings, such as how
Vice-Chancellors perceive each others’ universities. Arguably, the most
influential of these ranking systems is the Shanghai Jiao-Tong University Academic
Ranking System http://ed.sjtu.edu.cn/ranking.htm
. See also the Times Rankings http://www.thes.co.uk/worldrankings/
and the Melbourne Institute Rankings http://www.australian-universities.com/rankings/
Inevitably, ranking universities involves ranking the
academics within them and the quality of the research and teaching they do.
The UK
introduced their Research Assessment Exercise in 1986 and is coming up to their
sixth round of assessments in 2008 http://www.rae.ac.uk/.
The RAE is an exercise undertaken every 5 years on behalf of the four UK higher
education funding councils to evaluate the quality of research undertaken by
British higher education institutions. RAE submissions from each subject area
are given a rank by a subject specialist peer review panel. The rankings are
used to inform the allocation of quality weighted research funding each higher
education institution receives from the national funding council.
Australia
is set to introduce its own Research Quality Framework (RQF) along similar
lines in 2008, with the idea that Federal Funding of Universities for research
activities will be determined by these quality rankings.
There are two key variables that are commonly used to help
determine the quality of an individual’s research: citations, and the impact
factor of the journals in which publications appear.
A citation is
logged against your name every time some other researcher cites one of your
papers in their paper. Of course, not every obscure citation is captured: a
major source of information and data around citations is made available through
ISI - Thomson Scientific http://www.isinet.com/
who maintain up-to-date databases of citations through a master list of
journals across many disciplines. You can find out who is citing your work by
enquiring with the Library. Alternatively, Google Scholar (http://scholar.google.com/ ) provides an alternative
free online service, but this may not be so useful in certain disciplines that
have low web exposure. Google uses the Page Rank algorithm (see http://en.wikipedia.org/wiki/PageRank)
to provide a quality ranking of journals.
The Impact Factor of a journal is a measure of its
importance. It is calculated each year by the Institute for Scientific
Information for those journals which it tracks, and results are published in
the Journal of Citation Reports (JCR). Impact Factors have a huge, but
controversial, influence on the way published scientific research in particular
is perceived and evaluated.
The impact factor is a measure of the
frequency with which the "average article" in a journal has been
cited in a particular year or period. The annual JCR impact factor is a ratio
between citations and recent citable items published. Thus, the impact factor
of a journal is calculated by dividing the number of current year citations to
the source items published in that journal during the previous two years as
detailed below (see http://scientific.thomson.com/free/essays/journalcitationreports/impactfactor/
)
The impact factor calculation is based on a three-year period, and
can be thought of as the average number of times published papers are
cited up to
two years after publication. For example, the 2003 impact factor for a
journal
would be calculated as follows:
A = the number of times
articles published in 2001-2 were cited in indexed journals during 2003;
B = the number of
articles, reviews, proceedings or notes published in 2001-2;
2003 Impact Factor = A/B
ISI excludes certain article types (such as
news items, correspondence, and errata) from the denominator. New journals,
which are indexed from their first published issue, will receive an Impact
Factor after the completion of two years' indexing; in this case, the citations
to the year prior to Volume 1, and the number of articles published in the year
prior to Volume 1 are known zero values. Journals that are indexed starting
with a volume other than the first volume will not have an Impact Factor
published until three complete data-years are known.
There is a lot of debate about the usefulness of impact
factors, but they still have a huge influence in discussions and evaluations of the
quality of published work and the reputation of a journal.
Nature and Science are generally regarded as the most
prestigious journals in science. Nature has an ISI Impact Factor of 30.98 and a
Page Rank of 16.78. Science has an Impact Factor of 29.78 and a Page Rank of
16.38.
The average impact factor for various disciplines can vary widely. The practice
of measuring citations and impact factors is probably best established in the
medical and biological sciences, and least well established in the arts and
humanities.