Canalblog
Editer l'article Suivre ce blog Administration + Créer mon blog
Formation Continue du Supérieur
7 janvier 2012

Bibliometrics and the Leiden Ranking

http://www.significancemagazine.org/SpringboardWebApp/userfiles/sig/image/Adverts/Dec2011Banner.gifBy Graham Wheeler. The field of ‘Bibliometrics’ – derived from the Greek ‘biblion’ (meaning book) and the Latin ‘metricus’ (relating to measure) – is defined as the statistical analysis of a body of literature. Although the name for the field has been around for several decades, it is only until recently that with well-managed computer databases and clever citation maps (see Figure 1), researchers are better enabled to measure the impact of academic literature that universities publish.
In December 2011, the Centre for Science and Technology Studies (CWTS) at Leiden University published figures for the 2011/2012 Leiden Ranking. The Leiden Ranking is a scoring system designed to measure the impact of academic scientific research undertaken in the world’s 500 most research-intensive universities. In addition, the Leiden Ranking also looks at the collaborative research published by several institutions and considers the networks that are formed between different universities. With numerous output measures and a vast amount of bibliometric data, this new ranking system helps to paint a more accurate picture of which universities are really making an impact on the world in terms of research output.
Methodology and Data

The researchers at CWTS used bibliometric data from over 2.5 million journal articles, letters and reviews published between 2005 and 2009. The articles et cetera studied included both English and non-English language publications, with separate analyses being undertaken on English-only and all-language publications. Only publications in the sciences and social sciences were included in the research; papers from the arts and humanities were excluded since according to the lead researchers, the bibliometric indicators obtained from the data “do not have sufficient accuracy”.
The primary indicators used to measure the impact of a university’s research include the number of publications (P), the Mean Citation Score (MCS – the average number of citations for a publication from that university), the Mean Normalized Citation Score (MNCS – the MCS adjusted for field differences, publication year and document type) and the Proportion Top 10% Publications (PPtop 10% – the proportion of publications from that university that, compared with similar publications, belong to the top 10% most frequently cited). As an example, if Princeton University had a MNCS score of 3, then on average, publications from Princeton are being cited 3 times more often than the world average.
In terms of collaborative networks, the main indicators of interest were deemed to be the proportion of publications that were collaborative works (PPcollab), the proportion of collaborative publications co-authored between two or more countries (PPint collab), the mean geographical collaboration distance (MGCD) and the proportion of collaborative publications that have a geographical distance of over 1000km between two of the universities (PP>1000km). CWTS undertook analyses where full counting and also fractional counting of collaborative publications were considered. For a hypothetical publication written by 3 scientists at ETH Zurich and 1 scientist at McGill University, under full counting both ETH’s and McGill’s publication counts (P) would increase by one. Under fractional counting, ETH’s P-number would increase by 0.75 and McGill’s P-number would increase by 0.25.
Results

An article published on the Leiden rankings by the Times Higher Education Supplement highlighted several interesting results obtained from the data. However, I wish to consider the much broader picture of what is implied by the data via graphical illustrations and some non-parametric statistical testing. Here I consider all publications (regardless of language) and assume fractional counting for collaborative papers. I have also added a new categorical variable to the data indicating the region that each university’s country belongs to.
Evident disparities are observed between countries and between geographic regions. Africa’s only entries are four universities in South Africa; Cape Town, Pretoria, Stellenbosch and Witwatersrand. All of these institutions produced less than 5,000 research publications between 2005 and 2009 in the sciences; furthermore, Cape Town University had the highest PPtop 10% value of the African entrants, with a score of 10%. As for the universities in South America (countries include Brazil (light blue), Chile (yellow) and Argentina (black)), all universities score between 4% and 6% for PPtop 10%, regardless of P. The University of Sao Paulo has a PPtop 10% of 5%, which is very similar to the other universities considered here, yet published over 17,300 papers in the 5-year period considered.
Perhaps the most interesting results come from the large clusters in the plots marked “Asia”, “Europe” and “North America”. The black dots in the “North America” plot indicate universities in the United States, the purple ones from Canada; the highest point on the y-axis corresponds to the Massachusetts Institute of Technology (MIT) and the right-most point on the x-axis is Harvard University. The “Europe” plot shows Swiss and UK institutions performing highest; the two blue points are (from left to right) the École Polytechnique Fédérale de Lausanne (EPFL) and ETH Zurich. The grey points represent UK institutions and the highest points on the y-axis include the London School of Hygiene & Tropical Medicine, Durham, Imperial College London, Cambridge and Oxford (the latter two having almost identical PPtop 10% and P values). Whilst we can identify high fliers with respect to this criterion, are the genuine differences on average between one nation’s universities and another’s?
To answer this question we may perform a statistical test called the Mann-Whitney U Test, which tests whether one of two samples of independent observations tends to have larger values than the other. This test is non-parametric, meaning that we do not need to make distributional assumptions about the data we are using; we only assume that the observations are independent of one another and that the distribution of the bibliometric indicator of interest is continuous.
Using this statistical test to compare the null hypothesis (that a bibliometric indicator is distributed identically in country A and country B) against the alternative hypothesis (that there is a difference in the median of the distribution between country A and country B’s bibliometric indicator), we find that in some cases there is evidence for genuine median differences, and in other cases, not so much evidence.
Table 1 shows the p-values (not to be confused with the number of publications, P) obtained from applying the Mann-Whitney U test to compare the PPtop 10% scores of 4 nations; China, Germany, United Kingdom and United States. The p-values in the table each represent the probability of obtaining a difference in medians of PPtop 10% scores between two nations at least as great as that observed previously, assuming that the null hypothesis is in fact true. So the smaller the p-value, then the more in favour we are of rejecting the null hypothesis. As we can see, the comparisons of China versus the United Kingdom or United States, and Germany versus the United Kingdom or United States show that there is very strong evidence against the null hypothesis; i.e. that there is a genuine difference between the medians of the distributions of PPtop 10% scores. However, when considering the universities of China versus the universities of Germany, or the universities of the United Kingdom versus those of the United States, there is little evidence against the null hypothesis. In the example of UK versus US, whilst we may observed several extreme high-achievers for the US, there is little evidence to suggest that the average (median) PPtop 10% score differs significantly between the two nation’s universities.
With the plethora of data from CWTS, one can spend a great deal of time conducting further analyses and compare universities on all sorts of grounds. Perhaps the nicest thing of all, regardless of the analysis one may conduct, is that it is quite pleasant to have an academic league table that is based purely on objective bibliometric data, rather than other tables that use heavily-subjective measures or arbitrary weighting systems for certain assessment criteria.
References

The Leiden Ranking Webpage.
Data used in the Leiden Ranking research.
"UK fails to shine in citations league" - Times Higher Education Supplement, 22nd December 2011.
Commentaires
Newsletter
49 abonnés
Visiteurs
Depuis la création 2 782 984
Formation Continue du Supérieur
Archives