MESUR summary
Weighted Betweenness, normalized
1 0.035SCIENCE
2 0.032NATURE
3 0.020PNAS
4 0.017LNCS
5 0.006LANCET
Weighted Closeness, normalized
1 0.670SCIENCE
2 0.665NATURE
3 0.644PNAS
4 0.591LNCS
5 0.587BIOCHEM BIOPH RES CO
Two things to note: first, the “alternative” network metrics such as PageRank, closeness and betweenness centrality do pretty well. Just eyeballing their rankings it is easy to see that they may even do a better job at identifying highly popular and prestigious journals than the impact factor, e.g. Science and Nature. Second, the usage metrics do an excellent job of ranking journals according to their popularity or prestige as well. In fact, the results aren’t all that different from the citation metrics. Of course, this will always tend to be true for the top 5 journals. The interesting differences will be found in the medium to lower rankings.
Therefore, rather than eyeballing the top rankings, we can calculate the similarities between the rankings produced by a pair of metrics in terms of rank-order correlation coefficients. Here’s an example. The 
graph on the right shows the scatterplot of journal’s Impact Factor and PageRank values. The rank-order correlation is moderately positive (0.609) which means that a journals’ Impact Factor (x-axis) and PageRank values (y-axis) are correlated (one goes up when the other goes up and vice versa), but there are nevertheless significant differences. We can calculate such correlations for the rankings produced by each pair of metrics.
We calculated only 47 metrics in total; 23 for the citation graph, 23 for the usage graph, and the Impact Factor. So calculating correlation coefficients for each pair will lead to a matrix of 47 x 47 correlations (actually, 47 x 47 - 47 / 2 because they are symmetric). This matrix provides a full picture of how the rankings produced by all our citation and usage metrics relate to each other. It is sufficient information to produce a rough map like I discussed above. The map will layout the positions of each metric so that the spatial distance on the map respect the calculated correlations. Therefore metrics that express a similar aspect of “impact” will be clustered in the map, whereas those that express differing aspects of “impact” will be further apart.
The actual mathematical technique to do this is called “principal component analysis” (PCA). PCA attempts to determine a set of underlying components that best explain the variations in the similarities and dissimilarities among a set of items. The components are ranked according to how well they explain the variation in the item similarities, so when we select the 2 top ranked (hence “principal”) components we have a 2D model to most accurately maps the items according to their similarities. The result is the map shown below.
The x-axis is given by the first component, i.e. the one that explains the highest amount of variance in the metric correlations. The y-axis is given by the second component, the one that explains the second highest amount of variance. As expected, the x-axis splits the metrics results nicely into the usage (left) and citation metrics (right); it’s the most distinctive separation between the sets of metrics. The y-axis is a little more complicated because it corresponds to a secondary source of variation. The citation metrics split into three main groups. From the top: closeness, degree (with the Impact Factor) and betweenness. The latter leads to results close to Pagerank which is not all that surprising if you think about their definitions. The Impact Factor sits among the degree metrics which is also not surprising since it amounts to a normalized in-degree. The usage metrics are much less separated and seem to cluster rather strongly. Still we find a similar vertical distribution. From the top, degree and closeness, followed by PageRank and Betweenness.
The most distinctive feature of the map
PageRank
Betweenness
Impact Factor
Closeness
Usage
Citation
Degree
Closeness
Pagerank
Betweenness
Degree
The Los Alamos National Laboratory Research library was awarded funding from the Andrew W. Mellon Foundation for a two-year project on the investigation of usage-based scholarly evaluation metrics. The project’s Principal Investigator will be Johan Bollen who will conduct the work under the auspices of the Los Alamos National Laboratory Research library Digital Library Research & Prototyping led by Herbert Van de Sompel.
The introduction of digital dissemination models has introduced the need for novel means to evaluate the scholarly communication process. Usage data has attracted considerable attention since it does not suffer from publication delays and can in principle be recorded for any type of scholarly communication item. Unfortunately, the definition and validation of usage-based scholarly evaluation metrics has been problematic. Most usage data sets are recorded for the user communities of particular services, i.e. they are not representative of the scholarly community as a whole, and are insufficiently linked to other sources of information on the scholarly communication process for them to be cross-validated.
The proposed work therefore consists of the definition of a formal model of the scholarly communication process. This model semantically relates a range of bibliographic, citation and usage data. The project aims to use the model to obtain and organize data from a variety of potential sources, such as for example JSTOR, CiteSeer, HighWire and Medline. The Andrew W. Mellon foundation may play a facilitating role in the identification of appropriate data sources and the establishment of subsequent agreements. LANL will in addition leverage local data sets such as LANL’s usage data, recently acquired California State University usage data and licensed ISI data sets. Recent LANL-Ex Libris collaborations focused on the acquisition of usage data can complement these efforts. This work will result in a large-scale reference data set on which a program for the definition and validation of usage-based scholarly evaluation metrics can be conducted. The proposed work will first focus on determining the overall properties and structure of the scholarly communication process on the basis of this reference data set. The acquired information will subsequently inform the definition of a range of usage-based scholarly evaluation metrics. A program for the cross-validation of the defined metrics will then commence.
The project will benefit the scholarly community involved with usage-based evaluation metrics by the definition of a formal model of the scholarly communication process, lessons learned from the generation of a large-scale reference data set, the definition and validation of a range of usage-based scholarly evaluation metrics, the formulation of a set of guidelines on their semantics and a resulting taxonomic model of the concept of scholarly status.
