MESUR documentation

Weighted Betweenness, normalized

1  0.035SCIENCE

2  0.032NATURE

3  0.020PNAS

4  0.017LNCS

5  0.006LANCET

Weighted Closeness, normalized

1  0.670SCIENCE

2  0.665NATURE

3  0.644PNAS

4  0.591LNCS

5  0.587BIOCHEM BIOPH RES CO

Two things to note: first, the “alternative” network metrics such as PageRank, closeness and betweenness centrality do pretty well. Just eyeballing their rankings it is easy to see that they may even do a better job at identifying highly popular and prestigious journals than the impact factor, e.g. Science and Nature. Second, the usage metrics do an excellent job of ranking journals according to their popularity or prestige as well. In fact, the results aren’t all that different from the citation metrics. Of course, this will always tend to be true for the top 5 journals. The interesting differences will be found in the medium to lower rankings.


Therefore, rather than eyeballing the top rankings, we can calculate the similarities between the rankings produced by a pair of metrics in terms of rank-order correlation coefficients. Here’s an example. The
graph on the right shows the scatterplot of journal’s Impact Factor and PageRank values. The rank-order correlation is moderately positive (0.609) which means that a journals’ Impact Factor (x-axis) and PageRank values (y-axis) are correlated (one goes up when the other goes up and vice versa), but there are nevertheless significant differences. We can calculate such correlations for the rankings produced by each pair of metrics.


We calculated only 47 metrics in total; 23 for the citation graph, 23 for the usage graph, and the Impact Factor. So calculating correlation coefficients for each pair will lead to  a matrix of 47 x 47 correlations (actually, 47 x 47 - 47 / 2 because they are symmetric). This matrix provides a full picture of how the rankings produced by all our citation and usage metrics relate to each other. It is sufficient information to produce a rough map like I discussed above. The map will layout the positions of each metric so that the spatial distance on the map respect the calculated correlations. Therefore metrics that express a similar aspect of “impact” will be clustered in the map, whereas those that express differing aspects of “impact” will be further apart.


The actual mathematical technique to do this is called “principal component analysis” (PCA). PCA attempts to determine a set of underlying components that best explain the variations in the similarities and dissimilarities among a set of items. The components are ranked according to how well they explain the variation in the item similarities, so when we select the 2 top ranked (hence “principal”) components we have a 2D model to most accurately maps the items according to their similarities. The result is the map shown below.
The x-axis is given by the first component, i.e. the one that explains the highest amount of variance in the metric correlations. The y-axis is given by the second component, the one that explains the second highest amount of variance. As expected, the x-axis splits the metrics results nicely into the usage (left) and citation metrics (right); it’s the most distinctive separation between the sets of metrics. The y-axis is a little more complicated because it corresponds to a secondary source of variation. The citation metrics split into three main groups. From the top: closeness, degree (with the Impact Factor) and betweenness.  The latter leads to results close to Pagerank which is not all that surprising if you think about their definitions. The Impact Factor sits among the degree metrics which is also not surprising since it amounts to a normalized in-degree. The usage metrics are much less separated and seem to cluster rather strongly. Still we find a similar vertical distribution. From the top, degree and closeness, followed by PageRank and Betweenness.


The most distinctive feature of the map

 

PageRank

Betweenness

Impact Factor

Closeness

Usage

Citation

Degree

Closeness

Pagerank

Betweenness

Degree

Lectures and slides

A list of links to recent lectures and slides that document the project:

  1. -Video of lecture at OAI5 (2007)

  2. -Plenary at NISO conference in Fall 2007.

  3. -Overview of MESUR at NISO conference in Fall 2007.

Overview papers

Two papers that provide a good overview of MESUR’s objectives and methodology:

  1. -Marko. A Rodriguez, Johan Bollen and Herbert Van de Sompel. A Practical Ontology for the Large-Scale Modeling of Scholarly Artifacts and their Usage, In Proceedings of the Joint Conference on Digital Libraries, Vancouver, June 2007.

  2. -Johan Bollen, Marko A. Rodriguez and Herbert Van de Sompel. MESUR: usage-based metrics of scholarly impact, 2007.

MESUR official summary

A short abstract of MESUR’s methods and goals.