Metrics

MESUR metrics: counts and networks.

MESUR investigates an array of possible impact metrics that includes not only frequency-based metrics (citation and hit counts), but also network-based metrics such as those employed in social network analysis and web search engines, e.g. Google’s PageRank uses the web hyperlink structure to rank web pages. To rank journals according to network-based metrics of impact, we need journal networks. Journal networks can be created by connecting individual journals on the basis of a chosen relationship.

Citation networks.

Doing so is relatively straightforward for a citation relationship as citation databases such as Thomson Scientific's Journal Citation Records list the number of citations that point from one journal to another. Each row in the list represents a connection between a given pair of journals, and the number of citations indicates the strength of their connection. A journal citation network results when all connections are taken into account. This approach has been extensively applied in other efforts to map science on the basis of journal citation data.

Usage networks.

Usage networks are created differently. Usage data is not expressed as a list of journal-to-journal connections, but as a flat, time-sequential list of article-level usage events from which journal connections must be derived. MESUR determines these journal connections using a process that is commonly employed by online recommender services such as Amazon.com and Netflix (i.e.~"you may also be interested in these items"). The assumption at the basis of these systems is that the degree to which any pair of items is related is a function of the frequency by which users jointly purchase, download, or access them. This relationship is known as usage co-occurrence, and it is used to create MESUR's journal usage networks. However, since the MESUR usage data doesn't identify individual users, usage co-occurrence was reformulated in terms of sessions, indicated by anonymized session identifiers: the degree of relationship between any pair of journals is a function of the frequency by which they are jointly accessed within user sessions. The figure above illustrates this process. Within a usage data set, usage events are grouped according to the session in which they occur. This allows determining how frequently a given pair of journals is accessed within the same session. This frequency determines the strength of the connection between this particular pair of journals. The connections thus extracted for each pair of journals can then be combined to form a journal usage network.

For the usage and citation networks in the MESUR databases, we have at this point calculated a preliminary set of different “impact” metrics as a proof of principle. We are expanding this initial set in our research program.

Some metrics are network-based, others are based on frequency. This is the list of names for the metrics we calculated organized by the general type of metric. Each metric is preceded by an abbreviation we commonly use in publication. Each metric, except the Thomson Scientific Impact Factor, was calculated for both MESUR’s usage and citation data, leading to a total of 23 usage-based metric, 23 citation-based metrics and the Thomson Scientific Impact Factor.

Degree ~ popularity indicated by number of links/connections:

1) ID: In-degree (Number of links pointing to journal)

2) WID: Weighted In-Degree (Sum of link weights pointing to journal)

3) IE: In-Degree Entropy (Entropy of link weight distribution pointing to journal)

4) OD: Out-degree (Number of links pointing from journal)

5) WOD: Weighted Out-degree (Sum of link weights pointing from journal)

6) OE: Out-degree entropy (Entropy of link weight distribution pointing from journal)

7*) IF: Journal Impact Factor (Thomson Scientific’s Impact Factor)

Shortest Path ~ network distance and “power” positions:

8) UBW: Unweighted Betweenness centrality

Frequency by which journal sits on shortest path between any pair of journal,

shortest path calculation ignore link weights

9) WBW: Weighted Betweenness centrality

Frequency by which journal sits on shortest path between any pair of journal,

shortest path calculation takes into account link weights

10) UBW-UN: Unweighted Betweenness centrality un-normalized

Same as above, but frequency un-normalized by size of connected component

11) WBW-UN: Weighted Betweenness centrality un-normalized

Same as above, but shortest path calculation takes into account link weights

12) UCL: Unweighted Closeness centrality:

Average length of shortest path between journal and all other nodes,

calculation of shortest path does not take into account link weights

13) WCL: Weighted Closeness centrality

Same as above but shortest path calculation takes into account link weights

14) UCL-UN: Unweighted Closeness Un-normalized

Same as above, but un-normalized by size of connected component

15) WCL-UN: Weighted Closeness Un-normalized

Same as above but shortest path calculation takes into account link weights

16) UNM: Unweighted Newman’s “load”

Newman’s version of betweenness, also referred to as load. See Newman (2005), A measure of betweenness centrality based on random walk. Social Networks, 27(1), 39-54.)

Shortest paths do not take into account link weights.

17) WNM: Weighted Newman’s “load”

Same as above but link weights are taken into account

18) UNM-UN: Unweighted Newman’s “load” Un-normalized

Same as above, but un-normalized by size of connected component

19) WNM-UN: Weighted Newman’s “load” Un-normalized

Same as above, but un-normalized by size of connected component

Random Walk ~ prestige given by random walk simulations on network:

20) PR: Particle Swarm’s PageRank

calculated by Marko A. Rodriguez’s Particle Swarm method

21)UPR: Unweighted PageRank

same as above, does not take into account link weights)

22) UPG: Unweighted PageRank

calculated in traditional manner, does not take into account link weights

23) WPG: Weighted PageRank

calculated in traditional manner, takes into account link weights)

24)BE: Bucket entropy

Information entropy over calculated PageRank distribution

Weighted Betweenness, normalized

1 0.035SCIENCE

2 0.032NATURE

3 0.020PNAS

4 0.017LNCS

5 0.006LANCET

Weighted Closeness, normalized

1 0.670SCIENCE

2 0.665NATURE

3 0.644PNAS

4 0.591LNCS

5 0.587BIOCHEM BIOPH RES CO

Two things to note: first, the “alternative” network metrics such as PageRank, closeness and betweenness centrality do pretty well. Just eyeballing their rankings it is easy to see that they may even do a better job at identifying highly popular and prestigious journals than the impact factor, e.g. Science and Nature. Second, the usage metrics do an excellent job of ranking journals according to their popularity or prestige as well. In fact, the results aren’t all that different from the citation metrics. Of course, this will always tend to be true for the top 5 journals. The interesting differences will be found in the medium to lower rankings.

Therefore, rather than eyeballing the top rankings, we can calculate the similarities between the rankings produced by a pair of metrics in terms of rank-order correlation coefficients. Here’s an example. The

graph on the right shows the scatterplot of journal’s Impact Factor and PageRank values. The rank-order correlation is moderately positive (0.609) which means that a journals’ Impact Factor (x-axis) and PageRank values (y-axis) are correlated (one goes up when the other goes up and vice versa), but there are nevertheless significant differences. We can calculate such correlations for the rankings produced by each pair of metrics.

We calculated only 47 metrics in total; 23 for the citation graph, 23 for the usage graph, and the Impact Factor. So calculating correlation coefficients for each pair will lead to a matrix of 47 x 47 correlations (actually, 47 x 47 - 47 / 2 because they are symmetric). This matrix provides a full picture of how the rankings produced by all our citation and usage metrics relate to each other. It is sufficient information to produce a rough map like I discussed above. The map will layout the positions of each metric so that the spatial distance on the map respect the calculated correlations. Therefore metrics that express a similar aspect of “impact” will be clustered in the map, whereas those that express differing aspects of “impact” will be further apart.

The actual mathematical technique to do this is called “principal component analysis” (PCA). PCA attempts to determine a set of underlying components that best explain the variations in the similarities and dissimilarities among a set of items. The components are ranked according to how well they explain the variation in the item similarities, so when we select the 2 top ranked (hence “principal”) components we have a 2D model to most accurately maps the items according to their similarities. The result is the map shown below.

The x-axis is given by the first component, i.e. the one that explains the highest amount of variance in the metric correlations. The y-axis is given by the second component, the one that explains the second highest amount of variance. As expected, the x-axis splits the metrics results nicely into the usage (left) and citation metrics (right); it’s the most distinctive separation between the sets of metrics. The y-axis is a little more complicated because it corresponds to a secondary source of variation. The citation metrics split into three main groups. From the top: closeness, degree (with the Impact Factor) and betweenness. The latter leads to results close to Pagerank which is not all that surprising if you think about their definitions. The Impact Factor sits among the degree metrics which is also not surprising since it amounts to a normalized in-degree. The usage metrics are much less separated and seem to cluster rather strongly. Still we find a similar vertical distribution. From the top, degree and closeness, followed by PageRank and Betweenness.

The most distinctive feature of the map