Google Scholar releases 2016 journal rankings, controversy ensues

On July 14, Google released the 2016 version of its Google Scholar rankings of journals.  Bryce C. Newell collected the rankings for flagship law reviews in a post on Prawfsblawg. 

Brian Leiter colorfully refers to the rankings as "BS," asserting that Google Scholar is "worthless for measuring journal impact." Leiter argues that in the ranking "there is no control for the volume of publishing by each journal, so any journal that publishes more pages and articles per year will do better than a peer journal with the same actual impact that publishes fewer articles and pages."

Leiter's arguments are (mostly) incorrect. And as my previous posts about Google Scholar were used as part of the ranking, I felt the need to respond.

Google has two measures, one that it calls the "h5-index" and one that it calls the "h5-median." The "h5-median" measure is not inflated at all by "volume of publishing," as it is the median number of citations to articles making up the "h5-index." It is on a "per-article" basis, so publishing "more articles per year" will not increase it.

The "h5-index" may increase with the volume of articles, but it increases with the volume of highly cited articles. A journal that publishes an enormous number of articles that receive few citations will never achieve a high h-index. Even if a journal published "billions and billions" of articles with a few citations each it would not have a high ranking. Moreover, I think most people would agree that the ranking of a journal should increase with the volume of highly cited articles. One could imagine a journal so elite that it only published a single highly cited article, but that journal would have limited impact and should not be ranked high.

Leiter's comment about "more pages" having an effect on the Google Scholar ranking is (mostly) unfounded. Although at least one study has suggested that more pages in an article lead to more citations, the effect is modest compared to other factors. In fact, most of the highest ranking journals overall in Google Scholar are those in the sciences and the articles in those journals are usually fewer than 20 pages. Compare that to the much lower scores of even the highest ranking law reviews, which tend to have 50, or even 100 or more pages. As an example, one of the highest cited articles ("Rivaroxaban versus Warfarin in Nonvalvular Atrial Fibrillation") in one of the highest cited journals (The New England Journal of Medicine), is only 9 pages long but has been cited 4196 times.

One could argue that the journals should be ranked by the "h5-median" (which is easy to do from the provided information) or that the h5-index should be weighted less than the h5-median, but the ranking itself is only marginally sensitive to journal volume output and the number of pages doesn't directly affect the ranking at all.

So although Google Scholar may or may not be "the best" way of ranking law journals, it's certainly not worthless. It provides valuable information alongside other measures of journal quality, and is clearly superior to some other well-known methods of citation ranking. Among other advantages, Google Scholar is not limited to the somewhat provincial Westlaw JLR database.

Comments

[Brian]

Oh boy. Life is short, so I'm going to keep this short:

1. If a journal publishes more articles, it has more chances of publishing a highly cited article (this is especially true with law reviews, where almost anyting published, e.g., in the Harvard Law Review will be read and cited subsequently). That drives up the h5-index.
2. That latter fact will affect the h5-median.

This ain't rocket science! Everyone in academic philosophy now knows this, everyone in academic law should figure it out too!

[Rob Anderson]

Yes, life is short, but if you take the time to work out the math I think you'll find that I'm right.

A low-cited journal will never get there through volume. The h-index is always less than the maximum-cited article. Assuming that citations per article follow a poisson distribution or perhaps negative binomial with a reasonable variance, that maximum does not grow very fast with the number of articles. Even as the number of articles gets very large, as long as the poisson parameter is reasonably low, the h-index will never get very big. Indeed, citation rankings that merely count the number of citations are far, far more susceptible to the effect you describe, as they can grow to infinity with no articles ever receiving many citations. They are just the sum of a bunch of poisson variables.

Now, if the argument is that everyone will always cite the Harvard Law Review even if many of the articles published are garbage (which may well be true), that's just an argument against the use of citations in general, not Google Scholar in particular.

So it's not rocket science, but it may be closer to rocket science than to "academic philosophy."

[Brian]

I never said a "low-cited journal will never get there through volume." What I said, which is still correct, is that "any journal that publishes more pages and articles per year will do better than a peer journal with the same actual impact that publishes fewer articles and pages."