Citation count analyses of scholarly influence are controversial in the legal academy (as in the academy more broadly). The use of citation counts has the advantage of objectivity (some might say, a veneer of objectivity), but citation counts have many potential problems. Most of the problems with citation studies relate to the specific implementation of particular methodologies.
One of the potential problems with citation counts is the "throwaway" citation problem. Often a scholar or article will receive many citations for a banal proposition of law expressed in a memorable turn of phrase. Other articles receive citations as part of an almost boilerplate string citation in the first several footnotes of an article. In such cases, the counts of citations may be inflated compared to the contribution of the article.
How would those rankings differ if we counted the number of "engagements" with a scholar's ideas, rather than one-off citations? For example, a refined methodology might down-weight or even exclude counts citations to articles in which a particular paper is cited only once.
To make this more concrete, consider the influential Sisk et al. studies of scholarly influence in the legal academy. The 2018 Sisk study uses the following Westlaw search to generate citation counts:
TE(firstname /2 lastname) and date(aft 2012) and date(bef 2018)
One possible implementation of an "engagement count" ranking could be created by supplementing the Sisk search as follows:
TE(firstname /2 lastname) and date(aft 2012) and date(bef 2018) and TE(lastname /10 "supra note")
The bold added term at the end would restrict the search counts to those in which a particular author's article was cited more than once in an article, because law reviews typically signal follow-up citations with the "supra note" Bluebook format. The citation of a particular article more than once might suggest greater engagement with the ideas of a scholar, rather than simply ticking off a citation. Of course, this will require spot checking (as does the existing Sisk methodology) to determine false positives on authors with the same last names.
Some experimentation with this approach suggests a few things:
- Among highly cited scholars, about 50% of the citations to that scholar involve "engagements" as opposed to one-off citations
- The percentage of engagements relative to total citations appears to decline as an author's total citation counts decline (more study is needed on this point)
- The engagement ranking is closely correlated with the Sisk methodology, but clearly not identical, and some scholars would move significantly between the two methodologies
I would be interested in thoughts of those interested in this subject, whether from critics of citation count methods or supporters of those methods.
Comments
You can follow this conversation by subscribing to the comment feed for this post.