In a recent NPR piece, Nina Totenberg accused Chief Justice Roberts of “misconstruing” Census data in oral argument in a case. The comment she criticized came as part of Roberts’ questioning of Solicitor General Verrilli, in which Roberts asserted that Massachusetts “has the worst ratio of white voter turnout to African-American voter turnout” and that Mississippi has the best ratio.
In the article, Totenberg cited “Census officials” she spoke to who purportedly told her that the study Roberts cited is not reliable for inter-state comparisons such as the one made by Roberts. Specifically, Totenberg asserted that Roberts was either “wrong” or “did not look at the totality of the numbers” because he did not hedge his comments by referencing the margin of error, which was large for the data Totenberg chose to cite.
As has recently been pointed out, however, it was Totenberg who misrepresented the data, and in a way that has embarrassingly tricked some of her own fellow travelers. The margin of error for the data she cited was from a completely different survey year (2010) than that cited by the lower court (2004). Using the wrong data source, Totenberg argued that the margin of error made Roberts’ arguments “unreliable.” But such an assertion makes no sense. The fact that one study has a large margin of error (because of small sample size) does not cast doubt on statistical inferences from a different study on the same subject with a smaller margin of error (because of a large sample size).
It seems unlikely this switch of the studies was an oversight because Totenberg cited and linked to the lower court’s discussion of the data, complete with URLs. As a result of the sleight of hand of analyzing a different data set than the one used by the court, Totenberg found a larger margin of error that Roberts supposedly overlooked. She concluded that the assertion that there is a difference between the African-American turnout rate in Massachusetts and Mississippi is “simply not reliable.”
Unfortunately, the flaws in the article don’t stop with Totenberg’s misleading switch of the datasets. Indeed, almost every single thing Totenberg said in the article was either flatly incorrect or seriously misleading.
First, Totenberg’s analysis suggests that Roberts made a different claim than the one he actually made. Totenberg’s argument focuses on the margin of error for the African-American turnout rates in Mississippi and Massachusetts, concluding that they overlap and therefore inferences from them are “not reliable.” But Roberts did not say that the African-American turnout rate in Mississippi exceeded the African-American turnout rate in Massachusetts (even though it likely does). Roberts said that Massachusetts had "the worst ratio of white voter turnout to African-American voter turnout." The two claims are very different because Roberts’ claim depends on the white turnout in the two states, which is significantly higher in Massachusetts than in Mississippi. Thus, even if the African-American turnout rate were the same in both states, the ratio of white voter turnout to African-American voter turnout could be higher in Massachusetts than in Mississippi, suggesting that Mississippi did not have a disproportionately low rate of turnout among African-Americans. This fact does not support Totenberg’s argument, so she left it out.
In fact, it turns out that even the assertion Totenberg tries to shoot down (that, remember, Roberts didn’t actually make) is likely true. Indeed, the turnout percentage is higher for African-Americans in Mississippi than in Massachusetts in each of the studies (2004, 2006, 2008, and 2010). The combined sample sizes of those studies if combined into a meta-study would almost certainly support the claim Totenberg chastises Roberts for making (which again, he did not make).
Second, in addition to chiding Chief Justice Roberts for making a claim (which he did not make), which Totenberg says is false (when it is actually true) and critiquing his claim using data from 2010 (when he was using 2004 data) Totenberg makes it sound as if Roberts had made an argument about statistical significance when he didn’t. She says that Roberts is either “wrong” or “did not look at the totality of the numbers” because the difference between African-American turnout rates in Mississippi and Massachusetts (which again, is not what Roberts made an assertion about) was not statistically significant in the year 2010 (which again is not the year that Roberts or the lower court was discussing). But Roberts nowhere mentioned statistical significance; he simply said that Massachusetts had the lowest ratio and Mississippi had the highest. That is true. It’s called a point estimate, and saying that one is higher than the other does not imply that one is statistically significantly higher than the other. To the extent that Roberts could be construed as making a statement about the population from the sample, that would entail statistical inference; but of course, Totenberg doesn't both address any of these details.
Third, in addition to chiding Chief Justice Roberts for making a claim (which he did not make), which Totenberg says is false (when it is actually true), critiquing his claim using data from 2010 (when he was using 2004 data) and implying that he made assertions of statistical significance (that he did not make), she herself botches the basic arithmetic of margins of error. Totenberg says that “if you factor in the margins of error at their extremes — with Mississippi at the low end and Massachusetts at the high end — Mississippi could have had a black voter turnout rate that was 7.5 percentage points lower than Massachusetts.” But that’s not how margins of error work. You can’t simply take the low end of the higher estimate and the high end of the lower estimate and conclude that because one is larger than the other the effect is not statistically significant. It is entirely possible for two margins of error to overlap and yet to have a statistically significant difference between the estimates. Why not ask someone with a modest degree of numeracy before making an assertion like that?
I have no idea whether the Census data is reliable for the purpose for which it was apparently used by Chief Justice Roberts, nor do I know whether the 2004 or 2010 data is more relevant. Some reliable sources have suggested the data arguably should not have been used for the purpose for which Roberts cited it.
What I do know is that it was Nina Totenberg’s analysis that “misconstrued” the data, not Chief Justice Roberts’ question. The recklessness (or worse) that the “dean of the Supreme Court press corps” exhibited in criticizing the Chief Justice should be cause for concern about Totenberg’s impartiality and reliability. But, it certainly wouldn’t have been the “stupidest” remark she’s made in the last few years.I leave it to readers to interpret Totenberg’s motivation.