LSAC fires off bizarre statement attacking Law School Transparency

Law School Transparency, an advocacy organization for the interests of law school applicants and students, recently released a report entitled 2015 State of Legal Education. The report touches on many issues related to the current law school climate, but one central focus is on explaining about the effect of declining law school LSAT scores on law graduates' prospects for passing the bar. The report attempted to draw attention to the fact that the declining quality of the law school applicant pool (as reflected by declining LSAT scores at many schools) is putting more law school graduates in jeopardy of failing the bar examination. The fact that this has been happening is not really controversial among those paying attention. 

Imagine one's surprise, therefore, when Daniel Bernstine, the president of the Law School Admissions Council (the organization responsible for the LSAT) emitted a bizarre statement attacking LST's report. The statement called three factual claims contained in the report "demonstrably false" and "patently wrong." In fact, however, the term "demonstrably false" would be more accurately attached to the LSAC's press release than to LST's relatively modest claims. Each of LST's claims was either misrepresented by Bernstine or otherwise substantially accurate. Let's look at the "claims" that LSAC attacks:

"False Claim #1: LSAT scores can be used to assign bar passage risk." In fact, the LSAT can be used to predict the overall bar passage risk of a law school's students, which is all LST claimed. LST never claimed that the LSAT could predict every individual student's chances of passing the bar, only that changes in a school's overall LSAT profile would translate to lower average bar passage chances for its graduates. LSAC's "refutation" of this claim is based solely on the fact that two schools with similar LSAT scores had different bar passage rates (57% and 23%). But differences among the difficulties of state bars are so significant that two similar schools (2 or fewer LSAT points of difference in the LST report) in states with different bar difficulties could easily have a difference in bar passage rates this large when random variation from year to year is taken into account, even holding student quality constant. Bernstine's claim shows only that he is unaware of the extent of differences among state bar exam passing scores.

"False Claim #2: LSAT scores can be used to delineate risk categories." Here Bernstine argues that the fact that LST's "categories" are separated by only one LSAT point makes them inaccurate. But that's the nature of categories-- they have boundaries. So to say that categories are meaningless because the lowest member of one category is right next to the top member of the lower one is fatuous.

"False Claim #3: A study based on 25-year-old data can be used to assess current bar passage risk." This assertion by Bernstine reveals a profound ignorance about the many, many studies of the LSAT and the bar exam. LST uses the National Longitudinal Bar Passage Study, produced under the auspices of LSAC itself, to demonstrate the relationship between the LSAT and bar passage. Because the LSAT was on a different scale (10-48) at the time of that study and is now on a 120-180 scale, Bernstine claims that "The results from this study cannot be applied directly to current LSAT scores, as claimed by LST."

This assertion by Bernstine is absurd. The LSAT is equated over time, and so is the bar exam's MBE (to which the rest of the bar exam is scaled), so the age of the study is irrelevant if LSAC and NCBE have done their jobs of equating properly. The change in grading scale is also irrelevant, as LST lined up the percentiles from the last administration of the old scale with the percentiles for the first administration of the new scale to convert the scores. So unless there was some enormous change in the quality of LSAT taker over that one year period (there wasn't), the scores are directly comparable to one another, contradicting Bernstine's statement.

The relationship between the LSAT and bar exam scores is backed up by a mountain of empirical evidence over many decades. All evidence shows that both the old LSAT and the new LSAT are highly correlated with MBE score on the bar examination, and to approximately the same degree. Indeed, LSAT scores may correlate as well with the MBE as they do with law school GPA, which is what the LSAT is specifically designed and validated to predict! An analysis by the National Conference of Bar Examiners showed a .57 correlation of LSAT with MBE score, which is comparable to the correlations LSAC finds in its own "Correlation Studies" of the LSAT with law school GPA (average of .58). And this is not a new phenomenon or otherwise a product of the "new" LSAT scale. It has been true at least since 1979, when a study for the NCBE found the correlation between LSAT and MBE of .52 for ABA-accredited schools.

It is true that the LSAT explains a relatively small amount of the total variation in the MBE, as other factors are also relevant. But the fact is that LSAT (even when coupled with undergraduate GPA) also explains a similarly small amount of the variation in law school grades, which is what the LSAT is designed to do. Thus, making the claim that the LSAT is not a valid predictor of bar passage is also a claim that the LSAT is not much of a valid predictor for its intended purpose.

The statement by Bernstine demonstrated such a fundamental misunderstanding of the technical properties of LSAC's own main product, the LSAT, that I suspect Bernstine drafted it himself. LSAC has many wonderful and knowledgeable staff, but none of them could honestly argue that the LSAT does not reliably correlate with bar exam scores. Indeed, as mentioned above, the LSAT probably correlates about as well with bar exam scores as it does with law school grades. It is puzzling why someone like Bernstine would want to rock the boat when he's raking in over $600,000 a year for doing the extremely easy job of "running" an organization on autopilot with a monopoly on a very lucrative business for its executives. If Bernstine has the confidence of the LSAC, it may be time to encourage the ABA to end the LSAC's monopoly on the law school admission business.

Full disclosure: I am not affiliated with LST, but I assisted them in interpreting the National Longitudinal Bar Passage Study and am named in the LST report for having provided this assistance.

Comments

[Lis Pendens]

In my view, LSAC is an elitist, monopolistic institution that absolutely should not be allowed to dominate law school admissions; there should be other admissions paths—nor should they be allowed to mess with people’s gpas. The LSAT takes care of providing a universal, numerical measure of applicants. There is absolutely no bonafide need for this grossly unfair tampering. Their presence represents an artificially induced impediment to upward mobility and makes acquiring a legal education even more gratuitously depressing and difficult than it already has been made.

The truth is most of what LSAC does has almost nothing to do with the quality of legal education and the well-being of students but rather is directly tied to manufacturing prestige (driving up applications and garnering loads of money) in the form of the almost pornographic and obscene publication of law school rankings by US News.

It would be nice to live in a more relaxed, democratic, and modern world without this 18th century paranoia about lineage, rankings, and status—the very elements of elitism and bigotry. Legal education could not only be more dynamic, diverse, and reflective of the people living in the United States, but it could also be kinder and more decent, resulting in much better mental health outcomes of law students and those in cycle.