W I T N E S S E T H:

Analysis of the California Bar Examination Report, Part 2

This post continues my series on the recent report on bar passage commissioned by the California State Bar. Today's installment presents one of the most interesting charts in the report, one that everyone connected with legal education should understand. Figure 2.3, pictured below, shows the relationship between LSAT, undergraduate GPA (UGPA), first year law school GPA (FYGPA), cumulative law school GPA (LSGPA), and bar exam scores. The data is derived from a large number of California law graduates for 2013, 2016, and 2017. To compare California Bar Exam scores to those in other states they should be divided by 10 (1440 is a 144, etc.)

The first point to note is one that is fairly well known among those who study bar performance, which is that law school GPA (whether first year or cumulative) is much more predictive of bar performance than incoming predictors like UGPA or LSAT. For example, in 2013 he correlation of FYGPA to bar score was .582 and the correlation of LSGPA to bar score was .640, compared to LSAT's .424 and UGPA's .212.

I will make two related points here. The first is that if schools want to predict bar performance, incoming predictors like LSAT and UGPA are very weak compared to law school grades. The second, more speculative, point is that UGPA is likely diluted by the very high levels of noise from different types of classes. It is likely that there are at least some types of undergraduate classes that track better than others the type of thinking necessary for legal analysis. If law schools want to outmaneuver one another in admissions for maximum bar performance, it is likely there is some data analysis in this area that is going unexploited. Really, this is the type of value-added product that LSAC should be providing to law schools but doesn't. (Which is a symptom of the underlying problem with LSAC and why ETS will likely diminish the LSAC's role in law school admissions, etc.)

There are a couple of other interesting points to note for future posts in this series. The slope for LSAT to bar score in 2013 was quite a bit shallower than the same slope for 2016 and 2017. The same is true for UGPA. The report doesn't provide a statistical test of these slopes against each other, but the difference in slopes is potentially important for the bar debate. If one were predicting 2016 and 2017 performance based on 2013 data, one would predict higher performance at the low end of the range than what actually occurred. There are some econometric reasons why the drop in bar scores was not fully "predictable" from LSAT and UGPA alone using standard techniques. More on this later.

 

The California bar crisis: a changing bar exam or changing admissions standards?

The falling bar passage rates in California have prompted many law school deans to look for possible causes of the decline other than the drop in incoming predictors of law students. One argument I hear often is that the primary problem isn't admitting weaker students, it's that the bar exam has somehow become more difficult. Often, these arguments are based on misunderstanding of how the test is scored (such as the false notion that the exam is graded on a "curve" etc.)

In reality, the essay portions of the bar exam are scaled to the MBE, and the MBE is "equated"  to ensure that scores are comparable from year to year as explained on the California Bar website here. As a result, the exam does not become more or less difficult depending on the mix of takers in a particular administration.  The combination of scaling and equating means that unless the passing score is changed, the difficulty does not change (the LSAT does the same thing). The process is designed to ensure comparability from year to year. In theory, a passing grade thirty years ago is the same as a passing grade today. I say "in theory" because, of course, it's possible for the equating process to go awry.

One way to test whether the fall in passing rates is attributable to decreased admission standards or an exam "gone awry" is to compare the performance of recent law graduates to those who graduated earlier. Although this comparison is difficult to conduct directly, there is an important piece of information bearing on this question that hasn't been discussed in the existing commentary. Recent law school graduates are not the only ones who take the California bar. Lawyers seeking admission in California from other states take it as well. There are two versions of the exam for lawyers: the General Bar Exam, which is the same full bar exam taken by recent graduates, and the "Attorney's Exam" which omits the MBE portion, as explained here. As a result, hundreds of attorneys take both versions of the exam every year, providing a comparison group to the recent law school graduates.

The chart below compares the pass rate over time of the three groups (recent graduates of ABA schools, Attorney's Exam takers, and attorneys taking the General Bar Exam).

The plot shows that the attorney examinees have passed at pretty much the same passing rates every year from 2007 to 2018. Although there are a few bumps up and down, the rates do not move in tandem with those of recent graduates. In contrast, the passing rate for graduates of ABA law schools has declined significantly since 2008. Indeed, recent graduates historically have outperformed attorneys taking the General Bar Exam, but in recent years the two lines have converged. 

If the decline in passing rates were the result of changes to the difficulty of the bar exam, we should see the passing rates for attorney applicants declining in tandem with those for new law graduates. Indeed, if the MBE were specifically responsible, we should see that pattern for the recent graduates and the attorney General Bar Exam takers, with no decline for the Attorney Exam takers (as the Attorney Exam doesn't include the MBE). Instead, the data shows no changes for either category of attorney takers but a substantial reduction in the passing rate for recent graduates.

This is one more piece of evidence that the bar passage crisis is the result of admission decisions made by California law schools, rather than any changes to the bar exam itself.

 

Analysis of the California Bar Examination Report, Part 1

The California State Bar is out with Part 2 of its Report on Performance Changes on the California Bar Examination. This is a very valuable resource for anyone interested in understanding student performance in law school and on the bar examination, whether in California or elsewhere. I hope many law faculty members will at least skim the report and I plan to blog some of the highlights over the next few weeks.

The distressing decline in bar scores and passing rates in California triggered a series of studies from the California Bar. From 2013 to 2016 passing rate declined from 56% to 44%. That was on top of an already significant drop from 2008 when the passing rate was 62%. In terms of scores on the exam, between 2013 and 2016, the total average California Bar Exam score dropped 48 points (equivalent to 4.8 points on the national scale used in most states). In addition, the gap between California and other states has been closing, with California barely edging out nationwide takers on the MBE in 2018. (California has historically outperformed most other states on the bar exam).

The Report attempts to explain why the bar pass rate and bar scores have declined so much. It is well known that the incoming academic predictors of law students have declined precipitously in California (and nationwide) over the same period, so many have attributed the drop in bar scores to less competitive incoming classes of law students. The Report concludes that between 20% and 50% of the drop in bar exam performance may be traced to the decline in academic predictors of the incoming classes, leaving a lot to be explained. Many have speculated about what the other reasons might be, with common explanations centering around changes in how students learn, study, and take tests, often centered on the rise of technology in daily life.

However, there is an important piece of evidence the State Bar didn't analyze in the report. A very similar pattern occurred in the previous law school downturn (1994-2002), with bar scores dropping far out of proportion with LSAT . Between 1994 and 2002 the pass rate dropped about 13 points. At the same time, incoming student predictors dropped as well. The 2003 report contained the following table comparing the decline in LSAT to the decline in bar scores:

The same psychometrician used in the current studies, Roger Bolus, found the same thing in that study as the State Bar found in the current Report-- that changes in incoming predictors could not fully account for the decline in bar performance. Here is a concluding blurb from that study, dated 2003:

The fact that a previous LSAT drop was accompanied by an exaggerated drop in bar exam scores, paralleling almost exactly the current trend, is an important piece of evidence in unraveling the causes of the current decline. It seems unlikely that smartphones, Millennial culture, avocado toast, or Fortnite or anything uniquely associated with modern law student life is the "secret ingredient" of the decline in bar scores. Instead, it seems more likely that the culprit is a missing methodological or econometric piece of the puzzle.  I will explore some possibilities in subsequent posts.