Analysis of the California Bar Examination Report, Part 2

This post continues my series on the recent report on bar passage commissioned by the California State Bar. Today's installment presents one of the most interesting charts in the report, one that everyone connected with legal education should understand. Figure 2.3, pictured below, shows the relationship between LSAT, undergraduate GPA (UGPA), first year law school GPA (FYGPA), cumulative law school GPA (LSGPA), and bar exam scores. The data is derived from a large number of California law graduates for 2013, 2016, and 2017. To compare California Bar Exam scores to those in other states they should be divided by 10 (1440 is a 144, etc.)

The first point to note is one that is fairly well known among those who study bar performance, which is that law school GPA (whether first year or cumulative) is much more predictive of bar performance than incoming predictors like UGPA or LSAT. For example, in 2013 the correlation of FYGPA to bar score was .582 and the correlation of LSGPA to bar score was .640, compared to LSAT's .424 and UGPA's .212.

I will make two related points here. The first is that if schools want to predict bar performance, incoming predictors like LSAT and UGPA are very weak compared to law school grades. The second, more speculative, point is that UGPA is likely diluted by the very high levels of noise from different types of classes. It is likely that there are at least some types of undergraduate classes that track better than others the type of thinking necessary for legal analysis. If law schools want to outmaneuver one another in admissions for maximum bar performance, it is likely there is some data analysis in this area that is going unexploited. Really, this is the type of value-added product that LSAC should be providing to law schools but doesn't. (Which is a symptom of the underlying problem with LSAC and why ETS will likely diminish the LSAC's role in law school admissions, etc.)

There are a couple of other interesting points to note for future posts in this series. The slope for LSAT to bar score in 2013 was quite a bit shallower than the same slope for 2016 and 2017. The same is true for UGPA. The report doesn't provide a statistical test of these slopes against each other, but the difference in slopes is potentially important for the bar debate. If one were predicting 2016 and 2017 performance based on 2013 data, one would predict higher performance at the low end of the range than what actually occurred. There are some econometric reasons why the drop in bar scores was not fully "predictable" from LSAT and UGPA alone using standard techniques. More on this later.

 

Comments

[Anon]

Of course LSAT is the only pre-admission metric available. Also does this control for various forms of grade inflation? And is there a tighter fit with a subset of courses' GPAs? Asking for a friend at a law school on the edge.

[Rob Anderson]

I've been thinking of doing a post on this. I don't have any data to back it up though. I think there are some easy checks a law school could do if they had someone who knew data analysis.