W I T N E S S E T H:

Analysis of the California Bar Exam Report, Part 3

The last couple of weeks I've highlighted various findings from the report commissioned by the California State Bar on the decline in bar exam pass rates. The report contains a lot of interesting information not previously publicly available, and I encourage law school faculty and administrators to read it in its entirety.

Today I want to highlight one of the more important tables in the Report, one that sets forth the results of a regression of various demographic and other factors on bar exam score. Table III.13, included below, shows how MBE score, written score, and total score vary with age, gender, minority status, undergraduate GPA (UGPA), LSAT, law school GPA (LSGPA), and repeater status. Variables with asterisks instead of numbers were not statistically significant.

There are a number of results that are expected based on prior work. LSGPA strongly predicts bar passage score, so much that moving from less than a 2.8 to above a 3.51 predicts an increase of 224 points on the total bar score (22.4 points on the nationwide scale). That's predicted pass/fail percentage increase of 66%! LSAT continues to be significant, but its magnitude is relatively small once LSGPA has been controlled. Moving from less than a 154 to greater than a 164 predicts an increase in total bar score of about 44 points (4.4 points on the national scale). UGPA is statistically significant but somewhat inconsequential, adding only a few points. Also, the findings are consistent with the well-documented regularity that men tend to score higher on the MBE and women higher on the essay portions, which tend to cancel each other out producing a roughly equal total score.

One of the unusual results is how much age appears to affect bar score and the way in which age appears to affect bar score. A student who is 30 or older scores almost 34 points lower on the total bar score (3.4 on the national scale) than a student who is 25 or under. That translates into almost a 10% difference in pass rate on the exam. This is comparable to the effect of moving from the lowest LSAT category to the highest one. In a sense, one might expect increased age to result in lower scores if not all variables have been controlled. After all, life tends to get more complicated with family and other obligations as one moves into one's 30s. Also, older students are more likely to be employed or be attending law school part time. However, the "life's more complicated" explanation should affect the MBE and written portion similarly. Yet the age variable seems to only predict a substantially decreased score on the written portion; the MBE is roughly the same for younger and older takers. Also, the "life's more complicated" explanation should have affected LSGPA similarly, and LSGPA is controlled in this regression. Yet the result remains.

The age coefficient is large enough in magnitude that it might be worthwhile for law schools that have range of student ages to investigate this variable further. I don't have a theory for why the writing portion specifically would be more challenging for older students, but it seems like one where an intervention might make a difference.

 

Disaggregating undergraduate GPA for law school admission

In last Friday's post, I discussed results from the California Bar Exam Report showing how much better 1L GPA predicts bar performance than either LSAT or undergraduate GPA (UGPA). Several people commented on that post pointing out that 1L GPA obviously can't be used for admission purposes and wondering whether better predictors are available at the pre-admission stage. 

There are a couple of responses to this. First, in a sense 1L GPA could be available for admission if schools relied more on transfer students and attrition to shape their final graduating classes. I discussed this in a previous post about the transfer market.

Second, and more directly in response to the comments, even without 1L GPA there likely is much more information available at the admission stage than most schools are currently exploiting. Specifically, last week's post pointed out that UGPA is likely inferior to a more fine-grained analysis of an applicant's undergraduate transcript.

I will make two related points here. The first is that if schools want to predict bar performance, incoming predictors like LSAT and UGPA are very weak compared to law school grades. The second, more speculative, point is that UGPA is likely diluted by the very high levels of noise from different types of classes. It is likely that there are at least some types of undergraduate classes that track better than others the type of thinking necessary for legal analysis. If law schools want to outmaneuver one another in admissions for maximum bar performance, it is likely there is some data analysis in this area that is going unexploited. Really, this is the type of value-added product that LSAC should be providing to law schools but doesn't. (Which is a symptom of the underlying problem with LSAC and why ETS will likely diminish the LSAC's role in law school admissions, etc.)

What is needed is not a subjective, human analysis of each transcript, but a separate GPA calculation based on predictive quality of individual classes or groups of classes. This would be based on machine learning techniques and not on intuition. As mentioned in the quoted passage from the prior post, ideally LSAC would provide a product for law schools that would analyze each individual applicant transcript for GPA quality--that is, the predictive value of individual courses or groups of courses taken as an undergraduate. All of the information needed to conduct such an analysis is available to LSAC as it already processes transcripts. The fact that such a product doesn't exist after all these decades I can only attribute to (1) the lack of vision of LSAC vision over the last decade, plus (2) the fact that enhancing the predictive value of UGPA would reduce the relative predictive power of the LSAT.

However, law schools could conduct a version of this analysis internally without LSAC. It is possible that a law school would have enough data points to conduct such an analysis based on matriculated students' actual law school grades. But there is another possibility that would significantly increase the number of data points: the use of applicant data to conduct the analysis. Most law schools have far more applicants than matriculated students, so this greatly expands the data available to train the predictive model.

How is it possible to train such a predictive model without law school grades for all the applicants? The solution is to use LSAT as a stand-in for law school GPA. The undergraduate courses or groups of courses that correlate most strongly with LSAT will also likely better predict law school performance independently of LSAT. It's a bit complex to explain why this is true, but there are strong reasons to believe it is. And law schools wouldn't need to just take my word for it: they could build a predictive model based on applicants and LSAT, leaving out matriculated students, then test it on matriculated students' grades. 

I believe that UGPA would outperform LSAT as a predictor if this information were used to improve the UGPA measure. It is possible some law schools are already doing this without publicizing it...

Robert's Rules for Effective Law School Teaching

I have been teaching in law school for 11 years now, so it's time to start giving back to the teaching community with some of my teaching tips. These tips are aimed at newer law teachers who would like to maximize their scores on student evaluations, the definitive indication of teaching quality.

Robert's Rules for Great Law Teaching:

1. Oversimplify the law. Each legal issue should have a single black-letter test comprised of a list of not more than 4 "prongs." Students find differences among states/circuits complicated and frustrating. Why bring them up? If you must, one "minority rule" may be presented for up to 10% of issues. Subtle differences in verbal formulations in actual cases must be avoided at all costs. Multi-factor tests are frustrating to students and should be avoided.
2. Do not acknowledge ambiguity in the law. Avoid acknowledging that the law is uncertain or that it varies from state to state. Only inexperienced professors are unable to definitively resolve all issues into a canonical list of prongs (see previous Rule). 
3. Answer all questions with confidence whether or not you know the answer. The more uncertain you are about the answer, the more confidence you should project when providing a random answer. If you show uncertainty, students will perceive you as weak. If you answer confidently but incorrectly, few students will check you and nobody will believe them anyway.
4. In the event you are too unprepared to make something up confidently, respond with a question. Preferably, respond with a question that subtly ridicules the student who asked the question for reading the assignment too closely or being a "gunner."
5. Do not require students to know anything outside the lecture notes. Sometimes students will ask questions such as, "There's a lot of complicated junk on p. 357. Do we need to know that for the test?" Always answer reassuringly that the students are not responsible for complicated junk in the reading assignments. Exception: if you follow Rule 6, then require students to know everything from the reading.
6. Consider assigning a commercial supplement you wrote as "recommended" reading. All of the students will feel compelled to purchase the supplement, and you will make more money. In addition, this will provide a much needed ego boost of showcasing your rigorous scholarship.
7. If a student asks a complicated question say, "That's beyond the scope of the course." The student will feel special that he or she went beyond the scope of the course, and everyone else will be relieved they don't need to know the answer. Feel free to test on the question, however, as students likely will have already filled out their evaluations.
8. Draw distinctions between the good teachers and the professors who write articles. Everybody knows that good classroom teaching and good scholarship are mutually exclusive or at least not complementary, so steer the students away from faculty members who write articles instead of caring about students.
9. Dress up for class. Yes, slackers can accomplish simple tasks like founding and running a $400 billion company wearing a gray t-shirt and jeans every day, but teaching is a super-serious thing that requires you to dress like you're at a funeral.
10. Never expect students to know arithmetic. This isn't 4th grade. We don't use fractions or division in advanced graduate school.
11. Mention your degrees frequently. This is especially the case if your degree came from a significantly higher-ranked school than where you teach. If you have a Ph.D., refer to yourself as Dr. and insist that others do so also. Otherwise, simply put Esq. after your name. If you are ashamed to mention the names "Harvard" or "Yale," say "Cambridge" or "New Haven." If you went to Yale but don't teach at Yale, talk frequently about people at Yale in a way that suggests you are close to them. If If you do teach at Yale, it probably doesn't matter what you say or do.
12. Provide cookies. Studies show that providing cookies enhances learning. Donuts may be provided for a morning class.
13. Provide many similar sample exams that telegraph exactly the issues you will test on the final exam. This will give students a strong sense of confidence going into the exam that they know exactly what the issues will be. By the time they take the exam, the evaluations will likely have been submitted already.
14. Near the end of the course, tell students you wrote an easy exam this year. Of course, in a curved course "easy" versus "hard" is meaningless, but never point that fact out.
15. If more recently hired professors have outpaced you, talk about the time you "hired" them. Preferably refer to the professor as a wee lass/lad at the time. It is important to soften the point by saying that the professor "is a good kid."

Analysis of the California Bar Examination Report, Part 2

This post continues my series on the recent report on bar passage commissioned by the California State Bar. Today's installment presents one of the most interesting charts in the report, one that everyone connected with legal education should understand. Figure 2.3, pictured below, shows the relationship between LSAT, undergraduate GPA (UGPA), first year law school GPA (FYGPA), cumulative law school GPA (LSGPA), and bar exam scores. The data is derived from a large number of California law graduates for 2013, 2016, and 2017. To compare California Bar Exam scores to those in other states they should be divided by 10 (1440 is a 144, etc.)

The first point to note is one that is fairly well known among those who study bar performance, which is that law school GPA (whether first year or cumulative) is much more predictive of bar performance than incoming predictors like UGPA or LSAT. For example, in 2013 the correlation of FYGPA to bar score was .582 and the correlation of LSGPA to bar score was .640, compared to LSAT's .424 and UGPA's .212.

I will make two related points here. The first is that if schools want to predict bar performance, incoming predictors like LSAT and UGPA are very weak compared to law school grades. The second, more speculative, point is that UGPA is likely diluted by the very high levels of noise from different types of classes. It is likely that there are at least some types of undergraduate classes that track better than others the type of thinking necessary for legal analysis. If law schools want to outmaneuver one another in admissions for maximum bar performance, it is likely there is some data analysis in this area that is going unexploited. Really, this is the type of value-added product that LSAC should be providing to law schools but doesn't. (Which is a symptom of the underlying problem with LSAC and why ETS will likely diminish the LSAC's role in law school admissions, etc.)

There are a couple of other interesting points to note for future posts in this series. The slope for LSAT to bar score in 2013 was quite a bit shallower than the same slope for 2016 and 2017. The same is true for UGPA. The report doesn't provide a statistical test of these slopes against each other, but the difference in slopes is potentially important for the bar debate. If one were predicting 2016 and 2017 performance based on 2013 data, one would predict higher performance at the low end of the range than what actually occurred. There are some econometric reasons why the drop in bar scores was not fully "predictable" from LSAT and UGPA alone using standard techniques. More on this later.

 

The California bar crisis: a changing bar exam or changing admissions standards?

The falling bar passage rates in California have prompted many law school deans to look for possible causes of the decline other than the drop in incoming predictors of law students. One argument I hear often is that the primary problem isn't admitting weaker students, it's that the bar exam has somehow become more difficult. Often, these arguments are based on misunderstanding of how the test is scored (such as the false notion that the exam is graded on a "curve" etc.)

In reality, the essay portions of the bar exam are scaled to the MBE, and the MBE is "equated"  to ensure that scores are comparable from year to year as explained on the California Bar website here. As a result, the exam does not become more or less difficult depending on the mix of takers in a particular administration.  The combination of scaling and equating means that unless the passing score is changed, the difficulty does not change (the LSAT does the same thing). The process is designed to ensure comparability from year to year. In theory, a passing grade thirty years ago is the same as a passing grade today. I say "in theory" because, of course, it's possible for the equating process to go awry.

One way to test whether the fall in passing rates is attributable to decreased admission standards or an exam "gone awry" is to compare the performance of recent law graduates to those who graduated earlier. Although this comparison is difficult to conduct directly, there is an important piece of information bearing on this question that hasn't been discussed in the existing commentary. Recent law school graduates are not the only ones who take the California bar. Lawyers seeking admission in California from other states take it as well. There are two versions of the exam for lawyers: the General Bar Exam, which is the same full bar exam taken by recent graduates, and the "Attorney's Exam" which omits the MBE portion, as explained here. As a result, hundreds of attorneys take both versions of the exam every year, providing a comparison group to the recent law school graduates.

The chart below compares the pass rate over time of the three groups (recent graduates of ABA schools, Attorney's Exam takers, and attorneys taking the General Bar Exam).

The plot shows that the attorney examinees have passed at pretty much the same passing rates every year from 2007 to 2018. Although there are a few bumps up and down, the rates do not move in tandem with those of recent graduates. In contrast, the passing rate for graduates of ABA law schools has declined significantly since 2008. Indeed, recent graduates historically have outperformed attorneys taking the General Bar Exam, but in recent years the two lines have converged. 

If the decline in passing rates were the result of changes to the difficulty of the bar exam, we should see the passing rates for attorney applicants declining in tandem with those for new law graduates. Indeed, if the MBE were specifically responsible, we should see that pattern for the recent graduates and the attorney General Bar Exam takers, with no decline for the Attorney Exam takers (as the Attorney Exam doesn't include the MBE). Instead, the data shows no changes for either category of attorney takers but a substantial reduction in the passing rate for recent graduates.

This is one more piece of evidence that the bar passage crisis is the result of admission decisions made by California law schools, rather than any changes to the bar exam itself.