I had originally planned for the second part of this series to turn to gender differences, but I got so much constructive feedback on the first part that I decided to explore this aspect of the data in more detail first.
I should begin by reiterating that “in this series of posts I will be exploring the dataset in a broad way.” The purpose of these posts is to generate hypotheses and think out loud about the strengths and weaknesses of the data. Any firm conclusions about what the data show will require a much more complete statistical model than I’ve presented here so far.
What is Experience?
In reflecting on my use of age and the number of years on the bench as proxies for experience, it occurred to me that a closer proxy might be the number of appeals each judge has experienced. This reframes the question somewhat. Rather than “do judges get better at their job over time or with age?” the question becomes “do judges learn from being appealed?”.
This approach implicitly takes into account that different judges can have very different appeal rates, which can reflect differing caseloads and subject matter. To that end, I calculated the mean reversal rate at each appeal number.
After an initial ramp-up, the reversal rate holds steady through about 300 appeals and then declines. The reversal rate becomes increasingly noisy as the number of judges with that many appeals declines. Overall the shape of this graph does suggest that judges with a large number of appeals learn to avoid reversal, but there could be other explanations. Perhaps judges with high reversal rates are more likely to retire early, while judges with low reversal rates stay on the bench (survivor bias). Or perhaps the appeals of that small number of judges with a large number of appeals are disproportionately made up of low-reversal subject matter cases.
Another important caveat of this approach is that, of course, we only have appeals data beginning c. 2000, whereas many judges in the dataset were commissioned before that. Thus, what the data considers a judge’s “first” appeal might actually have been their 100th. So let’s take another look, limiting to judges commissioned after January 1st, 2000.
The data is noisier, and the number of judges drops off more sharply, but the same basic pattern remains.
One factor that I mentioned in the comments on the first post was differences between circuits. While the regional circuits address a broadly similar set of issues, the Federal Circuit is unique in being focused on particular subject matter areas, especially in cases appealed from a district court, as opposed to appeals from administrative agencies. And indeed we see that the Federal Circuit is a substantial outlier in its reversal rate. The regional circuits are broadly similar, with the exception of the 4th Circuit, which has an unusually low reversal rate.
This naturally leads to the question of whether the Circuit Judges of the 4th Circuit are unusually prone to reversal or whether the difference primarily lies in the district court judges of the circuit. To investigate this I looked at the mean reversal rate for each circuit and the mean total number of appeals for the judges with appeals in that circuit.
With the exception of the Federal Circuit, which is an outlier for reasons discussed above, there is a pretty clear inverse relationship between a circuit’s reversal rate and the number of appeals experienced by the district judges in that circuit. Excluding the Federal Circuit the Pearson correlation between the two is -0.901.
Again, a more robust model would also take into account factors such as ideology and subject matter. Even though the geographic circuits have the same subject matter jurisdiction, factors such as the concentration of industries in different parts of the country will naturally lead to differences in the distribution of case types.