Federal District Court Reversal Rates, Part 1: Experience and Age

In US District Courts by James Daily9 Comments

It seems almost axiomatic that, among the hundreds of federal district court judges, some are better at their job than others.  One possible measure of the quality of their work is the frequency with which their decisions are reversed on appeal.  A defining feature of our court hierarchy is that higher courts have the power to correct lower ones.  Thus, a reversal of a lower court decision means, in some sense, that the lower court judge made an error, the correction of which delayed the resolution of the case and incurred the considerable expense of an appeal.

Drawing and expanding on the methodology described by Royce de R. Barondes in his 2010 essay Federal District Judge Gender and Reversals, I collected Westlaw Litigation Analytics Appeals reports for 1052 district court judges commissioned between 1989 and 2020.  After limiting to judges with at least 50 appeals in the Westlaw data and excluding extremely early decisions, some of which were likely appeals from state court decisions, I was left with 252,703 observations covering 877 judges.  Due to export limitations in Westlaw, one judge with more than 1000 observations was limited to the first 1000.  Following Barondes, “reversed” is defined as reversed, reversed in part, or vacated.  In this series of posts I will be exploring the dataset in a broad way.

At the outset I should be clear that reversal rate is an imperfect proxy for quality.  Sometimes the district court decision was actually correct at the time, as when a lower court judge correctly applies a precedent which is then overturned on appeal.  And circuit courts make mistakes as well, hence such mechanisms as rehearing en banc and appeal to the Supreme Court.  However, only a small fraction of appellate decisions are appealed further and many of those are ultimately affirmed.  Still, we must recognize that a circuit court’s decision is not the “ground truth” and that not all reversals are equal (consider a complete reversal versus a reversal-in-part that affirms every substantive aspect of a case with only a minor change in the calculation of damages).

For scale, the histogram of the number of observations per judge:

We can see that there are some outliers with a large number of appeals, but overall this is a fairly typical half-normal distribution, as expected given that a judge cannot have fewer than zero appeals.  Here is the QQ plot against a half-normal distribution:

With that in mind, let us begin in perhaps the most obvious place: experience.  Do judges get better at their jobs over time?  First a histogram of the number of observations by the number of years since the judge’s commission.

(“Years Since Commission” is the number of years between the district judge’s commissioning and the date of the appellate decision.  I would prefer to use the date of the lower court decision being appealed, but that was not available in the Westlaw data.)

We see some positive skew here caused by the fact that judges leave the bench over time.  There is also a clear “startup” period from years 0 – 2 likely primarily due to the lag between cases progressing enough for a decision to be made and then again for an appeal to be made.

Now we can get to the main question: how does the reversal rate vary with experience?

I have excluded the extremes of the data due to the small number of observations and small number of judges.  As Barondes found in 2010, there is a modest positive correlation between years on the bench and reversal rate, our case R2=0.133.  On its own this is not enough to show significance, but that will wait for a more complete model.

One reason why this is not enough to show significance is that experience does not exist in a vacuum.  Experience comes only with age, and age could also be associated with a change in reversal rates.  First, the number of observations by age:

We can see there is very little data outside of the 1st and 99th percentiles of age 45 and age 80.  Looking at the reversal rate by age shows a fairly clear relationship.

This could be caused by many factors: perhaps older judges making bolder decisions or feel more confident in disagreeing with established precedent or perhaps it reflects a loss of mental acuity.  It is well known that there are differences in aging between men and women, and this leads naturally to the question of whether there is a difference in reversal rates as well, both across time and in general.  For more on that, the next post in this series will focus on gender.

Acknowledgements: I would like to thank Victoria Henige for asking the questions that led to this series of posts and Professor Lee Epstein for sharpening my thinking on it.


View some of the data from the post here

Comments

  1. This is a great post James. I wonder if the increased reversal rate might also be because of changes in the composition of the circuit court? In other words, let’s say President Carter had appointed a judge to the U.S. District Court for East Carolina in the fictitious 12th Circuit. Let’s say at the time that Circuit was stocked primarily by Democratic-appointees, but twenty years later the Circuit had flipped and now was staffed primarily by Republican-appointees.

    Alternatively, has the Supreme Court’s shift in jurisprudence caused an uptick in reversals?

    1. Author

      It’s entirely possible! I have linked the appellate court panels for each appeal to the Judicial Common Space for each judge on the panel (or the median for that circuit that year, if there were no judges listed in the Westlaw data). All of that will be part of a future post.

      The cohort nature of judicial appointments means that part of this could be explained by, eg, the Obama cohort of district court appointees being increasingly reviewed by Trump’s circuit court appointees. There would not be a corresponding balancing effect from Trump’s district court judges yet because they are underrepresented in the data because many are too recent to have a sufficient number of appeals.

  2. Wonderful start, James! As you iterate through the data, I wonder if it would be possibly a worthwhile exercise to take some (considerable) time to extract a random sampling of observations, and then by hand, read through the cases and learn the context. See for yourself if some of your assumptions are true for a random sampling, or if maybe the methodology might need rejiggering. Just a thought, and possibly not a good one.

    1. Author

      Taking a closer look at the data and spot-checking are nearly always worthwhile! Westlaw is a good data source, but it’s not perfect, and already in this post there are a few assumptions. For example, it could be that the time from lower court decision to appellate court decision has a high and non-random variance, high enough to call into question using the appellate decision date to calculate age and years on the bench.

  3. What if we provided an institutional explanation in addition to a behavioralist one?
    What if any given district court judge considers him or herself a potential circuit judge and has a preference matrix of likelihood of elevation that is a function of the judge’s age, the party of the president, the partisan makeup of the governing appellate court, and the existence of divided government. Ideologically extreme district court judges compared to their parent circuit will audition through their explicit ideological rulings; all other district court judges will appear moderate if ideologically close to the circuit or will otherwise moderate their decisions to increase their chances of being a compromise candidate. Given that every president has a preference to fill appellate vacancies with ideological copartisans and that a district court judge with revealed ideological preferences is harder to confirm whereas a president is less likely to nominate a district court judge with hard to decipher preferences, the median district court judge may determine that elevation to the circuit is less likely once they have served for 10 years without consideration. But given the ideologically distant district judges from their circuit are most likely to get reversed, those judges exercising an audition strategy of revealed ideological preferences may exit the court especially once eligible for senior status (hence the spike of reversals for judges 10.5-15.5 into their tenure) because auditioning is fruitless given reversal and no elevation payoff and what may be left are judges whose preferences matched the circuit’s.
    There’s also another factor which is that interest groups (federal court litigants) are the motivating action behind whether a district court decision gets appealed. Those interest groups can price the likelihood that the circuit will ignore a perceived error by a lower judge or correct it through reversal. Further those interest groups can create price signals for future presidents and senators on the district court judge’s competence (by appealing (or not appealing) fact errors that will guarantee reversal independent of circuit ideology) and extremism (by appealing (or not appealing) on questions of law where reversal is most likely in circuits with high ideological distance from a given district court judge).

    1. Author

      A good point and one I plan on investigating. I have already matched the appellate panel information from Westlaw against the Judicial Common Space data to get panel-level ideology scores. The JCS only covers the circuit courts and the Supreme Court, so I can’t look directly at how a district court judge’s score compares to the panel (or to that circuit as a whole), but I intend to use the party of the appointing president (or possibly the president’s DW-NOMINATE score) as a proxy. This will give us a view of the extent to which ideological divergence affects reversal rates.

      I also intend to look at promotions to circuit courts in a future post or possibly a paper. Promotion to a circuit court is arguably also a proxy for judicial quality, although like reversal rates that is debatable if one takes the view that presidents tend to appoint the most ideologically extreme person who can get 60 (or more recently 51) votes in the Senate.

      Another future post will look at the breakdown by circuit. As a patent attorney I was not at all surprised to see that the Federal Circuit has an unusually high reversal rate, since so many of its cases are patent cases, which are frequently decided on grounds that require either no deference (e.g. claim construction; subject matter eligibility) or essentially no deference as a practical matter (e.g. obviousness).

      Unfortunately this dataset doesn’t capture much about the litigants directly, although for cases for which Westlaw assigns one or more case types (e.g. “Securities Law”) it is possible to make some assumptions. There are also textual summaries, and natural language processing could be used to identify whether the parties included individuals, corporations, state governments, the federal government, etc. All grist for the mill!

  4. Does your study list individual reversal rates by judge? That is, can I review how I match up? Thank you.

    1. Author

      Each row in the dataset is an appellate decision including information about the district court judge, the appellate court and panel, and the decision on appeal. I based this post on a dataset that ends in 2020 because it was linked to the Judicial Common Space scores for each circuit for each year and that data is not available for 2021 yet. However, the base data runs through February 11, 2021.

      In your case the base data contains 157 appeals with a reversal rate of 20.4%, which is in the 86th percentile. Within the cohort of 91 first-term Obama appointees with at least 50 appeals* it is higher at the 92nd percentile.

      Your reversal rate at the Sixth Circuit is 17.8%, which is the 61st percentile among all Sixth Circuit appeals. Your reversal rate at the Federal Circuit is 100% (i.e. all 5 appeals). This is the 95th percentile among all Federal Circuit appeals.

      This is a great example of why this series of posts is building toward a more complete model that takes into account characteristics of the circuit court and even the particular appellate panel.

      * That excludes 6 first-term Obama appointees with fewer than 50 appeals in the data.

    2. Judge Black:

      As the primary post notes, I undertook this type of investigation some years ago. I investigated the relationship between reversal rate and each of:

      (i) law schools of judge’s clerks;
      (ii) ABA rating; and
      (iii) certain demographics (the latter being the paper linked in this post).

      My investigation of judicial reversal rates indicated, not surprisingly, that the subject of the opinion can be significantly related to the likelihood of reversal. So, when I investigated this, I reported results that took into account the subject for each individual opinion.

      The subject of a judicial opinion is, of course, related to an appeal being to the Court of Appeals for Federal Circuit. The author’s comment in reply separates out those opinions. However, without corresponding information as to the remainder of the opinions, it is not clear to me that one can draw any form of inference from the raw reversal rates. An individual judge may disproportionately write (or have appealed) opinions addressing subjects that are more frequently reversed (or those that are less frequently reversed).

      In your particular case, it occurred to me that it might be helpful to report what this data set would say about an appellate opinion addressing an opinion issued by your court where the appellate disposition was was subsequently reversed by the Supreme Court: DeBoer v. Snyder, 772 F.3d 388 (Nov. 6, 2014), rev’d, Obergefell v. Hodges, 576 U.S. 644 (2015). It had been my intent to suggest that Westlaw may have shown your lower-court opinion reversed, and to note that such a classification would, one supposes, not be fully reflective of the circumstances. However, as best as I can tell, this disposition in your court is simply omitted from the Westlaw database.

      I hope the above is informative.

      Regards,

      Royce

Leave a Comment