Interesting article evaluating recent draft efficiency from Football Outsiders.
Guest columnist Benjamin Ellinger breaks down every draft pick in the past 10 years. Did Cleveland's tanking policy pay off? Which teams had the best results in the draft, and which wasted their picks? And is there anything meaningful in this data, or is there really some skill involved? The...
www.footballoutsiders.com
NFL Drafting Efficiency, 2010-2019
Guest column by Benjamin Ellinger
Every year, NFL fans look back on the draft and immediately wonder: was it a good draft for my team? Pundits compare who was drafted to the consensus draft boards and are impressed when general managers make the obvious picks. Teams that have lots of high draft picks get excellent grades (unless they deviate too much from the conventional wisdom). The Seattle Seahawks get poor draft grades because nobody understands why they made the picks they did.
To make clear what my biases are: I'm a Seattle Seahawks fan. I would not trade Pete Carroll and John Schneider for Bill Belichick. Russell Wilson is the best quarterback in the NFL. Bobby Wagner is the best linebacker since Ray Lewis. Throwing the ball on the last play of Super Bowl XLIX was the right call. The Beastquake is the greatest run in NFL history. If my life depends on one catch, I'm praying that Doug Baldwin or Steve Largent is the target. So, if I tell you that the Seahawks are better at drafting players than all other teams over the last 10 years, you would be right to be skeptical. The data says I'm right, though.
But how can we really measure, objectively, how good a team is at drafting players? First, we must consider how many draft picks you have and how good those picks are. This is your team's "draft capital." If you have a bunch of high picks, that may be due to a clever GM who is good at making
trades, but they are only good at
drafting if they get more than an average GM does out of those picks. To see how much a team gets out of a draft, we need a measure that gives a value to all players, regardless of position, based on their actual on-field performance (not their potential, not their talent, not their time in the 40-yard dash). We can then compare how much total value a team got out of a particular draft relative to how much we would expect an average team with the same amount of draft capital to get.
So here's the method. First, to calculate draft capital, let's use
Chase Stuart's draft value chart (generally considered superior to the original Jimmy Johnson chart). Each draft pick, from the No. 1 pick down to the last pick of the draft, is given a value based on the average amount of career approximate value (
CarAV) that pick has generated. This is a solid approach, as the entire point of approximate value is to have a cross-position single number to compare any player's value to the team to any other player's value.
However, since we will want to compare drafts from different years to each other, we need to normalize these values so we can compare different years fairly. So each draft position is converted from an expected CarAV into a percentage of the total expected CarAV of the entire draft. We'll use this trick throughout this method, calculating the percentage share of an entire draft class (or multiple classes when aggregating multiple years).
Draft Capital
The following table shows each team's percentage of draft capital in each of the last 10 years, as well as the totals of the last decade and the last five years. In short, the teams at the top of the table
should have found the most talent in these drafts. Values shown in gold are in the top 1%, while those in green are in the top 10%. Those in red are in the bottom 10% and those in grey are in the bottom 1%.
| Total Draft Capital, 2010-2019 | | | | | | | | | | | | |
| Team | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 10-Yr | 5-Yr |
| CLE | 4.14% | 3.51% | 5.11% | 2.29% | 4.54% | 4.77% | 5.25% | 6.05% | 7.18% | 1.87% | 4.47% | 5.02% |
| SF | 3.87% | 3.52% | 2.10% | 3.75% | 3.92% | 3.38% | 3.90% | 4.43% | 3.66% | 4.26% | 3.68% | 3.93% |
| TB | 4.63% | 2.84% | 3.41% | 2.16% | 3.36% | 4.36% | 3.42% | 2.72% | 4.28% | 3.85% | 3.50% | 3.73% |
| LAR | 4.88% | 2.95% | 4.86% | 3.97% | 5.09% | 3.32% | 3.28% | 2.38% | 1.65% | 2.34% | 3.47% | 2.59% |
| CIN | 3.34% | 3.96% | 4.43% | 3.58% | 2.57% | 3.28% | 2.57% | 3.84% | 3.47% | 3.56% | 3.46% | 3.34% |
| JAX | 2.43% | 2.66% | 3.34% | 4.33% | 4.43% | 3.99% | 3.56% | 3.71% | 2.22% | 3.61% | 3.43% | 3.42% |
| TEN | 2.77% | 3.71% | 2.81% | 3.74% | 2.88% | 4.51% | 4.84% | 3.91% | 2.04% | 2.75% | 3.40% | 3.61% |
| BUF | 3.52% | 4.37% | 3.82% | 3.81% | 3.56% | 1.48% | 2.82% | 2.68% | 3.80% | 3.42% | 3.33% | 2.84% |
| OAK | 3.76% | 2.44% | 1.04% | 3.35% | 3.64% | 3.94% | 3.12% | 2.47% | 3.52% | 5.54% | 3.28% | 3.72% |
| NYG | 3.02% | 2.73% | 2.41% | 2.76% | 3.25% | 3.10% | 3.29% | 2.50% | 4.58% | 4.84% | 3.25% | 3.66% |
| BAL | 2.60% | 2.67% | 2.70% | 2.67% | 3.37% | 3.10% | 4.67% | 3.39% | 3.86% | 2.90% | 3.19% | 3.58% |
| DEN | 3.98% | 4.80% | 3.00% | 2.44% | 2.11% | 2.47% | 2.50% | 2.95% | 4.30% | 3.37% | 3.19% | 3.12% |
| ARI | 2.58% | 3.83% | 2.47% | 3.90% | 2.97% | 2.67% | 1.90% | 3.10% | 2.82% | 5.21% | 3.14% | 3.14% |
| DET | 3.84% | 2.83% | 2.87% | 3.69% | 3.38% | 2.56% | 3.10% | 2.85% | 2.76% | 3.39% | 3.13% | 2.93% |
| HOU | 3.17% | 3.37% | 3.14% | 2.95% | 4.79% | 2.62% | 2.76% | 2.94% | 1.92% | 2.97% | 3.06% | 2.64% |
| NYJ | 2.04% | 1.79% | 2.60% | 4.36% | 3.53% | 3.43% | 2.67% | 3.59% | 3.04% | 3.38% | 3.04% | 3.22% |
| PHI | 4.19% | 3.17% | 3.64% | 3.84% | 2.89% | 2.38% | 2.98% | 3.15% | 1.26% | 2.67% | 3.02% | 2.49% |
| MIA | 2.90% | 2.24% | 3.92% | 4.42% | 2.88% | 2.95% | 3.17% | 2.40% | 3.18% | 2.01% | 3.01% | 2.74% |
| MIN | 2.26% | 3.10% | 4.46% | 3.51% | 3.65% | 3.41% | 2.30% | 2.25% | 2.20% | 2.89% | 3.00% | 2.61% |
| IND | 2.27% | 2.53% | 4.70% | 2.03% | 1.21% | 2.48% | 2.96% | 3.35% | 5.22% | 3.29% | 3.00% | 3.46% |
| WAS | 2.30% | 3.42% | 3.75% | 1.81% | 2.42% | 4.07% | 2.41% | 3.17% | 2.93% | 3.60% | 2.99% | 3.24% |
| KC | 4.56% | 3.38% | 3.24% | 4.20% | 2.01% | 3.24% | 2.73% | 2.76% | 1.98% | 1.75% | 2.99% | 2.49% |
| SEA | 4.38% | 2.70% | 3.47% | 2.02% | 2.38% | 1.80% | 3.12% | 3.41% | 2.56% | 3.72% | 2.96% | 2.92% |
| NE | 3.93% | 4.13% | 3.22% | 2.48% | 2.54% | 3.06% | 2.28% | 1.18% | 2.91% | 3.72% | 2.95% | 2.63% |
| LAC | 2.45% | 3.42% | 2.84% | 2.95% | 2.28% | 2.61% | 3.96% | 3.45% | 2.84% | 2.39% | 2.92% | 3.05% |
| GB | 2.55% | 2.58% | 2.83% | 2.85% | 3.15% | 2.45% | 2.75% | 2.68% | 3.04% | 3.96% | 2.88% | 2.98% |
| CAR | 2.16% | 4.17% | 3.19% | 2.60% | 2.43% | 2.35% | 2.18% | 3.57% | 2.83% | 2.87% | 2.83% | 2.76% |
| PIT | 3.17% | 2.33% | 2.57% | 3.29% | 3.00% | 2.64% | 2.31% | 2.62% | 2.76% | 3.14% | 2.78% | 2.69% |
| DAL | 2.09% | 3.40% | 2.91% | 3.15% | 2.53% | 2.37% | 3.94% | 2.29% | 3.02% | 1.65% | 2.74% | 2.66% |
| CHI | 1.16% | 2.19% | 2.74% | 2.30% | 3.09% | 3.50% | 3.72% | 3.63% | 3.56% | 0.84% | 2.67% | 3.05% |
| ATL | 2.74% | 2.35% | 1.43% | 2.32% | 3.87% | 3.32% | 2.65% | 2.17% | 2.39% | 3.04% | 2.63% | 2.71% |
| NO | 2.27% | 2.88% | 0.97% | 2.47% | 2.28% | 4.37% | 2.89% | 4.42% | 2.20% | 1.16% | 2.59% | 3.01% |
| Percentage of CarAV expected to be generated based on draft picks used (using Chase Stuart's expected AV draft chart). | | | | | | | | | | | | |
Cleveland utterly dominates the competition for draft capital, with the two top drafts overall (6.05% in 2017 and 7.18% in 2018, which are actually the top two for the last
20 years) and six drafts in the top 10%, while only having two below-average drafts. (The average is 3.125%, which is exactly 1/32nd of 100%). That means that from 2015 to 2018, Cleveland effectively had three extra entire drafts.
It helps to be bad for a long time if you want a lot of draft capital, but how you finish the season doesn't determine everything. Teams that trade away picks for players will lose draft capital, but that's not a bad thing if those players are worth it. Teams like Seattle and New England still got a moderate amount of draft capital (23rd and 24th), despite never getting high (or even middle) picks because they continually traded down for multiple lower picks to increase their total draft capital. Chicago's 2019 draft is the lowest amount of draft capital in the last 20 years, while the Raiders' 2019 draft was the third-highest in the last decade. I'm sure these facts are related somehow.
The Browns are followed in total draft share by the 49ers and Buccaneers. If you only look at the last five years, the top three teams stay the same, and Cleveland becomes even more dominant. They had as much draft capital as the bottom two teams (Philadelphia and Kansas City) combined.
Draft Return
How much return has each team gotten from its draft capital, though? We can answer that by calculating the percentage of total CarAV in each draft class that was produced by each player. Add up all those values for the players a team drafted, and then you have the total draft return for that team (and year). Note that this is not relative to how much draft capital the team had (that step comes next), but is relative to the overall quality of the year being looked at. This is necessary, because not very much CarAV has been generated by players from the last couple of years yet, and we don't want to think that drafts from years ago were better just because the players have been around longer.
Of course, the numbers for recent seasons are going to change a lot in a year or two. Kansas City's 2017 draft is going to keep looking better and better as Patrick Mahomes keeps racking up approximate value and catching up for his "missing" rookie season. Numbers from years ago are a lot more stable and unlikely to change much. These are the teams that actually
did find the most talent in each draft.
| Total Draft Return, 2010-2019 | | | | | | | | | | | | |
| Team | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 10-Yr | 5-Yr |
| SEA | 5.77% | 6.05% | 8.20% | 1.56% | 2.09% | 3.14% | 3.65% | 3.63% | 3.10% | 2.74% | 3.99% | 3.25% |
| BAL | 1.59% | 3.69% | 2.30% | 4.68% | 3.65% | 2.90% | 4.37% | 2.47% | 7.16% | 3.42% | 3.62% | 4.06% |
| CLE | 3.47% | 3.36% | 4.08% | 0.89% | 2.77% | 3.67% | 4.18% | 3.83% | 5.54% | 3.08% | 3.49% | 4.06% |
| SF | 4.48% | 3.86% | 0.74% | 2.74% | 2.89% | 3.39% | 2.47% | 4.08% | 3.69% | 5.98% | 3.43% | 3.92% |
| GB | 4.55% | 3.12% | 2.54% | 5.86% | 3.95% | 2.30% | 3.84% | 3.02% | 2.66% | 2.74% | 3.46% | 2.91% |
| KC | 3.20% | 3.64% | 2.37% | 3.93% | 2.89% | 5.33% | 3.88% | 3.88% | 1.40% | 3.25% | 3.38% | 3.55% |
| NE | 5.45% | 3.93% | 3.89% | 3.18% | 3.12% | 4.73% | 4.34% | 0.71% | 2.07% | 1.54% | 3.29% | 2.68% |
| DAL | 3.22% | 4.26% | 1.75% | 3.49% | 4.42% | 2.47% | 6.43% | 2.37% | 3.32% | 1.03% | 3.28% | 3.12% |
| PIT | 5.96% | 2.91% | 3.06% | 3.95% | 2.71% | 1.87% | 2.28% | 3.83% | 2.29% | 3.59% | 3.25% | 2.77% |
| LAR | 3.29% | 2.05% | 4.22% | 3.44% | 4.77% | 4.17% | 2.32% | 3.53% | 1.70% | 3.08% | 3.26% | 2.96% |
| CIN | 4.42% | 4.60% | 4.67% | 3.35% | 2.33% | 1.62% | 2.89% | 3.13% | 3.47% | 1.88% | 3.24% | 2.60% |
| BUF | 1.56% | 3.62% | 3.84% | 3.75% | 3.33% | 1.91% | 1.52% | 4.33% | 4.06% | 4.44% | 3.24% | 3.25% |
| HOU | 2.87% | 4.34% | 4.29% | 3.29% | 3.30% | 2.33% | 2.40% | 4.18% | 2.44% | 2.91% | 3.23% | 2.85% |
| WAS | 2.25% | 3.48% | 4.32% | 2.14% | 3.71% | 4.38% | 1.56% | 3.28% | 2.14% | 4.96% | 3.22% | 3.26% |
| MIN | 2.14% | 2.36% | 3.98% | 3.15% | 3.80% | 5.76% | 1.18% | 3.88% | 2.07% | 3.59% | 3.19% | 3.29% |
| IND | 2.46% | 1.72% | 4.51% | 1.10% | 1.41% | 2.58% | 3.61% | 3.73% | 7.60% | 2.74% | 3.15% | 4.05% |
| DEN | 5.15% | 4.88% | 3.87% | 1.41% | 2.59% | 2.05% | 3.27% | 1.56% | 3.69% | 2.74% | 3.12% | 2.66% |
| MIA | 3.91% | 2.55% | 4.60% | 2.22% | 3.09% | 3.18% | 3.16% | 2.07% | 3.76% | 2.39% | 3.09% | 2.91% |
| OAK | 2.99% | 2.41% | 1.26% | 3.41% | 6.22% | 2.68% | 1.90% | 1.56% | 3.54% | 4.96% | 3.09% | 2.93% |
| TB | 2.85% | 1.91% | 3.77% | 2.37% | 2.62% | 5.48% | 1.60% | 2.12% | 3.32% | 4.27% | 3.03% | 3.36% |
| NO | 2.83% | 3.43% | 1.59% | 3.52% | 1.53% | 3.14% | 3.69% | 6.65% | 1.55% | 1.71% | 2.96% | 3.35% |
| PHI | 3.77% | 3.29% | 4.67% | 4.47% | 1.59% | 2.15% | 3.92% | 2.17% | 1.40% | 2.22% | 2.97% | 2.37% |
| JAX | 1.29% | 1.26% | 2.02% | 2.54% | 5.63% | 2.51% | 4.07% | 2.87% | 2.80% | 4.96% | 2.99% | 3.44% |
| DET | 3.31% | 0.93% | 2.47% | 5.28% | 3.42% | 2.37% | 3.96% | 3.18% | 2.73% | 2.05% | 2.97% | 2.86% |
| CAR | 2.46% | 3.03% | 3.94% | 3.35% | 3.21% | 3.00% | 1.75% | 4.23% | 2.73% | 1.88% | 2.96% | 2.72% |
| TEN | 2.09% | 3.50% | 2.28% | 1.82% | 3.54% | 2.61% | 4.64% | 3.02% | 2.14% | 3.76% | 2.94% | 3.24% |
| NYG | 2.97% | 1.43% | 1.04% | 2.31% | 3.80% | 3.04% | 2.55% | 2.27% | 4.06% | 5.81% | 2.93% | 3.55% |
| ATL | 2.30% | 3.62% | 0.81% | 2.51% | 3.74% | 3.85% | 3.96% | 1.81% | 2.80% | 2.91% | 2.83% | 3.07% |
| ARI | 2.69% | 3.24% | 2.66% | 4.10% | 2.42% | 3.78% | 0.99% | 1.97% | 1.99% | 4.44% | 2.83% | 2.63% |
| CHI | 1.49% | 1.76% | 1.64% | 3.44% | 3.04% | 2.54% | 4.15% | 3.93% | 3.69% | 1.37% | 2.70% | 3.14% |
| LAC | 2.23% | 2.86% | 2.44% | 3.41% | 0.91% | 3.32% | 2.59% | 4.13% | 3.03% | 1.37% | 2.63% | 2.89% |
| NYJ | 0.97% | 2.93% | 2.21% | 3.35% | 1.47% | 1.73% | 2.89% | 2.57% | 2.07% | 2.22% | 2.24% | 2.30% |
| Percentage of CarAV generated by drafted players (relative to total CarAV in the draft). | | | | | | | | | | | | |
Who has the top season by draft return in the last 10 years? The Seattle Seahawks, of course! The Seahawks came out of the 2012 draft with the best player in the class (Russell Wilson) … as well as the
second-best player in the class (Bobby Wagner). You can see the string of great drafts from 2010, 2011, and 2012 for the Seahawks that propelled them to two Super Bowls shortly thereafter, making Seattle the only team with more than one year above 5%. After Seattle in 2012, Indianapolis and Baltimore have the next two best drafts, both coming in 2018 when they combined to draft Quenton Nelson, Darius Leonard, Lamar Jackson, and Orlando Brown.
Seattle has the top average draft return over the past decade, followed by Baltimore and Cleveland. Baltimore has a good reputation for drafting, but remember that Cleveland has had a huge amount of draft capital, so it would be embarrassing if they were not in the top three. Chicago, the Chargers, and the Jets are at the bottom, which certainly fits with the struggles they've been having. It also shows that Philip Rivers hasn't really had a lot of help in the latter part of his career.
Over the last five years, Baltimore and Cleveland remain at the top, with Indianapolis coming in right behind. Seattle has dropped off a bit, but is still above average. Cincinnati and Philadelphia haven't found a lot of new talent recently, as they drop to the bottom three. But as sure as the sun rises, the Jets remain at the bottom for the last five years as well, without a single year even approaching average.
The most surprising numbers here (although maybe not to Tom Brady) are the last three years for New England. Their 2017 return is the worst for the whole decade at just 0.71%, but their results in 2018 (at 2.07%) and 2019 (at 1.54%) have been terrible as well. That puts New England at sixth-worst in the last five years (they would easily be the worst for the last three years). Has the master lost his touch? Or are there a bunch of young players in New England about to become valuable starters?
Return vs. Capital
So now we come to the true test of drafting ability. How much return did each team get relative to the draft capital it had? We can find out by dividing each team's draft return by its draft capital in each year, then expressing that as a percentage. A score of 100% means that teams got the talent they were expected to get given how much draft capital they had. That's a league-average GM in drafting ability.
| Draft Return vs. Draft Capital, 2010-2019 | | | | | | | | | | | | |
| Team | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 10-Yr | 5-Yr |
| SEA | 132% | 224% | 236% | 77% | 88% | 175% | 117% | 107% | 121% | 74% | 135% | 111% |
| GB | 178% | 121% | 90% | 206% | 125% | 94% | 139% | 113% | 87% | 69% | 120% | 98% |
| DAL | 154% | 125% | 60% | 111% | 175% | 104% | 163% | 103% | 110% | 62% | 120% | 118% |
| PIT | 188% | 125% | 119% | 120% | 90% | 71% | 99% | 146% | 83% | 114% | 117% | 103% |
| NO | 125% | 119% | 164% | 143% | 67% | 72% | 128% | 151% | 70% | 148% | 114% | 111% |
| BAL | 61% | 138% | 85% | 175% | 108% | 94% | 94% | 73% | 185% | 118% | 113% | 113% |
| KC | 70% | 108% | 73% | 94% | 143% | 165% | 142% | 141% | 71% | 185% | 113% | 142% |
| NE | 139% | 95% | 121% | 128% | 123% | 155% | 190% | 60% | 71% | 41% | 112% | 102% |
| WAS | 98% | 102% | 115% | 118% | 154% | 108% | 65% | 103% | 73% | 138% | 108% | 101% |
| ATL | 84% | 154% | 56% | 108% | 97% | 116% | 149% | 84% | 117% | 96% | 108% | 113% |
| MIN | 94% | 76% | 89% | 90% | 104% | 169% | 51% | 172% | 94% | 124% | 106% | 126% |
| HOU | 91% | 129% | 137% | 111% | 69% | 89% | 87% | 142% | 127% | 98% | 106% | 108% |
| IND | 108% | 68% | 96% | 54% | 117% | 104% | 122% | 111% | 146% | 83% | 105% | 117% |
| CAR | 114% | 73% | 123% | 129% | 132% | 128% | 80% | 119% | 97% | 66% | 104% | 99% |
| MIA | 135% | 114% | 117% | 50% | 108% | 108% | 100% | 86% | 118% | 119% | 103% | 106% |
| CHI | 128% | 80% | 60% | 149% | 98% | 73% | 112% | 108% | 104% | 162% | 101% | 103% |
| PHI | 90% | 104% | 128% | 117% | 55% | 90% | 131% | 69% | 111% | 83% | 98% | 95% |
| DEN | 130% | 102% | 129% | 58% | 123% | 83% | 131% | 53% | 86% | 81% | 98% | 85% |
| BUF | 44% | 83% | 101% | 98% | 94% | 129% | 54% | 162% | 107% | 130% | 97% | 115% |
| DET | 86% | 33% | 86% | 143% | 101% | 93% | 128% | 111% | 99% | 61% | 95% | 97% |
| OAK | 79% | 99% | 121% | 102% | 171% | 68% | 61% | 63% | 101% | 90% | 94% | 79% |
| LAR | 67% | 70% | 87% | 86% | 94% | 125% | 71% | 148% | 103% | 131% | 94% | 114% |
| CIN | 132% | 116% | 105% | 94% | 91% | 50% | 113% | 81% | 100% | 53% | 94% | 78% |
| SF | 116% | 110% | 35% | 73% | 74% | 100% | 63% | 92% | 101% | 140% | 93% | 100% |
| NYG | 98% | 52% | 43% | 84% | 117% | 98% | 77% | 91% | 89% | 120% | 90% | 97% |
| LAC | 91% | 84% | 86% | 116% | 40% | 127% | 65% | 120% | 107% | 57% | 90% | 95% |
| ARI | 104% | 85% | 108% | 105% | 81% | 141% | 52% | 64% | 71% | 85% | 90% | 84% |
| JAX | 53% | 47% | 60% | 59% | 127% | 63% | 114% | 77% | 126% | 137% | 87% | 101% |
| TEN | 76% | 95% | 81% | 49% | 123% | 58% | 96% | 77% | 105% | 137% | 87% | 90% |
| TB | 62% | 67% | 111% | 110% | 78% | 126% | 47% | 78% | 78% | 111% | 86% | 90% |
| CLE | 84% | 96% | 80% | 39% | 61% | 77% | 80% | 63% | 77% | 164% | 78% | 81% |
| NYJ | 47% | 164% | 85% | 77% | 42% | 51% | 108% | 72% | 68% | 66% | 74% | 71% |
| Percentage of CarAV generated by drafted players (relative to total CarAV in the draft), divided by percentage of expected CarAV from draft picks used. | | | | | | | | | | | | |
After adjusting for the amount of draft capital used, the Seahawks remain in the No. 1 position. By a mile. At a 135% average over the last decade, they are 15% higher than the next two teams, Green Bay and Dallas. Schneider wins!
(Although Carroll deserves some credit for developing those players, of course.) The Seahawks have the two best drafts of the last decade (2011 and, of course, 2012). Even if you only look at the last five years, they are still doing well at tenth in the league (Kansas City and Minnesota are now on top, while Dallas remains in third). No, 2019 wasn't good at 74%, but when Marquis Blair, L.J. Collier, and Travis Homer all become starters, it will start looking much better, right?
If you look at the top teams on this list, you see a lot of well-run organizations with a lot of stability: Seattle, Green Bay, Dallas, Pittsburgh, New Orleans, Baltimore, Kansas City, New England.
It seems like Dallas has done a great job of drafting, but they haven't really had the success these other teams had. Maybe their coaching wasn't that good? We can also see how badly New England has drafted in the last three years. It doesn't look quite as bad as the raw numbers, but it's still very bad, easily the worst in that time. But over the last five years, they are right about at average, so maybe things aren't that bad?
At the bottom of the list you have Tampa Bay (86%), Cleveland (78%), and the Jets (74%). Jets fans are not surprised, I'm sure, with only two above-average drafts in the last decade but three in the bottom 10%. They are at the bottom of the list for the last five years as well. Tampa Bay has struggled in the draft, not getting what they should have out of their third-best draft capital, while Cleveland has squandered the best draft capital of the decade. At least the 2019 draft is looking good for Cleveland, but that's only because they had very little draft capital (very unusual for them). The actual raw return is still below average. The factory of sadness is still open for business.
Consistency and Variance
But does this show that particular teams are actually better at drafting over time than others? At first glance, the results still look awfully random. The year-to-year correlation for each team's return vs. capital results in Table 3 is just 0.081 -- there is practically zero relation between any team's results from one season to the next. This would imply that good or bad drafting results can be attributed almost entirely to luck. However, things change ever so slightly if we look at the big picture. If we compare any team's draft results in any given season to their average draft results of their
other nine seasons, we get a correlation of 0.122. If we use the median of the other nine years instead of the average -- which should limit the impact of outliers like Seattle in 2012 -- we get an even stronger correlation of 0.145. This suggests that while the data is still mostly random noise, there is some faint evidence that some teams really are better (or worse) at drafting talented players than their competition. (A future expansion of this analysis might look at correlations by administration, rather than by team, to account for teams that have changed front offices over the past decade.)
We can also analyze variance by using an ANOVA test. The short-hand explanation of an ANOVA test is that we are comparing how much variation we see between teams compared to how much variation we see for each individual team over the ten years in the sample. The math for this is a little complicated, but in the end, this gives us a p-value of 0.034 (based on a generic F-distribution) for the chance that the variation between teams in this data set is due to randomness (about a 1-in-30 chance). This passes the generic 0.05 threshold for statistical significance, but not by a lot. Still, this is moderately strong evidence that drafting is not completely random.
If you are familiar with ANOVA tests, though, you might immediately be worried because using a generic F-distribution to get p-values can be a problem if your distribution is not fairly normal (or if the standard deviations within each group are too different, but that's not the case here). So what does our distribution look like? If we look at a histogram of the Z-scores, we can see how things get skewed by the top 20 or so drafts. The purple bar is close to a 0.0 Z-score, while the different shades of blue represent about a standard deviation each. You can see the 2011 and 2012 Seahawks drafts on the far right, along with the 2013 Green Bay draft. There is definitely a strong tail to the high end, but overall the distribution is moderately normal in shape.
To be safe, we can run the ANOVA test using the medians (of each team, along with the overall data set) instead of the means when calculating the variances at each step. Again, doing this somewhat neutralizes the effects of outliers, particularly when the outliers are biased in a particular direction. We when do this, we get a p-value of 0.000072, which corresponds to over a 1-in-13,000 chance that the variation between teams is entirely due to randomness. This is pretty close as we could ever expect to get to a smoking gun telling us that there is real skill in drafting.
A reasonable interpretation of all this is that while there is a good amount of randomness in how well teams draft, over the course of a decade, the skill of the drafting GM is fairly important in how well a team does. Of course, one could also make the case that the dominant factor here is the coaching staff's ability to get the most out of the players who are drafted. It's likely to be a combination of the two.
So clearly, the Seahawks are just vastly more skilled than other teams at drafting over the last ten years, right? As much as I would like to believe that, there is a reason the p-value is only a little under 0.05 when you don't attempt to neutralize unusual outliers. The fact that the p-value drops so much when those outliers are neutralized likely means that those outliers have a lot of luck in them. But even then, how could the Seahawks get the top two drafts of the decade, back-to-back, without that being evidence of incredible skill in drafting (rather than just good skill and a heavy dose of luck)?
Well, when you have a skewed distribution, there is usually a reason why, and that reason is right in front of us. When we use CarAV as our measurement of value, it inherently means that the best players have more upside than the worst players have downside. If you get a Russell Wilson or a Tom Brady, their value is massive due to positional value, longevity, lower risk of injury (especially in the modern NFL), etc. But if you draft a JaMarcus Russell, there is only so much damage he can do. If the Raiders had been forced to start Russell for every game for a decade, then he might be a Tom Brady-level outlier. Instead, he gets benched and the damage is limited to just wasting a draft pick and having a bad season or two.
What this all tells me is that drafting well is a lot of luck, mixed with some skill and an extra layer of a random "jackpot" on top (the one or two later-round picks each draft that become unexpected Hall of Famers). This would explain the data we see (including the outliers) pretty well. The Seahawks are probably pretty good at drafting, but also had some crazy luck in hitting three jackpots in a row (Wilson, Wagner, and Richard Sherman). What this should tell NFL teams is that you need to roll the dice as many times as you can (trading down for additional value whenever possible), get the best GM you can possibly find,
and get the top coaches in the league to develop the talent you draft -- which is what we already see consistently good teams generally do.
Benjamin Ellinger is a program director at the DigiPen Institute of Technology. He teaches programming and game design following a long career as a game developer.
www.bloggingtheboys.com
