Photo: Jim Brown, USA Today Sports

Let’s talk about science.

You may not have heard, but science today is grappling with something of a reproducibility crisis. Over the last several years, it has become more and more apparent that a shockingly large number of its findings are not consistent when experiments are run a second time.

The reasons for this are easy to point out. Because of how tests for statistical significance typically work, we would expect scientists to find a “statistically significant relationship” where there really was none entirely by chance in around one out of every 20 experiments.

The problem is that more than 1/20 of the juiciest findings we read about are going to suffer from this flaw, though. Why? Simply put, science has a publication bias. Prestigious journals crave to publish surprising, new, and unintuitive findings. Nobody wants to run a story that, yes, people really do get mad when you poke them with a stick, (p = 0.001).

Studies in the 1/20 “false positives” bucket are dramatically more likely to get published than the 19/20 “true negatives”, so they’re disproportionately represented in the literature. Additionally, scientists rarely gain prestige by double-checking someone else’s work, (and journals are rarely interested in publishing negative results), so some of these errors aren’t found for years— if ever.

Here at Dynasty, in Theory, we’re committed to doing the hard, unglamorous work of Real Science(TM). Two years ago, I wrote about how preseason ADP predicted late-season performance better than performance in the first four weeks. Last year, I re-ran the test and found that, for 2013, both preseason expectations and early-season performance carried roughly equal predictive power.

As you may have guessed, it’s “Revisiting Preseason Expectations” week. Let’s run the results for 2014!

Methodology:

Once again, I have compiled a list of the top 24 quarterbacks, 36 running backs, 48 wide receivers, and 24 tight ends according to MFL’s 2014 preseason ADP. From that list, I have removed any player who missed more than one of his team’s first four games or more than two of his team’s last twelve games. As always, we’re looking by team games, rather than by week, to account for players with a week 4 bye.

I have used standard scoring, because that was the easiest for me to look up with the databases I had. For the remaining players, I tracked where they ranked at their position over the first four games and over the final twelve games. Finally, I’ve calculated the correlation between preseason ADP and stretch performance, as well as the correlation between early performance and stretch performance.

Here’s the data:

Quarterback

The average delta between preseason ADP and stretch performance was 5.26 spots. The correlation was 0.422.

The average delta between early performance and stretch performance was 7.74 spots. The correlation was -0.019.

Running Back

The average delta between preseason ADP and stretch performance was 9.85 spots. The correlation was 0.568.

The average delta between early performance and stretch performance was 13.30 spots. The correlation was 0.472.

Wide Receiver

The average delta between preseason ADP and stretch performance was 17.41 spots. The correlation was 0.333.

The average delta between early performance and stretch performance was 18.67 spots. The correlation was 0.477.

Tight End

The average delta between preseason ADP and stretch performance was 12.06 spots. The correlation was -0.051.

The average delta between early performance and stretch performance was 12.31 spots. The correlation was 0.416.

Conclusions

In 2014, preseason ADP was much more predictive at quarterback, slightly more predictive at running back, slightly less predictive at wide receiver, and much less predictive at tight end. Over all four positions, the correlation between preseason ADP and stretch performance was 0.466. The correlation between early performance and stretch performance was 0.560.

All correlations are down dramatically from the aberrationally high ones of 2013 and more in line with the ones I found from 2012 and before. 2014 also marked the first time where, over all positions, early season performance was significantly more predictive than preseason ADP, (though it still varied wildly on a position-by-position basis).

I'll be interested to see next year whether 2014's performance was a trend or whether, like 2013s unnaturally high correlations, it was merely a 1-year aberration. Either way, I believe the data still supports the conclusion that four games is typically the tipping point around which early performance and preseason ADP tend to carry similar predictive weight.

One last note must be made; because I am excluding players who missed substantial time, these data are subject to selection bias. Players who lose their job due to underperformance rather than injury will not be counted, and they would surely drive the correlations for early-season performance up. As an example, consider Johnny Manziel, who was drafted on average as one of the top 24 quarterbacks, but who was not counted in these results because he couldn't win the starting job.  

I've included all of the data used, so please feel free to peruse at your leisure and draw your own conclusions.

Follow @AdamHarstad