Benchmarking Bob McKenzie, Draft Projection Variability

With the final month of the regular season underway, realities are sinking in, playoff bubbles are bursting, and playoffs droughts are ending (I hope...). For teams like Florida, Tampa, Calgary, and Colorado, the 2013 draft couldn't come soon enough.

Speaking of the draft, I try to make it a regular routine to do some reading and podcast-listening on prospects over the weekend. In browsing some of the new projected rankings for the upcoming draft in New Jersey, I came across TSN's archive of Bob McKenzie's draft projections for the past 9 drafts.  His methodology involves an annual survey of NHL scouts, asking them which players they think will be taken in the first round. The final rankings are then determined by a consensus of opinion, and does not consider the order of teams, or team needs. 

"86%, bitches..."

The site claims that in the last nine years, "McKenzie is 231/270 in terms of projecting first rounders (~86%)", and after looking at the data myself, their claim is true. Now, 86% is pretty accurate and I'm not about to sling mud at McKenzie's record.  However, if we define accuracy as the degree to which actual picks deviate from the projections, how accurate is McKenzie, really?

In other words, TSN and I are usually two different definitions of accuracy. In McKenzie's case, of the 30 names he projects as first rounders, usually 86% of those names end up being picked in the first round. The accuracy I'm trying to get at is to see by how much the projections vary in their final position on the board. For those of you who are familiar with rudimentary statistics, what I'm basically looking at is the standard deviation of the projections at each position.

*(Ignore the next paragraph is you're already familiar with standard deviation)*

I don't intend for this to be a stats lesson but briefly, what standard deviation shows is how much variation exists from the mean, or average. For the mean, I took the average of the eventual pick position for each projection.  Finally, to determine the standard deviation, I squared the cumulative differences between the actual draft positions and the mean, divided the squared cumulative differences by the sample (9 drafts), and found the square root of that.  All you need to know is that a high standard deviation indicates that the data points (actual draft position) are spread out over a large range from the mean (projection), and a low standard deviation indicates that the data points tend to be close to the mean.

First, let's take a look at how McKenzie has fared. As the chart below indicates, over time, more and more of his projections have ended up as first rounders. In 2012, he matched his highest accuracy rates (90%) from the 2008 and 2009 drafts.  On the other hand, the two years prior to that , 2010 and 2011, have been below average (84%), but not by much. 

The chart below shows the standard deviation of each projection, at every position from 1-30. Assuming a normal distribution (bell curve ring any bells..?), the way to read this is, for example, 68% of the eventual positions of the projected 10th overall pick usually land +/- 4.6 spots from the projected position.  

What's interesting about the chart is that up until the 20th overall pick, deviations from the projections remain in a pretty tight band, ranging from 0 (1st overall) to 7.6. (20th overall). After that, things gets a little wacky and eventual picks deviate big time from their projections. Assuming the Leafs do not capture a division title, or the Stanley Cup this year (playoffs, sure - but let's be realistic), Nonis and Co. could be looking at a first round pick ranging from 14th-24th overall. Luckily for Leaf fans, the deviation from 14th-24th also happens to be lower, on average, than from 25th-30th. However, I'm willing to bet that no matter which pick the Leafs end up with, the TSN projection at that position will likely not be the name Nonis calls out at the Prudential Center this summer.

If you fancy yourself a draft prognosticator, or want to see how you fair against the insider of all insiders, feel free to use this as a benchmark of sorts for your own projections.

As always, feel free to leave any comments or questions below or hit me up on Twitter (@Mapleleafmuse)