Ranking the Rankers: Introduction

Ranking the Rankers: Introduction

Finding the Best NHL Draft Rankings

By David Venturi

June 21, 2016

View this entry at Blue Steam Hockey

"Ranking the Rankers" is a 7-piece series that quantifies past NHL draft rankings from various prognosticators based on player performance to date. View code

Part 1: Introduction

Your favourite hockey team just drafted a teenager. He, presumably, is good at hockey. How can you be sure? Because this set of draft rankings from [insert writer / scout / website here] says so.

There is no shortage of NHL draft rankings. Bob McKenzie, Corey Pronman, International Scouting Services (ISS Hockey), and Craig Button, among several others, put out carefully curated prospect lists year after year. Their lists vary. Their lists are criticized because they vary (some more than others). The draft passes and there isn’t much analysis of who ranked well, qualitative or quantitative. The public’s perception of NHL draft rankings is coloured with bias because of the above cycle.

This series attempts to determine which of the above sets of draft rankings is best at predicting future NHL success (and which one you should hope your team follows on draft day).

Measuring Player Performance

A catch-all-statistic called Point Shares (PS) is used as the measure for player performance. An excerpt from Rob Vollman’s Hockey Abstract:

It’s very useful to be able to summarize all of a player’s contributions, both offensive and defensive, into a single catch-all statistic. This catch-all statistic can be used to compare players of different types, across different teams, eras or leagues.

Point Shares attempt to quantify how many standing points a player contributes to his team. You can read more about the statistic here and here.

Though other catch-all-statistic alternatives exist, Point Shares are used because they are well-suited for comparing across seasons and historical data is easily accessible through Hockey Abstract.

Gathering Draft Rankings

Only top-30 rankings are used in this exercise. Projecting prospects in rounds 2-7 is a crapshoot.

Five sources of NHL draft rankings are analyzed in this series:

  1. The actual NHL draft order
  2. Bob McKenzie (TSN)
  3. Corey Pronman (ESPN Insider)
  4. International Scouting Services (ISS Hockey)
  5. Craig Button (TSN)

The actual NHL draft order acts as a proxy for the rankings of NHL team scouts and general managers. Bob McKenzie’s lists are based on the consensus opinion of a group of scouts that he trusts most.

2014 and 2015 NHL draft rankings are not used for the 2016 edition of this series. There isn’t enough NHL data on the prospects drafted in those years to perform meaningful analysis.

Draft rankings from 2010, 2011, 2012, and 2013 are used for sources 1-4. I could not find Craig Button’s 2010 list despite an epic Google search quest. Finding rankings pre-2013 was surprisingly difficult.

Measuring Scout Performance

After wrangling Rob Vollman's historical Point Shares and time-on-ice (TOI) data, Hockey Reference's draft history, and each prognosticator’s draft rankings, the following number-crunching is performed:

  • Sorting each year’s set of drafted prospects by career Point Shares/60 minutes of playing time and applying a "PS/60 Rank". Minimum TOI cutoff is 100 minutes. If a player fails to make this cutoff, they are assigned a tied-for-last ranking. Example:
Player
(2011 Draft)		 Career PS/60
Rank
TOI (min) PS PS/60
Nikita Kucherov 3310.33 21.0 0.381 1
Johnny Gaudreau 2991.87 17.0 0.341 2
Ondrej Palat 4043.33 21.3 0.316 3
. . . . . . . . . . . . . . .
Tyler Biggs N/A N/A N/A 85
  • Calculating the absolute difference between each source's prospect rankings and that player’s "PS/60 Rank". Example:
Player
(2010 Draft)		 Source
Rank		 PS/60
Rank		 Absolute
Difference
Taylor Hall 1 6 5
Tyler Seguin 2 2 0
Erik Gudbranson 3 60 57
. . . . . . . . . . . .
  • Calculating the sum of the "Absolute Difference" column for each list

The smaller the total absolute difference for each year, the better that set of draft rankings performed. By comparing each prognosticator's average performance from 2010-2013, we get an idea of whose NHL draft rankings are best at predicting future NHL success.

Limitations

This exercise has two main limitations:

  1. Point Shares. Catch-all-statistics in hockey are still in their infancy compared to statistics like baseball’s Wins Above Replacement (WAR). Though Point Shares are not perfect, they have a track record of identifying performance and are likely the best option for this exercise.
  2. TOI sample size. Players drafted from 2010-2013 are at most six years into their NHL careers and are entering their prime. The 100-minute TOI cutoff generates some funny "PS/60 Ranks", i.e., Josh Leivo and his 20-goal pace through 28 games are ranked ahead of Mark Scheifele and Dougie Hamilton for the 2011 draft crop. Also, only 35 players from 2013 make the TOI cutoff. This noise must be considered when drawing conclusions. The playing field for scouts is level, however. There is also something to be said about the ability of players that make it to the NHL faster.

The Results

Links to each "Ranking the Rankers" piece will be hyperlinked below when complete.