Confidence vs. Statistics: The Directed Network Effect of Wins on the NCAA Tournament John Ezekowitz Patrick Mannion Francis Thumpasery Monday, March 21, 2011 Economics 970jl Jenn Larson
The purpose of this analysis is to determine the factors that are responsible for a college basketball team’s success in the NCAA tournament. Success here will be measured in terms number of NCAA Tournament Wins, which ranges from 0 to 6. NCAA Wins = 0 indicates that a team was eliminated in the first round of the tournament bracket, while NCAA Wins = 6 indicates the single team that is the winner of the championship. To create a model of the likelihood of NCAA wins taking values {0-6} based on a team’s characteristics, we perform an ordered probit regression of team characteristic variables on NCAA Wins. These independent variables to be considered are Adjusted Offensive Rating (AOR), Adjusted Defensive Rating (ADR), and Consistency. AOR and ADR measure how many points per 100 possessions a team scores and allows, respectively, while controlling for opponent strength. Consistency is a measure of a team’s variance in point spread at the conclusion of a game, i.e. winning by 5 = +5, losing by 20 = -20. This can be used as a measure of a team’s in-season strength because under the assumption that the teams being analyzed are winning teams (fair for the NCAA tournament bracket), consistency represents a measure of variance of performance. There are no “consistently bad” teams in the NCAA Tournament. It has been hypothesized that being more consistent (having a lower consistency number) is better for single-elimination tournaments like the NCAA Tournament. High variance teams may win games they are not favored to win, but also may lose those they were supposed to win. We incidentally examine this hypothesis here. Additionally, simple winning percentage does not account for the margin of victory. The use of consistency and metrics that account for the number of possessions (“tempo-free metrics”) like ADR and AOR instead of simple winning percentage is an accepted standard of the sports analytics community, so we will not go into depth defending its use in this analysis. Finally, Strength of Schedule (SOS) is an index of the strength of opponents played by the team in-season. Additionally, because variables like seed and strength of schedule are correlated with other predictors, we included (and tested) interaction terms between variables. Specific predictors such as a team’s free throw percentage and rebounding numbers were
included in early models. Due to the high covariance between these terms and AOR/ADR, however, the specific predictors were dropped for the final model in favor of the more accurate catch-alls. Also included was a dummy variable indicating whether or not the team was a mid-major, or a team that did not play in a major conference that usually sends many teams to the NCAA Tournament. This dummy was intended to capture the effects of potential opportunity bias: mid-major teams have fewer opportunities to play major conference opponents, who make the tournament more often. We ran two types of regressions, one ordinary least squares (OLS) and one ordered probit. The OLS model assumptions are not particularly well-suited for this regression: the dependent variable is ordinal and has only seven possible outcomes. OLS (and LPM) use the standard normal distribution and can produce spurious predicted probabilities that are less than zero or greater than one. Ordered probit, on the other hand, uses the normal cumulative density function and includes “cut-offs� for the dependent variable (in this case the number of NCAA Tournament wins). The downside of the ordered probit is that the coefficients are very hard to interpret. Instead of focusing on magnitude of effects, we will look at significance and direction. The interaction terms between SOS and AOR and ADR were both statistically significant at the one percent level. As SOS*AOR increases, the odds of winning more tournament games increase. This makes sense: the more points per possession a team scores against tougher competition, the better they should do in the Tournament. Similarly, as SOS*ADR increases, the odds of winning more Tournament games decrease. This, too, is intuitive: as a team allows more points against weaker competition, they should be expected to do worse in the NCAA Tournament. Finally, the Consistency metric was a significant predictor at the five percent level. This regression seems to support the above hypothesis: as consistency decreases, the expected odds of winning NCAA Tournament games also decrease. These results were not unexpected. They tell us that the most important factors for winning NCAA Tournament games are Strength of Schedule, offensive and defensive efficiency, and consistency.
In addition to the standard team-specific factors that help describe a team's success in the NCAA tournament, we wanted to identify a network-based explanation for why some teams thrive while others fail. The specific question addressed by this network will be whether or not playing and beating stronger NCAA tournament teams during the season increases a team’s likelihood of success in the NCAA tournament? Does psychology play a factor in NCAA runs, or are a team’s expectations of being an underdog or sure winner irrelevant? The answer to this question might seem obvious, but perhaps not in light of the use of controls included in the original regression. We defined a directed network with teams that make the NCAA tournament as nodes and a link defined as a previous in-season win in a game between two tournament teams. Thus, if Butler and Georgia were both in the tournament and Butler beat Georgia during the regular season, there would be a link from Butler to Georgia reflecting that relationship. A notable characteristic of this directed network is its transitivity. This is critical because the NCAA tournament committee actively avoids creating games between two teams who have already played each other, so the network factor measures the confidence that comes from having beaten “similar teams.” It is easy to see how a team might relate their next opponent to a previous victory in a transitive manner, however, i.e. “We can surely beat Penn since we beat Princeton, and Princeton beat Penn.” With the added understanding of the network’s transitivity, the network accounts for the psychological factor of having beaten teams at least as good as those you will face. Our premise is that the specific network consisting of NCAA tournament teams provides insights into how teams within the tournament are affected by their history of interactions with top-tier teams. General statistics provide insight into how a team has performed against all basketball programs they have faced, but more subtle psychological factors can be measured by a network approach. For example, a team that comes in having seen and beaten 8 other teams in the tournament would have more confidence in the tournament than another team that had only beaten 3 tournament teams but had
superior statistics for the regular season. A networks approach is the best way to access intangibles such as the impact of being a favorite or underdog and the power of familiarity and comfort in the NCAA tournament. There is no existing data about this sort of network, so we used three years of regular season schedule data to develop a network in a way that we could draw conclusions from. We looked at the NCAA tournament teams from 2008, 2009, and 2010, copied their schedule with the outcome of each game into Excel, and created a 65x65 matrix for each year. When read down the columns, the matrix shows a team’s wins, and when read across columns, its losses. In this way, we were able to produce a directed network that demarked successful "relationships" or games. Using this matrix, we could identify the number of wins that a team had during a regular season against tournament teams, allowing us to calculate a variable we called "Weighted Wins", which took the number of wins and weighted them based on the seed of the opponent. Specifically, we used a weighting scheme based on the inverse of the defeated team’s seed number. The seed number is a value ranging from 1 to 16, with each value being taken by four teams, where lower seeds represent stronger teams. Thus a win over a number 1 seed (a top team in the tournament) was worth one, while a win over a 16 seed was worth 1/16. The variable weighted wins, then, was a sum of these values for each individual team. This makes weighted wins a weighted measure of degree and of centrality in the network, where degree is measured as the number of links leaving a node, and centrality is simply a monotonic transformation of degree (degree ÷ number of nodes). The formula for this measure of degree centrality, weighted wins, is as follows for each individual team: Where d(j)=1 if team j was beaten, and 0 if team was not beaten or if j=i, and s(j) is the seed number of team j An example of the depiction of such a network can be found in Figure 1, which shows 1through 8-seed teams in the 2011 NCAA tournament as nodes, shows the directed links (wins), and
weights the size of the nodes based on their seed number, where lower seed numbers carry heavier weights. When we regressed this variable with a number of other more traditional team-specific predictors against the NCAA Wins success measure, we found that this network-based variable was significant, indicating that there is a role for network effects within a model of NCAA tournament results. This effect can be attributed to the network’s capture of the psychological effects of big inseason wins and to its transitivity. This allows a team like Butler and their defeat of Georgia to have an impact on its performance that goes beyond that single link. Concretely, if the 2011 Butler team, an eight seed, had beaten an additional No. 1 seed in the regular season, then their odds of making the Sweet Sixteen (winning two games), holding all other factors constant, increase by almost 5 percent. We attribute this to psychological or otherwise intangible benefits that come from having played and beaten good competition during the regular season. Many basketball experts often opine on the importance of “having been there before” and “not being afraid to win” in the NCAA Tournament. By defining our directed network, we are able to access these psychological factors in a quantitative manner that has real predictive power. Of course, our model is not perfect. Sticking with the Butler example shows why: last year, Butler made a surprising run to the Championship Game of the NCAA Tournament, winning five games. This year, they returned almost every player from that team. Because returning minutes data showing what critical players remain key players the next year are unavailable for college, our model has no way of accounting for potential psychological or intangible benefits derived from having played in the NCAA Tournament before. Adding this data would help to better define the nodes of the network and allow for a more complete intangible picture of a team. Additionally, because Butler plays in a “Mid-Major” conference that usually only gets one team into the NCAA Tournament, they have fewer chances (exclusively at the beginning of the season) to get weighted wins against future NCAA Tournament teams. We did test the interaction between weighted
wins and a Mid Major dummy variable as there was a significant negative correlation between the two, but an F test found the interaction terms to be not quite jointly significant. It is probable, however, that if we had enough time to extend our data gathering to more years and increase our sample size that the opportunity bias of playing in a smaller conference would become a significant factor in determining the predictive power of weighted wins. Thus, by defining our directed network, we are able to access these psychological factors in a quantitative manner that has real predictive power. We conclude that above and beyond the predictive power of a team’s on-paper statistics, there is indeed a psychological network effect based on the confidence of having beaten a greater number of more highly ranked NCAA tournament teams during the regular season.
Table 1. Predictors of NCAA Tournament Success Dependent variable: Regression method: Regressor: Adj Ortg Adj Drtg Consistency SOS*Adj Ortg SOS*Adj Drtg Strength of Schedule
(1) NCAA_Wins OLS
(2) NCAA_Wins Ordered Probit
(4) NCAA_Wins OLS
(5) NCAA_Wins Oprobit
-0.0711 (0.0725) 0.158* (0.0693) -0.0849** (0.0247) 0.280* (0.105 -0.461** (0.105) 11.41 (13.965)
__
-0.159* (0.073) 0.156* (0.074) -0.0867* (0.0279) 0.403** (0.093) -0.445** (0.099) __
__
__ -0.0756* (0.0292) 0.2024** (0.021) -0.248** (0.031) __
Weighted Wins Intercept
0.255* (0.123) 5.502* (2.74) __
__ -0.0685* (0.0297) 0.213** (0.0224) -0.252** (0.032) __ 0.227+ (0.136)
z-cutoff #1
3.385 (2.361) __
z-cutoff #2
__
z-cutoff #3
__
z-cutoff #4
__
z-cutoff #5
__
z-cutoff #6
__
F-Test on SOS*AdjO, SOS, Adj Orating
26.16 (p<0.0001)
__
__
__
F-Test on SOS*AdjD, SOS, Adj Drating
29.5 (p<0.0001)
__
__
__
R^2 (or Pseudo R^2)
0.53 168
0.22 168
0.55 168
0.23 168
-1.808609 (.858) -0.6815219 (0.872) 0.1174 (0.24145) 0.802 (0.8517) 1.39 (0.8701) 1.979 (0.855)
__ __ __ __ __
-2.009** (1.208) -0.884 (1.215) -0.0800 (1.214) 0.568 (1.184) 1.181 (1.184) 1.789 (1.18)
Notes: Standard errors are reported in parentheses below coefficients, and p-values are reported in parentheses below F-statistics. Standard errors for regression (1) are heteroskedasticity-robust. Coefficients are individually statistically significant at the +10%, *5%, **1% significance level.
Figure 1: Directed network of in-season wins between 1-8 seed 2011 NCAA Tournament teams, with node size weighted by seed (lower seeds larger).