It is looking more likely by the day that there will soon be a 4-team playoff to determine the national champion for college football. One of the details that is still being determined is how the 4 playoff teams will be selected, described in this SI article. There are 2 interesting issues raised by this article that I want to address.
1. Computer programs are currently used in the BCS system to determine which 2 teams will play for the national title. The problem is that "computer programmers ... refuse to reveal the formulas that determine their rankings". One of the current hot topics in statistics is reproducibility of research - when you get your work published in a journal, you need to describe your methods so that anybody else reading the article can reproduce the results. If you don't reveal your method, then other researchers are not able to accurately evaluate your work and your conclusions become suspect. How can we trust in a ranking where we don't know the input variables and model? This is especially relevant in this context, as there is no way to evaluate the rankings (i.e., there is no "true" ranking that we can compare model performance with).
If the authors are worried about others stealing their work, all they have to do is file for a patent/copyright. If that is not the issue, I am left to believe that they are worried that others will improve upon their model and obtain "better" results. That doesn't instill much confidence. And because no one knows the model, how can we be sure that the model isn't tweaked each week for someone's favorite team to move up the rankings? (I guess the BCS is overseeing things to make sure that this isn't the case, but who knows).
2. The article supports evaluating and selecting playoff teams only after the entire regular season is finished, rather than ranking teams after each week: "Because committee members ... would evaluate the entire body of work, schools will be more apt to schedule quality out-of-conference opponents." What a novel idea ... NOT! One of the major rules in designing experiments is that you cannot modify your experiment half way through because you do not like the preliminary results (exception: cancelling a clinical trial resulting in many deaths). You have to wait until you have all of the data before testing a hypothesis. Games are played to determine the best team on the field, so let's collect all of the results before trying to determine the final rankings. Otherwise, the preseason polls are very likely to be the tiebreaker between evenly matched teams, which has absolutely nothing to do with performance on the field.
For example, image that all of the preseason top 5 teams go undefeated. After each week, none of the teams have a reason to slide down the rankings because they all won. Similarly, it is tough for the team ranked #5 to jump ahead of the other teams that also did not lose. From week to week, voters tend to assume that the previous ranking is truth, and so need exceptional evidence to move one team above another. This also requires the voters to admit that their previous ranking was incorrect, and who wants to admit they are wrong?
With this post, I am officially throwing my hat into the ring of choosing the 4 college football playoff teams. I promise to wait until the end of the season, when all of the data has been collected, to make my selections (even if this means not watching ESPN during the fall when they update their playoff teams every hour). If I use a model to select the teams, I will be fully transparent, allowing everybody access to the methods I used, so that my results can be replicated. Because some may not agree with choices related to my model, I would also be prepared to defend my model by evaluating its performance using previous years' data or using some other metric. Finally, because I am a new PhD graduate, I would be willing to accept a discounted salary compared to other BCS executives (I'm thinking about $100k would be fair for the month I would be working each year).
No comments:
Post a Comment