76 replies to this topic

[NorEastern]

Yesterday the founder of Wages of Wins wrote an article titled "The Standard Small Sample Warning!". This man does have a PhD in stats and is respected by basketball analytic authors around the globe (but hated by some but that is another story.) A few quotes:

"It really is going to be a few weeks before we can start to make sense of the 2013-14 season.

And that means we need to keep this disclaimer in mind: Just because you see some numbers right now, it does not mean those numbers can tell us much. Or as I tell my students, sometimes the answer is “I don’t know.” We are often reluctant to say this. But when the samples are small, that is often the best we can do."
 

Those statements should put this entire discussion to rest. So I guess my position "I don't know yet and neither can you" is correct. You cannot draw any truly meaningful conclusions from the current sample set. But it is a free world and I guess you can jabber away all you want. When will be the appropriate time to start drawing inferences from the advanced stats? Ideally one waits for 29 games to be on the books. But one has to at least wait until the PhDs start dipping their toes into the data.

 

http://wagesofwins.c...sample-warning/

 

Timetodie: No. The argument is over the influence of small sample size.

j_wehr: No I will not be PMing you. I learned a long time ago that one can talk sense all day long to a brick wall. It will never make that wall knowledgeable.

[timetodienow1234567]

Okay. I just think all advanced nba statistics are flawed and the key is to be the least flawed. Small sample sizes are to be avoided at all costs.

[j_wehr]

How about responding to my actual point? I'll make it very precise for you:

What percentage of players who finish the season with a PER at or above 24 start the season with a PER at or below 18.3 through 204 minutes or fewer? I don't have the data to answer the question, but I'm confident that it's pretty tiny.

Edited by j_wehr, 08 November 2013 - 10:07 PM.

[Rahat Huq]

P.S. - This goes to everyone but especially NorEastern: When you disagree with me, please try to disagree with my statements rather than criticizing my "knowledge" or "understanding." It's really irritating when you make it personal. I do not want the discussion to devolve into citing credentials in some sort of power play.

 

I need to second this.  

 

This is a very interesting discussion, but could we please stick to the substance of Justin's posts rather than attacking his credentials?  He's an economist by profession so that shouldn't even be in question.  

[Mason Khamvilay]

I'm pretty lazy and the quality of my posts suffer because of it, but for those who want to raise the quality of their posts here's my advice: Try to write articles, or think of your posts as mini articles. Attacking someone's credentials is something I never see good writers do so I never do it. 

[NorEastern]

Okay. I just think all advanced nba statistics are flawed and the key is to be the least flawed. Small sample sizes are to be avoided at all costs.

After this post I am done here. Move on people.

 

The least flawed by far of course is xRAPM, then WS, then WP ~= PER. For last season. xRAPM typically even beats the Vegas odds makers many years. PER should not be confused with Hollinger's actual NBA season forecasts, which approached xRAPM in accuracy. But the guy is a NBA GM so it comes as no surprise that Hollinger had better unpublished metrics.

 

http://ascreamingcom...t.blogspot.com/

 

xRAPM available here:

 

http://stats-for-the-nba.appspot.com/

 

Going back and recreating years of PER data at the 100-200 minute mark would take me days. I do not care enough to do that.  However this season a person who very well could make an incredible turnaround is Derrick Rose. His current PER is 3.10. Most experts writing about his season start expect him to be a much better soon. Will he reach 24? Probably not. In his MVP season he only just exceeded 23.

 

I find it interesting that Rose is perhaps the player who most closely resembles Howard's situation this season. The major difference is that Rose delayed his return to the NBA until he was 100% healthy (much to the chagrin of Bulls fans since he could have come back late last season). Howard stepped on the court as soon as he was able. Just a passing thought.

 

fyi. I do have a math degree. But not in statistics. However I spent years data mining gigabyte Oracle DB's and doing analysis (statistical and otherwise) on the resulting data sets. I am not talking out of the wrong end of my body here. With statistical analysis one always needs to err on the side of caution. It was painful to have to tell upper management that I don't know. But then as now I can always say "I don't know YET".

[j_wehr]

I dug into the numbers a bit. I looked at all 18 players who finished one of the last three seasons with a PER above 24 (one of whom is Dwight). Through their first six games, all but one had a PER above Dwight's current PER, most of them well above. The exception: 2010-11 Durant (he had an unusually low FG% and an unusually high turnover rate to start the season, and had a PER of 17 through six games). 

 

So 1/18 (=6%) is the best quick-and-dirty estimate you're going to get that Dwight will have Orlando-level stats by the end of the season. 

 

And that's being generous. 

 

On the flipside, you could ask, "what percentage of players who have a PER around 18.3 through 6 games (or 200 minutes) end up finishing the season with a PER at or above 24?" The answer would certainly be much less than 1%.

Edited by j_wehr, 09 November 2013 - 01:30 PM.

[Rahat Huq]

My problem with PER is that I see it as a baseline...a beginning point.  In the olden days, people would point to simple basic per-game stats like "20 and 10" and masturbate to them as if it meant something.  Luckily, we're past that.  PER helped us get past that point in providing context to those numbers in helping us understand what they truly meant.  But beyond that efficiency, I don't know what its really telling us.  It's certainly not doing much in the way of telling us about the overall impact certain players have.  That's why I'm fundamentally against using PER as an evaluation on how a player is playing etc etc.  It can tell you if a player is scoring efficiently and rebounding well, but it's not capturing the little things.  I know RAPM isn't perfect, but it atleast makes an attempt at capturing the player's overall impact on the court.  

 

A guy like Dwight may not be scoring the way he used to or boarding the way he used to, but he's setting some great picks which is opening up the offense...and people also aren't driving the lane when he's in there (and he's not getting a block for it.).  PER isn't capturing any of that.

[j_wehr]

Rahat, do you believe that Dwight is doing MORE of those uncaptured things (setting good picks, intimidating ball handlers) in Houston than he was in Orlando?

 

I agree that RAPM is an overall better metric and that it does a better job of capturing Dwight's total influence. I'm only using PER because of its availability and better reliability over small sample sizes. And since Dwight's PER and RAPM have been so highly correlated over his career -- as evidenced by the charts in the original post -- I don't think there's any problem with using PER for this analysis unless you believe the answer to the question I posed above is yes.

 

Personally, I think it's more likely that he's doing less of those uncaptured things in Houston. I don't think ball handlers are nearly as intimidated of him as they used to be (but I didn't watch him all that closely in Orlando so I don't really know, just my nostalgic perception).

[thejohnnygold]

I could be wrong here, but I think what RAPM actually claims to measure and what YOU think RAPM claims to measure are two different things.

 
2016, posts like this are not productive.  The passive-aggressive innuendo you have written here does nothing for this conversation.  You should be able to recall that you and I have personally gone over this RAPM issue more than once and it was I, not you, that delved into exactly what RAPM is and is not.  Ignoring its methods (which I do not approve of), the database is filled with anomalies that strain the eye-test beyond belief.  So many that it makes it impossible to not question every other player's score.
 
Further, it still fails to qualify the fact that a team game like basketball is nearly impossible to individualize--especially on defense with so many moving parts and variables.  You always respond with, "it isn't supposed to give us answers, but help us ask the right questions".  Which would be fine--if that is what anyone did with it.  Instead, much like Rahat said about PER when it first surfaced, people are getting off to this stat at an alarming rate.  I have gone into detail time and again about how +/- is abused by people.
 
You fell prey to it a couple of days ago when you referred to Greg Smith's performance in game 1 of the OKC series.  Go check the game log.  I did.  Turns out that Greg Smith was the only Rockets' player on the court that did play well during that stretch.  His teammates bombed and he takes the blame--and people bludgeon him with that +/- score for eternity.  It is a fallacy that helps people tell whatever story fits their ideology while they ignore anything that falls outside of that.
 
So, once again, please refrain from your inferences.  I've said it before and I will say it again--I am pretty sure I understand this stat better than you.  If you want to say otherwise please make an actual point.  Starting your sentence with , "I could be wrong here" is nothing but a self-serving disclaimer that allows you to follow it with whatever you like.  Much like all qualifying sentence starters, the truth of the statement generally resides there.  (i.e., I'm not racist but..... or I don't mean to be offensive but.....,).
 
One last thing, I will end with another of your posts in this very thread.  Seems a tad bit contradictory...
 
"I'm pretty lazy and the quality of my posts suffer because of it, but for those who want to raise the quality of their posts here's my advice: Try to write articles, or think of your posts as mini articles. Attacking someone's credentials is something I never see good writers do so I never do it."

[Mason Khamvilay]

JG, you're absolutely right, people do use RAPM the wrong way and that alone is a good enough reason to hate RAPM. Conclusions should not be made based on RAPM alone. And I really did sound  unproductive didn't I? It was very rude of me. I'm sure you can agree that a flawed tool like RAPM can still be useful if used correctly within context, and it is undoubtedly better than the primitive +/- or PER if one had to choose. Lets not discourage the use of information just because it's used incorrectly, but rather educate people on how to use that information correctly. I think you're a pretty a pretty smart guy, good writer and influential figure here at Red94, so if you wanted to educate people in how they're misusing RAPM you're definitely the right guy to do it.

Edited by 2016Champions, 09 November 2013 - 05:50 PM.

[j_wehr]

I'm a little sad to see this thread stray from the Dwight point. And it seems like this is a recurring theme in stats-related posts: the discussion devolving into a debate over the value and appropriate uses of NBA analytics. Could we have a separate thread for that? (FWIW, I find discussion of analytics in general really unproductive. It's very hard to talk about these things in vague terms since so much depends on the particular question/exercise.)

[thejohnnygold]

j_wehr, I think we tried a stats thread once with little success, but we can always try again--you are right--we keep missing the forest for the trees.

 

2016, your point is valid and I apologize for the heated post.  I do want us to be able to use all the tools at our disposal including RAPM.

 

For me, RAPM falls short in a few ways that I find intolerable.  Two fall on the generating-of-data side of it and one on the application side.

 

In the generation of this data, the idea is to adjust the normal +/- stats for a player to account for who is on the court with them.  This seems like a good idea in theory.  The problem is that each play is a microcosm of randomness that is virtually impossible to assess in a simplified manner.  I know the idea is that over a large enough sample these discrepancies will become negligible--and I imagine that for many players this winds up being true.  For myself, the numbers ultimately become meaningless without the requisite context.  Now, if we took a players RAPM for 1 game and were able to combine it with video support to show what went right/wrong then I could see the merit in its generation.

 

I just can't let go of the notion that defense in basketball is largely a team function and to break it down into individual parts misses the point.  I forget who is currently doing it (sorry for that), but one of our members is currently collecting data on our defense using a system similar to what I think is more useful--by watching each play and assigning, as objectively as possible, credit for good play and bad play on defense.  Scoring in the NBA happens--even when excellent defense is present--so to use that as the base metric for determining good vs. bad defense misses the mark for me.  How many times have we seen a play where a guy does everything humanly possible short of fouling on defense only to see his opponent make a ridiculously good (or lucky) shot?  It happens all the time.

 

My second complaint is that once these lists come out there are too many players that just do not pass the eye test (I call them anomalies).   DRAPM clearly favors bigs over wing players.  I will start out by pointing at #1 and #2 overall from last season.  Lebron James and Chris Paul--both renowned for their defense--but Lebron sports a 1.6 DRAPM and Paul a .5 DRAPM.  What does that mean?  It means you have to take what you know and basically over-rule the number.  Would anyone honestly suggest Lebron is not an elite defender?  Chris Paul?  That is a tough sell.

 

Without belaboring the point, the list goes on and on with these questionable numbers--on offense and defense.  At a certain point one realizes, "Hey, wait a minute...I'm only seeing the obvious misses here....how many of these numbers that appear reasonable are equally off-base?"  It is at this point I throw my hands up because there are too many anomalies in the list to be able to trust any of the numbers.  Thus, the whole thing just seems like a colossal waste.

 

Finally, we come to the application of RAPM.  Now, I like that you say it is to be used to ask questions (although I do think most of these questions are things we should already be asking without the need to bring RAPM into it), but more and more it is used as a crutch to help support someone's narrative of what they believe is happening.  Oh, Kenneth Fareid has a .6 ORAPM and Al Horford's is only .5--guess that means Fareid is the better player!  (and seriously, Horford a .5?!?!?!  What does that mean?  I'll take that dude's .5 ORAPM on our team all day long thank you very much)

 

Here's another one us Houston fans can laugh at--Luis Scola.  Last year's RAPM: ORAPM: -.9 DRAPM: +.7.  So, based off of this I am supposed to believe Scola is a better defender than offensive player.  That does not jive with everything I have seen about Scola on the court.

 

Klay Thompson was a -.2 ORAPM last year....   What does this tell me?  As far as I can tell, nothing relevant.

 

Greg Smith last year: ORAPM: -.1 DRAPM -.8  So, either he wasn't one of the most efficient scorers last season with a raging TS% of .636, a PER of 16.1, and on Ortg of 125 while being, allegedly, a revolving door on defense....or something isn't right....which is it?

 

By the way, here is the LINK for the list I am using.

 

I believe RAPM should only be applied to 5-man line-ups.  It's the only thing that makes sense--especially on defense.

 

In a nutshell, that's why I view RAPM as basically a big, heaping pile of numbers that can be fun to play with but lack anything resembling credibility or actual usefulness.  I may be wrong.  I may be missing the point.  For now, I haven't seen or heard anything that leads me to believe that is the case.

 

I'm fine with advanced metrics being used to play with and predict stuff or whatever, but for me it's all style and no substance.  I don't judge anyone for liking it.  Just try to use it with context rather than using it as context.

[Mason Khamvilay]

You make a convincing case, I agree completely that a players role affects his RAPM. I'm not so sure you chose the most convincing anomaly though, I  think LeBron is a good defender but not a DPOY caliber defender so I wouldn't call that an anomaly. 

Edited by 2016Champions, 10 November 2013 - 12:07 AM.

[thejohnnygold]

OK, so he didn't get a DPOY...lots of elite defenders share that problem.  Yet, there are dozens and dozens of players with higher defensive ranks...I'm not going to count them all and this list isn't sortable....does anyone have a link for a better RAPM wesbite?  Still, The scores seem to top out around 6.5 or so....are they 4+ times better than Lebron?  It's just too simple to do all those calculations and make all those adjustments and then come out of all of that and say, "Lebron is a good defender".

 

For the same season (2012-13):

 

Lebron was 15th in blocks/personal foul (although I will concede his star treatment on this...still, we have seen him block plenty of shots and they tend to be clean)

12th in total steals, steals/game, and 2nd in steals/personal foul

His team was 5th in points allowed/game

Tied 5th in Opp. fg%

He sported an opp. PER of 12.7 vs. SF's and an Opp. PER of 17.2 vs. PF's (which may be the crux of the problem as he spent 42% of last season playing PF according to 82games.com.  It should be noted Lebron's PER vs. PF's was 36.7...yowza!)

 

The bottom line is I think Lebron is easily a top 10 defender in this league.  He has made the all-defensive team for the last 5 years.  He has finished top 10 in Defensive Win Shares the last 5 years.  About the only thing that says he is not an elite defender is RAPM.

 

I guess what would be more compelling is to convince me that every player ranked above him deserves to be there....I realize this isn't going to happen, but the point is clear.  It's a toss up between George, Iguodala, and James at SF.

 

And you seem to be turning a blind eye to the other anomalies I listed....and those are just the ones I listed...there are plenty.

[Mason Khamvilay]

I admire your passion, and I also find it funny that you're using blocks and steals to measure defensive impact while preaching about the flaws of RAPM. Dennis Rodman averaged less than 1 block and 1 steal per game, anomalies don't get much worse than that     

Edited by 2016Champions, 10 November 2013 - 03:01 AM.

[j_wehr]

JG, I appreciate the thought you put into that. It was good to hear your perspective. While I think all of the points you make about anomalies are reasonable, I think your big error is using those anomalies to conclude that it's just "a big, heaping pile of numbers that can be fun to play with but lack anything resembling credibility or actual usefulness." That's easy to disprove. I'll lay out my premises and conclusion:

 

Premise: One thing that's "useful"/"credible" is accurately predicting wins.

Premise: Historically, Vegas predicts wins more accurately than any individual using his/her judgment.

Premise: RAPM predicts wins as well or better than Vegas. http://ascreamingcom...er-metrics.html

Conclusion: RAPM is useful/credible.

[thejohnnygold]

I admire your passion, and I also find it funny that you're using blocks and steals to measure defensive impact while preaching about the flaws of RAPM. Dennis Rodman averaged less than 1 block and 1 steal per game, anomalies don't get much worse than that     

 

You are cherry-picking every post I make to attempt some sort of rebuttal.  If that makes you feel better fine.  I have asked you to make an actual point.  Make the case for RAPM.  You can't.  I would suggest you leave it to someone else at this point. 

 

I used a collection of statistics and data to point out that your opinion of Lebron is not shared by many experts nor is it shared by non-manipulated statistics.  Further, his team's statistics support the notion that his defense is excellent--especially considering he often guards the opposing team's best player.

 

I made it a point to use the per foul committed stats to show that he is able to execute these things consistently in a controlled manner--to me this is a delineation between good and elite defenders.

 

Here's something you might find more respectable:

 

"SI.com's Zach Lowe saw things similarly, noting James' "incredible speed and deft positioning" (along with more of the same from Miami point guard Mario Chalmers) in forcing just enough hesitation on catch-and-shoot opportunities to turn open looks into contested jumpers and chances for quick ball movement into defended one-on-one isolation plays. Durant hit some of those one-on-one shots, because Durant's fantastic, but the Thunder's offense as a whole slowed, bogged and suffered, which was a big win for Miami. And something that doesn't happen if literally anybody else but LeBron is tracking Durant." (on Lebron's defense against Durant in the '12 finals)

 

Believe what you like, 2016, and I will do the same.  If you have a point to make that isn't a half-baked attempt to debunk my post I'll gladly hear it.  Also, kudos for bringing up Rodman--you have probably forgotten that week long debate as well (or not and that is why you bring it up).  We could not disagree more on Rodman so to use him as a counterpoint is either ignorant, naive, or trolling.  By the way, your sentence doesn't make sense because you gave no context as to what the Rodman anomaly actually is.

 

 

j_wehr, the win prediction possibility is interesting.  Personally, I'm not overly impressed by a one year throw-down between all the advanced metrics where most had their bright spots and xRAPM won, but it's not like no one else was even close.  Personally, I would need to see 5 consecutive years to even attempt to draw a conclusion from this notion.  I mean, if it's that accurate why aren't we at the sports book right now?    Show me a track record of predictive success and you'll get my attention for sure.

[Mason Khamvilay]

 

JG, I appreciate the thought you put into that. It was good to hear your perspective. While I think all of the points you make about anomalies are reasonable, I think your big error is using those anomalies to conclude that it's just "a big, heaping pile of numbers that can be fun to play with but lack anything resembling credibility or actual usefulness." That's easy to disprove. I'll lay out my premises and conclusion:

 

Premise: One thing that's "useful"/"credible" is accurately predicting wins.

Premise: Historically, Vegas predicts wins more accurately than any individual using his/her judgment.

Premise: RAPM predicts wins as well or better than Vegas. http://ascreamingcom...er-metrics.html

Conclusion: RAPM is useful/credible.

 

Very concise post. Couldn't agree with you more.

[Mason Khamvilay]

What I think RAPM claims to do: improved version of +/-

 

What silly people think RAPM claims to do: rank players

Edited by 2016Champions, 10 November 2013 - 07:03 PM.