Jnai said:
Sure. Check out Chadwick as well.
Also, what method are you going to use that's more rigorous than what Russell Carleton has done? Just curious.
It's more an extension of what Russell Carleton has done than a novel method; most of his numbers are probably close, but there
are two limitations to his prior approach.
One limitation is that the error of the reliability estimates was not computed; it is possible that splitting the data one way (e.g. even/odd vs. assigning pairs or thirds to each group) can arbitrarily lead to a higher or lower r-value than splitting the data another way. In other words, I want to answer questions like: how confident are we that 200 at-bats is sufficient to reliably estimate line-drive rate?
The second limitation is that Carelton has only shown his work for a specific threshold, an r of 0.7. However, it is important to know the shape of the reliability curve as well. When the sample reaches 0.5 reliability can tell us how much we should regress towards the mean when achieving a "reliable" sample.
To extend what Carelton did, I will use resampling techniques to calculate reliability for a set of hitting/pitching metrics, produce error estimates for reliability, and show the effect of increasing numbers of PA on both the mean reliability and error intervals. I plan to run between 10,000 and 100,000 simulations (I will determine how many I need with a separate analysis that is not necessary to describe here). Per simulation, I will permute the assignment of each at-bat to two groups using the mersenne twister algorithm and compute the correlation coefficient across the players for a set of metrics for every 25 PA (e.g. compute it for 25 PA, 50 PA, 75 PA...1200 PA). The permutation approach will allow me to determine the central tendency for reliability, as well as the standard error at each sample size.
From such findings, not only would we be able to determine our confidence for metrics at SSS, we can also measure how much we should regress towards the mean given the sample size provided.
If this has already been done, please let me know. This stuff is more of a hobby for me than my work, so I'm limited in when I have time to work on it, and may be unaware that someone has already done this.
EDITED: for clarity.
