P-Value Problems: Crash Course Statistics #22
video description
Date: 2022-04-04
Related videos
Comments and reviews: 10
Jorge
I'm a PhD student in economics and my honest opinion is that p-values is that forcing a policy asking for ever lower p-value cutoff just encourages researchers to get ever larger data samples, which is not bad. However, we run into the fact that with large enough sample, the distributions thin out and we will get ever more likely to reject null hypotheses. The problem is that statistical significance can be meaningless when we fail to have economic significance: i. e. the effect is large enough for it to matter. So, p-values are important but by no means should one ever take too seriously the result of any given statistical test in isolation.
reply
I'm a PhD student in economics and my honest opinion is that p-values is that forcing a policy asking for ever lower p-value cutoff just encourages researchers to get ever larger data samples, which is not bad. However, we run into the fact that with large enough sample, the distributions thin out and we will get ever more likely to reject null hypotheses. The problem is that statistical significance can be meaningless when we fail to have economic significance: i. e. the effect is large enough for it to matter. So, p-values are important but by no means should one ever take too seriously the result of any given statistical test in isolation.
reply
Atrijit
about problem #2 (that p-values assume null hypothesis to be true):
Can we not say that this isn't really a problem since the way we use p-value is not just the conditional probability? We use the -reductio ad absurdum- argument in relation to p-value.
So to say that the fact that p-values assume null hypothesis to be true means it cant help us judge whether the null is true or not, we would also have to say that -reductio ad absurdum- is a faulty argument in general. Which, i suppose, could be argued but would be a hard task.
reply
about problem #2 (that p-values assume null hypothesis to be true):
Can we not say that this isn't really a problem since the way we use p-value is not just the conditional probability? We use the -reductio ad absurdum- argument in relation to p-value.
So to say that the fact that p-values assume null hypothesis to be true means it cant help us judge whether the null is true or not, we would also have to say that -reductio ad absurdum- is a faulty argument in general. Which, i suppose, could be argued but would be a hard task.
reply
Alexander
A great video. I'm a risk manager, so I work a lot with probability and hypothesis testing, and a lot of people in my field misinterpret p-values as probability of rejecting a true null hypothesis given data, which it absolutely isn't. It's a damn confusing thing to understand, but every student of statistics should learn that all that a p-value tells you is how extreme your sample is given that the null is true. It's important because this type of misunderstanding then creeps into academic papers.
reply
A great video. I'm a risk manager, so I work a lot with probability and hypothesis testing, and a lot of people in my field misinterpret p-values as probability of rejecting a true null hypothesis given data, which it absolutely isn't. It's a damn confusing thing to understand, but every student of statistics should learn that all that a p-value tells you is how extreme your sample is given that the null is true. It's important because this type of misunderstanding then creeps into academic papers.
reply
emilwestin
It's important to remeber that the statistical tests can be overly insensitive at very large sample sizes since the power increase. The alpha then needs to be adjusted to a lower level, possibly 1%. It's also important to study the practical significance in these cases. For example, when testing if data is normally distributed and it's rejected, to test the practical significance a graphical approach can be done to study if its approximately normally distributed.
reply
It's important to remeber that the statistical tests can be overly insensitive at very large sample sizes since the power increase. The alpha then needs to be adjusted to a lower level, possibly 1%. It's also important to study the practical significance in these cases. For example, when testing if data is normally distributed and it's rejected, to test the practical significance a graphical approach can be done to study if its approximately normally distributed.
reply
Peter
great job explaining p-values! These videos are very good. However, I would recommend to the content creator that rapidly firing a new research question (orange juice vs coffee, old vs young chess players, cat spots, etc) just makes the video feel harder to follow. I would recommend discussing 1 or 2 max research problems in the video so we aren't spending half our time wrapping our heads around the strange research questions being discussed
reply
great job explaining p-values! These videos are very good. However, I would recommend to the content creator that rapidly firing a new research question (orange juice vs coffee, old vs young chess players, cat spots, etc) just makes the video feel harder to follow. I would recommend discussing 1 or 2 max research problems in the video so we aren't spending half our time wrapping our heads around the strange research questions being discussed
reply
verdatum
IMHO, this episode is basically the entire reason why this series needed to exist. P-Hacking continues to be one of the most detrimentally misunderstood concepts of my lifetime. It started getting talked about a few years back, but it hasn't stopped lazy science journalists from picking up the odd worthless deceptive science press-release and failing to scrutinize the validity.
reply
IMHO, this episode is basically the entire reason why this series needed to exist. P-Hacking continues to be one of the most detrimentally misunderstood concepts of my lifetime. It started getting talked about a few years back, but it hasn't stopped lazy science journalists from picking up the odd worthless deceptive science press-release and failing to scrutinize the validity.
reply
Michael
My null hypothesis: The data is normally distributed with a mean u, and a standard deviation of sigma.
As I collect data, the data has mean x_bar, and standard deviation of s. I know that the normal distribution is a conjugate prior to itself, so I update my distribution to more consistent with the given data.
reply
My null hypothesis: The data is normally distributed with a mean u, and a standard deviation of sigma.
As I collect data, the data has mean x_bar, and standard deviation of s. I know that the normal distribution is a conjugate prior to itself, so I update my distribution to more consistent with the given data.
reply
Dr. Honey
Most of the error here referred in defining/limiting p-value are surely because of sampling error.
I'm pretty sure if better sampling techniques are adopted they will have lot more to explain for the sake of what they had explained it now.
reply
Most of the error here referred in defining/limiting p-value are surely because of sampling error.
I'm pretty sure if better sampling techniques are adopted they will have lot more to explain for the sake of what they had explained it now.
reply
Greg
Thank you for the series. I find there are too many examples per video. Some, like the black swan example, are necessary to prove the point, but I wonder if other ones (cats' weights, bees, et al) could be rolled into one?
reply
Thank you for the series. I find there are too many examples per video. Some, like the black swan example, are necessary to prove the point, but I wonder if other ones (cats' weights, bees, et al) could be rolled into one?
reply
Matt
I'm currently taking an Experimental Methods course in Psycholinguistics for my Master's degree, and we're currently doing a TON of significance testing in R, so these videos are amazingly timely for me!
reply
I'm currently taking an Experimental Methods course in Psycholinguistics for my Master's degree, and we're currently doing a TON of significance testing in R, so these videos are amazingly timely for me!
reply
Add a review, comment
Other channel videos
