The Replication Crisis: Crash Course Statistics #31
video description
Date: 2022-04-04
Related videos
Comments and reviews: 10
Frank
The prime problem is publish or perish. If you just pay a scientist (=prolong the grant) if he publishes, thus if he has to publish to not starve, so he will publish no matter what. that's exactly as intended. As unfortunately scientific discoveries can't be forced and are often just luck, but the scientist has to publish something. anything to not end up as a hobo, so he will find something. anything to publish. If not hell make something up if necessary. Bending the truth is bad. being without a place to live is worse, so guess what people do? What do you expect? Ruining your life. just to be honest? Seriously. just drop the measurement that doesn't fit the intended explanation and everything will be fine. That's not even lying or faking data, it's giving people what they want. The same, but even worse is true for PhD students. Without a PhD you are academically dead meat and can think about driving a taxi for Uber or working as callboy in the future to not starve. To avoid these unpleasant alternatives you'll do anything to find something, somehow that's publishable. If it is reproducible or relevant is completely irrelevant, because you need the money to not starve and pay the rent.
In the end it works exactly as it was intended: financial incentives (=not starving) lead to an increase in the number of papers (= formally scientific progress) and everyone is happy. The scientists don't starve the people founding it get their papers, so everything is dandy, well until they find out 20 years later, that 90% of all the stuff published ain't worth the paper it used to be printed on because it is scientific junk (and that's not only true for pseudo -sciences- like social -sciences-, but also for hard natural sciences. In economics such unintended outcome due to wrongly set incentives is called cobra effect. The real problem here is that scientific progress can't be forced. If one tries to force people to do something that they can't really influence, they'll find another way to somehow comply with the requirements. and the result is scientific junk. As long as the incentives are set towards mass production of papers and not towards producing real scientific research, this won't change. As a result does every scientist know that a newly published paper ain't worth anything until it has been reproduced several times by independent (=not affiliated) labs.
The major problem here is not science. the problem is people who don't understand science, ruining it by thinking that something unknown and never done before can be solved by just throwing money at the problem and putting pressure on people. That's NOT how science works. That's how paper production works.
reply
The prime problem is publish or perish. If you just pay a scientist (=prolong the grant) if he publishes, thus if he has to publish to not starve, so he will publish no matter what. that's exactly as intended. As unfortunately scientific discoveries can't be forced and are often just luck, but the scientist has to publish something. anything to not end up as a hobo, so he will find something. anything to publish. If not hell make something up if necessary. Bending the truth is bad. being without a place to live is worse, so guess what people do? What do you expect? Ruining your life. just to be honest? Seriously. just drop the measurement that doesn't fit the intended explanation and everything will be fine. That's not even lying or faking data, it's giving people what they want. The same, but even worse is true for PhD students. Without a PhD you are academically dead meat and can think about driving a taxi for Uber or working as callboy in the future to not starve. To avoid these unpleasant alternatives you'll do anything to find something, somehow that's publishable. If it is reproducible or relevant is completely irrelevant, because you need the money to not starve and pay the rent.
In the end it works exactly as it was intended: financial incentives (=not starving) lead to an increase in the number of papers (= formally scientific progress) and everyone is happy. The scientists don't starve the people founding it get their papers, so everything is dandy, well until they find out 20 years later, that 90% of all the stuff published ain't worth the paper it used to be printed on because it is scientific junk (and that's not only true for pseudo -sciences- like social -sciences-, but also for hard natural sciences. In economics such unintended outcome due to wrongly set incentives is called cobra effect. The real problem here is that scientific progress can't be forced. If one tries to force people to do something that they can't really influence, they'll find another way to somehow comply with the requirements. and the result is scientific junk. As long as the incentives are set towards mass production of papers and not towards producing real scientific research, this won't change. As a result does every scientist know that a newly published paper ain't worth anything until it has been reproduced several times by independent (=not affiliated) labs.
The major problem here is not science. the problem is people who don't understand science, ruining it by thinking that something unknown and never done before can be solved by just throwing money at the problem and putting pressure on people. That's NOT how science works. That's how paper production works.
reply
Lystic
-No single study is going to show us the way the world REALLY is. But that study and the studies that follow it that do and don't find the same relationships will get us closer and closer. - Well said. Though if there is a political or intellectual dogma/homogeneity in universities and work environments it could be that doing more studies will only lead to further misdirection. I do think looking critically at the level of diversity of thought and political interest is going to be part of a solution to improve the accuracy of studies.
I also see a point raised repeatedly in comments here and elsewhere, that it doesn't matter what you publish as a scientist as long as you can keep publishing. I have no experience with this, but if true, that doesn't sound like a system that inspires honesty in science. Maybe there is something to be gained by taking a closer look at how scientists keep their jobs.
However it may be, this was a well put together, thought inspiring video.
reply
-No single study is going to show us the way the world REALLY is. But that study and the studies that follow it that do and don't find the same relationships will get us closer and closer. - Well said. Though if there is a political or intellectual dogma/homogeneity in universities and work environments it could be that doing more studies will only lead to further misdirection. I do think looking critically at the level of diversity of thought and political interest is going to be part of a solution to improve the accuracy of studies.
I also see a point raised repeatedly in comments here and elsewhere, that it doesn't matter what you publish as a scientist as long as you can keep publishing. I have no experience with this, but if true, that doesn't sound like a system that inspires honesty in science. Maybe there is something to be gained by taking a closer look at how scientists keep their jobs.
However it may be, this was a well put together, thought inspiring video.
reply
Michael
I think it-s not intentional and, although simple human error does occur, it is not simply a mistake. The dogma we live by is - publish or perish-. To be tenured or promoted or get funding you have to have a steady output of papers. So, in many labs there is the concept of the LPU, or least publishable unit. The smallest amount of data for a publication is probably just a couple of p values less than 0. 05. Also, in animal studies, there is a commitment to using the smallest number of animals that provide the statistical power necessary to generate a statistically significant result. To demand that all animal studies be replicated will require a societal consensus that we are going to probably triple the number of animals used in experiments. In other fields funding increases will be required.
reply
I think it-s not intentional and, although simple human error does occur, it is not simply a mistake. The dogma we live by is - publish or perish-. To be tenured or promoted or get funding you have to have a steady output of papers. So, in many labs there is the concept of the LPU, or least publishable unit. The smallest amount of data for a publication is probably just a couple of p values less than 0. 05. Also, in animal studies, there is a commitment to using the smallest number of animals that provide the statistical power necessary to generate a statistically significant result. To demand that all animal studies be replicated will require a societal consensus that we are going to probably triple the number of animals used in experiments. In other fields funding increases will be required.
reply
dave
Peer review and subsequent publication also need to be buttressed. Many poorly constructed studies with indifferent support are nonetheless positively reviewed and published simply because they seem topical, shocking and, in short, make great leading articles. Having read many poor papers I've come to believe that evidentiary standards should be most stringent for new and unexpected results. One bad paper can be the pillar for 100 even worse ones. Finally, fewer bad papers would be written in the first place if we remember Carl Popper. Theories can only be DIS-PROVEN! Theories should be excepted only after two conditions are met: the theory can be dis-proven in principle and has survived all attempted and conceived falsifications.
We need fewer and better papers.
reply
Peer review and subsequent publication also need to be buttressed. Many poorly constructed studies with indifferent support are nonetheless positively reviewed and published simply because they seem topical, shocking and, in short, make great leading articles. Having read many poor papers I've come to believe that evidentiary standards should be most stringent for new and unexpected results. One bad paper can be the pillar for 100 even worse ones. Finally, fewer bad papers would be written in the first place if we remember Carl Popper. Theories can only be DIS-PROVEN! Theories should be excepted only after two conditions are met: the theory can be dis-proven in principle and has survived all attempted and conceived falsifications.
We need fewer and better papers.
reply
tbeck27
It's amazing the scope of this. And the fact that public policy/decisions are often based on scientific studies that can't even be replicated. A lot of confirmation on part of the researchers. Take the IAT for example, let millions of people thinking they are racist, and then comes out that the test is irrelevant, especially considering the same person can get a very opposite score the next time they take it. Yet the researchers who made it defended it for years even knowing its shortcomings. WHY? Because they made it! They weren't interested in good science, but to get their name out and to influence people to their own ideology about something.
reply
It's amazing the scope of this. And the fact that public policy/decisions are often based on scientific studies that can't even be replicated. A lot of confirmation on part of the researchers. Take the IAT for example, let millions of people thinking they are racist, and then comes out that the test is irrelevant, especially considering the same person can get a very opposite score the next time they take it. Yet the researchers who made it defended it for years even knowing its shortcomings. WHY? Because they made it! They weren't interested in good science, but to get their name out and to influence people to their own ideology about something.
reply
crash_course
If we are speaking about the -Study finds: People who drink beer are smarter than people who drink wine- type of research, than I'm not baffled. But still, this is a huge problem. Maybe there needs to be a different vetting process or some sort of re-shaped peer review in which reproducibility is required. This would vet out all of those studies which everybody seems to buzz about. Quality could win against quantity.
reply
If we are speaking about the -Study finds: People who drink beer are smarter than people who drink wine- type of research, than I'm not baffled. But still, this is a huge problem. Maybe there needs to be a different vetting process or some sort of re-shaped peer review in which reproducibility is required. This would vet out all of those studies which everybody seems to buzz about. Quality could win against quantity.
reply
Teun
A point missed in this video is that of publication bias. Rejections of null hypotheses are more likely to get published than failure to reject null hypotheses. Finding no statistically significant effects is not very interesting reading material, but might help prevent other researchers wasting time and money on the same fruitless hypothesis.
reply
A point missed in this video is that of publication bias. Rejections of null hypotheses are more likely to get published than failure to reject null hypotheses. Finding no statistically significant effects is not very interesting reading material, but might help prevent other researchers wasting time and money on the same fruitless hypothesis.
reply
Irwain
I see it little bit differently. The problem is how people think about published results. It should be something -Hmmm, there might be something here. -. And only those studies, which have been replicated and re-tested and analyzed to death, would get published. Just like SW developement. When you first create something, it's buggy.
reply
I see it little bit differently. The problem is how people think about published results. It should be something -Hmmm, there might be something here. -. And only those studies, which have been replicated and re-tested and analyzed to death, would get published. Just like SW developement. When you first create something, it's buggy.
reply
TheControlBlue
You took the time to say that political -activists- used the replication problem to discredit SCIENCEE. but failed to say that this problem only concerns the Social Sciences. Nice!
I mean, I have seen nobody refute gravity and electricity, but there being more than two genders on the other side.
reply
You took the time to say that political -activists- used the replication problem to discredit SCIENCEE. but failed to say that this problem only concerns the Social Sciences. Nice!
I mean, I have seen nobody refute gravity and electricity, but there being more than two genders on the other side.
reply
Jesse
I wish there was a journal dedicated to re-running experiments. It would only accept papers that were expressly attempting to reproduce papers that had already been published somewhere. Ideally there would even be funding grants awarded to researchers reproducing results.
reply
I wish there was a journal dedicated to re-running experiments. It would only accept papers that were expressly attempting to reproduce papers that had already been published somewhere. Ideally there would even be funding grants awarded to researchers reproducing results.
reply
Add a review, comment
Other channel videos
