Is there a reproducibility crisis in science? - Matt Anticole
video description
Date: 2020-08-22
Related videos
Comments and reviews: 10
education
In 2011, a team of physicist reported a startling discovery. Neutrinos travelled faster than the speed of light by 60 billions of a second. In their 730 kilometer trip from Geneva to a detector in Italy. Despite 6 months of double checking, the bizarre discovery refused to yield. But rather than celebrating a physics revolution, the researchers published a cautious paper arguing for continued research and an effort to explain the observed anomaly. In time, the error was tracked to a single incorrectly connected fibre optic cable. This example reminds us that real science is more than static textbooks. Instead, researchers around the world are continuously publishing their latest discoveries, with each paper adding to the scientific conversation. Published studies can motivate future research, inspire new products, and inform government policy. So, it is important that we have confidence in the published results. If their conclusions are wrong, we risk time, resources, and even our health in pursuit of false leads.
When findings are significant, they are frequently double checked by other researchers, either by re-analysing the data, or by re-doing the entire experiment. For example, it took repeated investigation of the CERN data, before the timing error was tracked down. Unfortunately, there are currently neither the resources nor professional incentives to double check the more than 1 million scientific papers published annually. Even when papers are challenged, the results are not reassuring. Recent studies that examined dozens of published pharmaceutical papers managed to replicate the results less than 25% of them. And similar results have been found in other scientific disciplines. There are variety of sources for irreproducible results. Errors could hide in the original design, execution, or analysis of the data. Unknown factors, such as patients undisclosed condition in a medical study can produce results that are not repeatable in new test subjects. And sometimes, the second research group cant reproduce the original results, simply because they dont know exactly what the original group did. However, some problems might stem from systematic decisions in how we do science. Researchers, the institutions that employ them, and the scientific journals that publish findings are expected to produce big results frequently. Important papers can advance careers, generate media interest, and secure essential funding, so theres slim motivation for researchers to challenge their own exciting results. In addition, little eincentrive exists to publish results unsupportive of the expected hypothesis. That results in a deluge of agreement between what was expected and what was found. In rare occasions, this can even lead to deliberate fabrication, such as in 2013, when a researcher spiked rabbit blood with human blood to give false evidence that his HIV vaccine was working. The publish or perish mindset can also compromise academic journals traditional peer review processes, which are safety checks where experts examine submitted paper for potential shortcomings. The current system, which might involve only one or two reviewers, can be woefully ineffective. That was demonstrated in a 1998 study, where eight weaknesses were deliberately inserted into papers, but only around 25% were caught upon review. Many scientists are working toward improving reproducibility in their fields. Theres a push to make researchers raw data, experimental procedures, and analytical techniques more openly available in order to ease replication efforts. The peer review process can also be strengthened to more efficiently weed out weak papers prior to publication. And we could temper the pressure to find big results by publishing o more papers that fail to confirm the original hypothesis. An event that happens far more than current scientific literature suggests. Science always has and always will encounter some false starts as part of the collective acquisition of new knowledge. Finding ways to imporove the reproducibility of our results can help us weed out those false starts more effectively, keeping us moving steadiliy towards exciting new discoveries.
reply
In 2011, a team of physicist reported a startling discovery. Neutrinos travelled faster than the speed of light by 60 billions of a second. In their 730 kilometer trip from Geneva to a detector in Italy. Despite 6 months of double checking, the bizarre discovery refused to yield. But rather than celebrating a physics revolution, the researchers published a cautious paper arguing for continued research and an effort to explain the observed anomaly. In time, the error was tracked to a single incorrectly connected fibre optic cable. This example reminds us that real science is more than static textbooks. Instead, researchers around the world are continuously publishing their latest discoveries, with each paper adding to the scientific conversation. Published studies can motivate future research, inspire new products, and inform government policy. So, it is important that we have confidence in the published results. If their conclusions are wrong, we risk time, resources, and even our health in pursuit of false leads.
When findings are significant, they are frequently double checked by other researchers, either by re-analysing the data, or by re-doing the entire experiment. For example, it took repeated investigation of the CERN data, before the timing error was tracked down. Unfortunately, there are currently neither the resources nor professional incentives to double check the more than 1 million scientific papers published annually. Even when papers are challenged, the results are not reassuring. Recent studies that examined dozens of published pharmaceutical papers managed to replicate the results less than 25% of them. And similar results have been found in other scientific disciplines. There are variety of sources for irreproducible results. Errors could hide in the original design, execution, or analysis of the data. Unknown factors, such as patients undisclosed condition in a medical study can produce results that are not repeatable in new test subjects. And sometimes, the second research group cant reproduce the original results, simply because they dont know exactly what the original group did. However, some problems might stem from systematic decisions in how we do science. Researchers, the institutions that employ them, and the scientific journals that publish findings are expected to produce big results frequently. Important papers can advance careers, generate media interest, and secure essential funding, so theres slim motivation for researchers to challenge their own exciting results. In addition, little eincentrive exists to publish results unsupportive of the expected hypothesis. That results in a deluge of agreement between what was expected and what was found. In rare occasions, this can even lead to deliberate fabrication, such as in 2013, when a researcher spiked rabbit blood with human blood to give false evidence that his HIV vaccine was working. The publish or perish mindset can also compromise academic journals traditional peer review processes, which are safety checks where experts examine submitted paper for potential shortcomings. The current system, which might involve only one or two reviewers, can be woefully ineffective. That was demonstrated in a 1998 study, where eight weaknesses were deliberately inserted into papers, but only around 25% were caught upon review. Many scientists are working toward improving reproducibility in their fields. Theres a push to make researchers raw data, experimental procedures, and analytical techniques more openly available in order to ease replication efforts. The peer review process can also be strengthened to more efficiently weed out weak papers prior to publication. And we could temper the pressure to find big results by publishing o more papers that fail to confirm the original hypothesis. An event that happens far more than current scientific literature suggests. Science always has and always will encounter some false starts as part of the collective acquisition of new knowledge. Finding ways to imporove the reproducibility of our results can help us weed out those false starts more effectively, keeping us moving steadiliy towards exciting new discoveries.
reply
Straw
This is a great video, usually the general public THINKS that scientist are great people. They are not. I been working in this industry for half a decade. The scientists I'm working with are assholes. They use the lab funds for their own holiday trip to a conference in a fancy country. They cover up mistakes and make it seems like a grain of salt. They use subordinates to get what they want. They pretend to know-it-all person, but in the scientific industry. that is not always true. So to the general public, your taxes are used as funds for the scientific industry. Integrity, honesty, reproducible, validity and reliability is a non-existence principals in our society. So, if next time you saw a celebrity contributing to a cancer research. just think they are idiots.
reply
This is a great video, usually the general public THINKS that scientist are great people. They are not. I been working in this industry for half a decade. The scientists I'm working with are assholes. They use the lab funds for their own holiday trip to a conference in a fancy country. They cover up mistakes and make it seems like a grain of salt. They use subordinates to get what they want. They pretend to know-it-all person, but in the scientific industry. that is not always true. So to the general public, your taxes are used as funds for the scientific industry. Integrity, honesty, reproducible, validity and reliability is a non-existence principals in our society. So, if next time you saw a celebrity contributing to a cancer research. just think they are idiots.
reply
Charles
No replications published, huh? The greatest physical law is, I propose, the second law of (classical) thermodynamics, expressed as it is not possible to construct a system that exchanges heat with a single reservoir and delivers net work. The classical second law depends exactly upon reproductions of experiments in which the failure to produce net work is confirmed.
Now, in number theory, Colatz' conjecture is considered to have been proven because billions of sequential calculations from the initial number (index) have all led to 1, which is what Colatz conjectured. Clausius ' mode of thought lives!
Charles A Berg
reply
No replications published, huh? The greatest physical law is, I propose, the second law of (classical) thermodynamics, expressed as it is not possible to construct a system that exchanges heat with a single reservoir and delivers net work. The classical second law depends exactly upon reproductions of experiments in which the failure to produce net work is confirmed.
Now, in number theory, Colatz' conjecture is considered to have been proven because billions of sequential calculations from the initial number (index) have all led to 1, which is what Colatz conjectured. Clausius ' mode of thought lives!
Charles A Berg
reply
&Abel
I think that we shall have to get accustomed to the idea that we must not look upon science as a body of knowledge, but rather as a system of hypotheses, or as a system of guesses or anticipations that in principle cannot be justified, but with which we work as long as they stand up to tests, and of which we are never justified in saying that we know they are true or more or less certain or even probable. -Karl Popper
reply
I think that we shall have to get accustomed to the idea that we must not look upon science as a body of knowledge, but rather as a system of hypotheses, or as a system of guesses or anticipations that in principle cannot be justified, but with which we work as long as they stand up to tests, and of which we are never justified in saying that we know they are true or more or less certain or even probable. -Karl Popper
reply
Oskar
Can't emphasize enough how well done andto the point this video is.
And it's true - it really comes down to the fact that there's no motivation in the scientific community to cross-verify anything. As said, it's actually a detriment to one's career to do that, as it's a delay to career progress. Conferencies and journals are simply not interested to publish validation studies.
reply
Can't emphasize enough how well done andto the point this video is.
And it's true - it really comes down to the fact that there's no motivation in the scientific community to cross-verify anything. As said, it's actually a detriment to one's career to do that, as it's a delay to career progress. Conferencies and journals are simply not interested to publish validation studies.
reply
Edward
Psychology gets a bad wrap because it's more of a heuristic. The brain is very complex and hard to understand. When a psychologist seeks to understand human behavior it's like trying to figure out the inner workings of a complex math equation knowing only a few inputs and an output. They do good work though so take it easy on them, they have a very difficult job.
reply
Psychology gets a bad wrap because it's more of a heuristic. The brain is very complex and hard to understand. When a psychologist seeks to understand human behavior it's like trying to figure out the inner workings of a complex math equation knowing only a few inputs and an output. They do good work though so take it easy on them, they have a very difficult job.
reply
Lali
So isn't the reproducibility issue, in part, cancelled out by the fact that if new studies are build upon incorrect older studies, eventually the results don't add up and you can track it back to what may have been false? Sure it's a waste of time, money and other resources but it does prevent false information from being acted upon in the long run )
reply
So isn't the reproducibility issue, in part, cancelled out by the fact that if new studies are build upon incorrect older studies, eventually the results don't add up and you can track it back to what may have been false? Sure it's a waste of time, money and other resources but it does prevent false information from being acted upon in the long run )
reply
ZachLuscher
Ha! Science research is all about the money and most researchers are forced to p hack there way to groundbreaking results to continue their funding. It would be better to provide funding to quality research rather then just the break throughs of which 10-25% are accurate.
reply
Ha! Science research is all about the money and most researchers are forced to p hack there way to groundbreaking results to continue their funding. It would be better to provide funding to quality research rather then just the break throughs of which 10-25% are accurate.
reply
Marie
Wouldn't that make it easier for people to steal credibility, but even worse some of our technological advances. I can see how it can be helpful too, but we already have institutions and government agencies that can step in and take years of research from people.
reply
Wouldn't that make it easier for people to steal credibility, but even worse some of our technological advances. I can see how it can be helpful too, but we already have institutions and government agencies that can step in and take years of research from people.
reply
tomas
I dont see nobody talking about episthemology and that is a shame: ( the main problem (or at least from my point of view) is in the reliability of the scientific mehtod. Maybe we should criticize and rebuild what we think it is the scientific method.
reply
I dont see nobody talking about episthemology and that is a shame: ( the main problem (or at least from my point of view) is in the reliability of the scientific mehtod. Maybe we should criticize and rebuild what we think it is the scientific method.
reply
Add a review, comment
Other channel videos
