As a statistics student, one of the most important thing I learned is P value but the first thing that surprises me is that p value and its significance are not reliable. From my study in statistics, 5% significance level and p value are the most essential concept and if we can prove it is significant at 5% level, we can make our conclusion directly. However, people are tired of p value and it is not as reliable as scientists believed. One problem of P-value is that it ignores the effect size. Another problem is resulted from the overreliance of p value and P-Hacking. The overreliance leads to P-Hacking which is a human tendency to find evidence that can confirm their own research outcome. Researchers will tend to manipulate the data to get 5% significance level.
On the other hand, advocates of P value can still argue that their worries are not reliable. The first reason is that given the same set of data, different analyzing methods will lead to very different result like the football example in the article “Science Isn’t Broken: It’s just a hell of a lot harder than we give it credit for”. Although all teams have same purpose and are motived to find whether there is discrimination in the football game, they got very different results. This research proves that different methods can lead to different outcomes, though they have same data and without manipulation. The figure “Same Data, Different Conclusion” shows that many results are statistically significant, but they are very different. This implies that “playing around with different methods shouldn’t be thought of as a dirty trick, but encouraged as a way of exploring boundaries”(Aschwanden, 2015). Different research methods shouldn’t always be considered as a manipulation because different methods maybe different perspective people didn’t realize before. The self-correcting of the science is another reason the author provided. He indicates the false statements will encourage people to do more research and correct those statements which will be a virtuous circle.
Therefore, we can’t say the science is broken or science is false. The problems we observe in the statistics research is a warning sign for us and we should be careful and avoid overreliance, effect size problems when processing the data. Science is not feudal superstition, we should use all information available now to make the best decision (Aschwanden, 2015). Science has uncertainty and the uncertainty makes it dangerous and mysterious.
Reference:
Aschwanden, C. (2015). Science Isn’t Broken: It’s just a hell of a lot harder than we give it credit for. Retrieved from https://fivethirtyeight.com/features/science-isnt-broken (Links to an external site.)