We've all heard this ever. Some may even have used this argument: "it works for me." It seems that it works as a defense of the effectiveness of everything. From homeopathy to ouija including NLP or MMS. If you've tried it and you've seen that it works and maybe you even know more people that say that it works, we could say that its effectiveness is proved, right? Let's see why this argument could be wrong.
Our perception of efficacy is affected by certain psychological and sociological biases that prevent us from correctly assessing the results and even alter them unconsciously. This is a well-known phenomenon and this is why personal experience is a questionable way of generating knowledge. Instead, we use studies carried out using the scientific method that has been designed to attempt to correct and neutralize these biases in order to try to get results as objective and solid as possible. Also, if the results resist falsification tests and replication by different teams, we ensure a more consistent result.
For greater clarity when explaining why personal experience is not a valid proof, we are going to review with a little more detail some of these biases with examples of how they affect the assessment of an outcome. It should be clear that all we apply these biases, mostly unconsciously. So we can't assume bad faith on those who fall into these biases and can't either consider ourselves free of them.
Confirmation bias. Probably one of the best known and studied. It may be referred to by many other names or subtypes, some of which are also explained below, like recall bias, cherry-picking, biased interpretation, etc. It is the tendency we have to give more value to cases that confirm our point of view. This applies, for example, if we take a homeopathic preparation every time we have a cold and the third part of the times we do it, we are healed. If we tend to believe that homeopathy works, we will give more validity and representativeness to that cases where we've cured even though it is less the times —and however, it may not be representative even if they were most of the cases, as we will see below. It is the unconscious tendency we have to give more relevance to cases that favor our point of view and disregard those who contradict it.
Anecdotal evidence. It's a type of cherry-picking confirmation bias but sometimes it can be a valid argument, especially for counterexamples. Anecdotal evidence involves taking a small sample of cases (or a single case!) that confirms our hypothesis ignoring the rest. It's for example the case of a user whose smartphone breaks shortly after the warranty expires and based on that the user hypothesizes that the manufacturer designs smartphones with the intention of spoilage upon expiry of warranty. It's not possible to support this claim based on this only case and we will need to see a general tendency for this to happen with a significant percentage of devices of this manufacturer to support this claim with more basis. As advanced above, there may be cases where single anecdotal evidence may be valid, as in the case of the counterexamples. If we argue that there are no green dogs and it is impossible that they ever exist, finding only one green dog would be enough to show that this statement is false.
Forer effect. This other type of confirmation bias is named after the psychologist that conducted the most famous experiment about it. Forer put together a set of heterogeneous people and gave them their personality analysis based on a previous test. Then he asked them if they felt identified with the description of their personality gave by their test. He got overwhelmingly positive responses. At the end of the experiment he unveiled that all the descriptions were the same when obviously not everyone had the same personality, but the descriptions were so generic and vague that anyone could feel that they were talking about themselves. This shows that we can agree with any statement if it is sufficiently vague and imprecise (do you know Nostradamus?), especially if that statement confirms our position, in which case we will be more willing to forgive minor errors and inaccuracies (bias confirmation), especially if we introduce some subtle compliment to flatter the ego of the reader. This is the case of personality pseudosciences like graphology and phrenology.
Spurious relationships. Spurious relationships are those made between two elements that are not directly related. Instead, the relationship is really established through a third factor or there is no relationship at all beyond a simple statistical correlation produced by mere chance. The most common and easy to understand (and fall into them) are cum hoc ergo propter hoc and post hoc ergo propter hoc. They are those that predict a cause and effect relationship between two events when both occur at the same time (cum hoc) or one after the other (post hoc). The simplest example is superstitions: "I made an offering to a saint and I passed the exam." Maybe you passed because you studied. Or maybe even making the offer allowed you to go to the exam with more confidence allowing you to get better results. This bias can also be reinforced by confirmation: if we are predisposed to believe, we conveniently forget the cases in which the offering to the saint did not work. Or we may believe that we fell down to the floor because of that black cat we saw, even if we saw the cat two weeks ago. Another simple case is "I had flu, I took this homeopathic preparation and a few days later I am cured. Homeopathy works." No, it is normal for flu to heal by itself after a few days. You would have cured the same way without homeopathy.
A fairly recent and publicized case was a study showing a correlation between chocolate consumption per capita in each country and the number of Nobel Prizes per capita for that country. The paper warned about the temptation to see spurious relationships, but some media ignored that disclaimer and proclaimed: eating chocolate increases intelligence allowing to win more Nobel prizes. Well, there are other explanations that look more plausible: they may be related by a third factor: chocolate is a food that could be considered a little luxury. That is, it stands to reason that poorer countries consume less chocolate. If these countries are poorer, their citizens have less access to an education of quality and fewer chances to develop cultural, philanthropic or research careers that increase their possibility of getting a Nobel prize for their country.
Placebo effect. This must be the best known of these biases and therefore should not be necessary to explain it in much detail. We can say that the belief that a study will produce a certain result can lead the studied people to favor that certain result. The best-known example is that of drugs: believing that we're getting a drug to ease a particular symptom can make us feel like we are actually experiencing a certain improvement in that symptom. This applies, for example, to acupuncture. If you believe that acupuncture is effective, just realizing that you are receiving acupuncture treatment that you think that will be effective will trigger the production of dopamine acting on the parts of the nervous system that influence the perception of welfare independently of the applied treatment and its real effectiveness by its own. That is, the welfare is not due to the effectiveness of the treatment but because the faith the patient put in its effectiveness triggers dopamine production. If you also surround this experience with a pleasant environment and a friendly and close relationship with the patient, you have more chances of getting a better result.
The mechanisms used by the scientific method to nullify this effect are control groups. A control group receives a treatment that it's known that will produce no results. So the positive results obtained in the control group can be compared with those positive results in the study group that receives the treatment whose effectiveness is being studied. If the difference in positive results with the study group is not significant, it can be said that the positive results in the study group are due to the placebo effect. For example, in the case of acupuncture, it has been proven that the number of patients who have noticed an improvement after an acupuncture treatment is equivalent to the number of satisfied patients after being randomly pricked.
Observer-expectancy effect. In the above case, we explained how the subjects used in the study may unconsciously favor a particular outcome depending on what result they expect to get. And to fix this, the patient does not know if the treatment is real or placebo. This is called a "single-blind" or "masked" study. A rejoinder often wielded by supporters of homeopathy is that it has been "proved" that homeopathy gives positive results treating animals. How is this possible? Is not supposed that animals are immune to placebo effect? Well, not really. For example, animals are subject to conditioning like us (or maybe more), which can influence the placebo effect. But let's forget this point and concede that animals are not subject to the placebo effect, although we know that this is not true. It is also common that many of these studies purporting to demonstrate efficacy in animals of therapies without scientific basis have no control group or it is very weak. I have even find statements about the effectiveness of homeopathy in horses using a single subject! But a study without a control group can't be taken seriously, as we have already seen in the previous points. Let's talk now about animal studies using control groups. If the study is masked, that's enough, right? Well, no. For example, in an experiment in which you want to check the effectiveness of a particular medical treatment, it is inevitable for the people who apply this treatment to be predisposed to achieve a particular result. This predisposition may lead them to give different treatment to patients depending on the outcome they expect from them. For example, creating a more hostile environment for those patients they expect a negative result from and one more friendly for those patients they expect a positive result from. To counteract this bias, another level of "blindness" is introduced: the "double-blind", in which the person administering the treatment doesn't know if they are administering placebo or the treatment being studied. Once you have gathered the reactions of the subjects of the study, it will be necessary to interpret data and draw a conclusion. Here again, the people in charge of this task may be influenced by their bias toward a particular result, putting more interest in finding positive results in a particular group. For that exists a new level of blindness: the "triple-blind", in which people who analyze the results don't know which group results they are studying. For example, if the study tries to compare the effectiveness of two drugs, it should be possible to find that the drug that was being provided to a certain group was better without knowing which one of the drugs it was ... until it is too late to influence the result.
Echo chamber. I find echo chambers fascinating because, as you might guess from its name, they expand other biases as if they were an echo, this is why I left them for the last. This is what happens when someone says: "How could that party win the elections? I don't know anybody who said they would vote them." What happens is that the group used to gather those opinions is not a representative sample of society. The social relationships we build not surprisingly tend to be with people similar to us: cultural level, political ideas, purchasing power, beliefs... And of course, by surrounding ourselves with people who think like us, our opinions are reinforced and we deprive ourselves of hearing different points of view that could make us doubt by offering evidence that may show us that we might be wrong. If by taking our social circle as a valid sample of society we are being biased, it will be worse if those relationships were established in a medium that is already biased by itself: "in Twitter everybody is against this measure that has been approved by the Government." As already mentioned, the tweets that we see in our Twitter timeline are written by people that we have chosen to read and that, of course, are more tailored to our points of view. If we are against this measure adopted by the Government, it stands to reason that we observe a major opposition in Twitter. In addition, by establishing our social relationships through Twitter, we are using a medium that is already biased because it represents a particular sector of the population that doesn't represent society.
This way we will relate to others convinced of the same issues we want to demonstrate. They will reinforce our perception of a real demonstration of our issues. Thus, those who had a positive experience with astrology (either by a placebo effect, post hoc, Forer effect or whatever) will be more likely to relate to people who also had favorable experiences, reinforcing the perception that astrology works: "not only works for me, I know more cases of people who said that worked for them." The echo chambers can also get more and more closed on themselves and extreme their positions: the more people you follow satisfying a particular point of view, Twitter will continue to recommend you to follow other users more akin to that trend. Similarly, when searching on Google about, say, Homeopathy, if you often click on links of homeopathy supporters, Google will remember that preference and the links that can display information contrary to homeopathy would be increasingly relegated to lower positions so it will be increasingly difficult to obtain information to contrast our beliefs.
For all that it's necessary to contrast our views with people of the opposite opinion, if our views can resist the attack from those who intend to refute them, we can hold them on a greater basis.