We find what we’re looking for
By: Divya Darling
June 9, 2020
I discovered a study published in the British Journal of Psychology that indicates the role researcher bias plays in obtaining results, even in extremely well designed experiments. In it we see both a believer and non-believer are able to find results that confirm their pre-existing beliefs.
The study was a collaboration by Marilyn Schlitz, a US psychologist who believes in the existence of psi—direct interactions between mental processes and the physical world or other mental processes occurring outside currently understood channels—and Richard Wiseman, a UK psychologist who does not believe in it.
Originally both researchers conducted separate studies within their own laboratories as a means to replicate findings of previous experiments where participants could detect when they were being stared at from someone behind them or behind a one-way mirror. In each experiment the researcher was able to find significant evidence to conclude his/her position. Thus, the researchers collaborated to see what would happen within the context of a study where participants were randomly assigned to either one of them—a psi proponent and skeptic.
Each of the 32 participants was assessed individually. An independent researcher coded the order of stare/not-stare intervals. When participants arrived at the lab, they were welcomed by either Wiseman or Schlitz (randomly and opportunistically assigned) who explained the instructions, sat them down in front of a video camera, and then connected their fingers to a machine that measured their electrodermal activity (sweat response). Participants then completed a questionnaire about their belief in psi.
During this time Wiseman/Schlitz was standing in a different room, approximately 20 meters away from the participant, watching the participant on a 14 inch monitor, alternating between staring and not staring at the participant based on the prepared sheet (which they had not been exposed to prior to greeting the participant).
The analysis showed slightly higher belief is psi scores for participants run by Schlitz as compared to Wiseman’s participants, but it was not enough to reach significance. As a whole participants tested by Wiseman showed no significant difference in the stare/non-stare conditions, thus affirming his hypothesised position. Participants tested by Schlitz, however, showed an effect greater than can be explained by chance, which aligns with her beliefs. When comparing participants’ detection on an individual basis, by subtracting stare response from non-stare, Schlitz’s participants were not significantly different to Wiseman’s which may indicate that Schlitz had a greater effect on participants skin response in general as compared to Wiseman, perhaps because of her belief that it is possible.
I find this study fascinating because in the context of the same carefully designed experiment, each researcher was able elicit the sweat response that they had presupposed. The implications of this experiment are interesting and very important. While a study with 32 participants is hardly enough to discredit a plethora of other research where scientists make every effort to control conditions, it does make me raise an eyebrow about the scientific studies I have read and cited in the past.
Have you ever considered the role that researchers’ beliefs play in experiments? Every study published in a peer-reviewed scientific journal addresses potential biases, but rarely do they answer the question in a non-speculative manner, “How would these findings differ if the researcher believed something else?”
If you think about it, a hypothesis is just a synonym for presupposition, or belief. When researchers have a hypothesis, they suspect that X could be connected to Y in a particular way. They design an experiment to test this relationship. Even the most well designed experiments can be impacted by the researcher’s beliefs, so scientific studies are not as infallible as some like to imagine. It is, after all, humans conducting the experiments and despite the attempt to control for them, all humans have preferences and beliefs and experience cognitive biases.
Due to novelty bias, researchers often conduct new experiments rather than replicating studies from the past. This is exacerbated by funding preferentially going towards scientists who want to do groundbreaking research. Matthew Makel from Duke University and two of his colleagues have uncovered just how uncommon replications are in psychology, especially by independent groups. By scanning the top 100 psychology journals since 1900, and analysing 500 randomly selected articles more deeply, they showed that just 1% of publications are replications of earlier work. Of these, only 14% are direct replications that follow the original experimental methodology, while the others are conceptual replications that test related hypotheses using different context and methods.
Makel also found that around half of these replications are done by the same scientists behind the original experiments (and many are published as part of the original papers). This matters because the odds that the replication will be successful are 92% if the original authors were involved, but just 65% if done by an independent group. We think we’re being scientifically sound, yet we still manage to support our previous positions in a much higher percentage than can be explained by chance.
My point is simply that we find what we’re looking for due to confirmation bias: our tendency to seek information that confirms our preconceptions. As we go about our lives, our brains and bodies continuously seek to affirm what we believe, even if we are wholly unconscious of this fact. I don’t miss the irony in that I find myriad research that depicts the placebo effect, our bodies’ response to what we believe.
Fortunately, research is only a small part of my focus as the majority of my work is dedicated to working with clients. I witness as they come to me with disempowering beliefs and I offer them new perspectives in which to view their circumstances. As they integrate those new paradigms into presuppositions over time, I see their physical world transform in alignment with their internal representation of it. Perhaps it is the placebo effect that’s occurring, in that once they believe new feats possible, their bodies actually accomplish them. Before Roger Bannister ran a mile in under four minutes back in 1954, people didn’t believe it possible; after he accomplished it, several people repeated the feat it the same year.
I will not extrapolate Wiseman and Schlitz Two Minds study beyond what it really is—an example of the remarkable way in which we create the outcomes we believe, consciously or unconsciously. Psychologists are exploring alternative methods of experimental design that will further mitigate belief bias but at the moment there appears to be cultural inertia around transformation in this space. Thus, the onus is on readers to critically examine a particular scientist’s position as they interpret research. And be aware: far more often than not, we find what we’re looking for.