This is in no means an extensive and exhaustive review or introduction into how one goes about conducting research. It’s a rather quick overview of the various steps taken when conducting research, and how one can utilize and apply the conclusions of such research when forming one’s own view of a subject.
A while ago Illimitable Man wrote a great essay entitled “Critical Thinking & The Citation Needed Fallacy” in which he explored the question of whether one can justify holding a particular view on a subject without peer-reviewed academic studies to support it. This was without a doubt born from the realization that to most people, a well-reasoned, and researched empirical observation explored through logic is no longer admissible in discourse without academic studies to back it up, studies which are in turn dismissed for a variety of reasons.
This came to to me while I was watching a youtube video that can be summarized as “If one has none of the symptoms of low testosterone, in fact one’s physical manifestations indicate high testosterone, yet one’s blood test shows low testosterone values, does one have low testosterone?”
In this case, one has two contradictory observations. On one hand, if a man is carrying a lot of muscle mass, has a ton of energy, a high libido, no signs of depression and low body fat percentage, plus is performing very well in athletics, these are the antithesis of the symptoms for low testosterone. On the other hand we have an empirical observation using a quantitative, positivist method, interpreted according to the optimum range or reference range. Do we put this person on TRT or not?
How Studies Are Made
I wrote a bit on research philosophy and methods in prior essays and I’m not going to repeat it here. The general steps taken by a researcher when they want to do a study, is to begin with an observation, and then formulate a question, very often a “why“, “how” or “what” question. Then they do their background research, has anyone asked the question before, or a related question, if so what did they find out and how did they go about it. Once they finish the background research, they formulate one or more hypotheses that can then be tested with an experiment. If the tool that one has designed to test the hypothesis/hypotheses works, one analyzes data and presents the findings, if not one goes over the tool and theory steps again.
To take our example with testosterone ranges, our research question is given “What is the reference range?” (population average range) and “What is the optimal range?” (the one where you gain maximum positive benefits with minimal negative effects).
In the case of the former, that’s pretty easy to design an experiment for, all you have to do is to gather up a huge group of men, that are representative for the population, and test their testosterone. That gives us a bell curve for the male population at large, that we can divide into age brackets, and these are our reference values within a group. This is also how the government figures out average height, and weight of their population, mean and median wealth, income and many other statistical population measures.
In the case of the latter, it’s much more difficult to design an experiment, because there are more variables you have to establish, control and adjust for. You are still asking a what question, but instead of asking “What is“, you are asking “What would be ideal” for health outcomes. You can see how the answer to the two questions could be different, in the former case you are establishing the range that currently exists within a certain group of men, in the latter you are trying to find out what should be the range, so to speak.
There is also the question of “ideal” for what? If your ideal life is a long one with maximal negative consequences, that may be one range. If you don’t really care how long you live, as long as you manage to deadlift 1200 lbs, that is most likely a different range.
Once you’ve concluded on both, this serves as data that can be replicated and validated by subsequent studies done in order to track the development of both the reference range and optimal range. Replication and validation are very important for studies, because they confirm that the tool is working when applied by different people in different contexts, and it strengthens the findings. If one group of researchers studying American males ages 20 – 25, establishes a reference range of 200 nano-grams per deciliter to 1000 nano-grams per deciliter, and a group using the same tool, on the same group finds 1000 nano-grams per deciliter to 1500 nano-grams per deciliter, then the spread is big enough to question the reliability and validity of the study.
Some important factors to pay attention to when reading a study is the sample. Meaning, what is the population they gathered data from, how large was it and how were participants selected. This is of great importance because it speaks to the generalisability of the findings meaning can we apply the findings to settings other than the ones in which they were originally tested. If one tested the effect of a high protein diet on college aged male athletes, the findings may not be applicable to sedentary women ages 55 – 65. How large was the sample? If one only did an exploratory study using 20 participants, this is less reliable than one using 2000 or 20000 participants.
One also needs to pay attention to the methodology, was qualitative or quantitative methodology used? Government Census data is an example of a grand scale quantitative survey that is highly reliable because it samples the entire population of a country. Within my own area of expertise qualitative case studies are a common methodology, the findings of a single case study are generally viewed as only being applicable in the setting they were tested, meaning that in general if you do a case study of the effects of corporate culture on share performance at Apple, those findings are only applicable for Apple. In order to make them more general, more case studies must be done and the results of all the case studies reviewed to find common factors.
What was the tool used to collect the data? Self-reports are notoriously unreliable, if interviews are done what was done to minimize the influence of the researcher. How were the data interpreted, and were they interpreted consistently? Was any data eliminated from the analysis?
How to Use Studies
One could say that studies and papers for the bachelor or master level student are viewed as facts, whereas for students beyond those levels, they are viewed as indications. If many studies confirm the same findings those findings are viewed as being more probable, but not iron clad doctrine. The key concept in science is that it’s an iterative process in which the clarity of our understanding is changing and improving incrementally with each study done.
The accuracy differs somewhat from field to field, and within the STEM fields (Science, Technology, Engineering and Mathematics) the findings of studies more often reflect facts in the way that most people would define “a fact“. As clear cut empirical observations of cause, effect and natural phenomena. The average melting point of iron for instance, is something we can test in a highly controlled environment and which is fairly easy to replicate and validate. The reason or reasons why men are falling behind in school on the other hand, is very difficult to arrive at due to among others convoluted causal relationships, many different variables, multiple variables at different intensities, differences between samples and so on.
If I were to cite 5 studies that all support a link between having a written business strategy and commercial success as measured by return of investment over a 2 year period. This lends credence to the view that having an articulated and written business strategy is positively correlated with return on investment, but it doesn’t say that having such a strategy is a requirement or guarantee of such success. It also doesn’t say why this is the case, is it because of the act of writing the strategy itself or something else?
If I were to cite 5 studies where researchers utilized the same methodology that I’m proposing, what that does is build confidence that I’ve selected a robust tool to measure what I want to measure with some degree of accuracy. I can still mess up the application of the tool, by asking the wrong questions, not controlling for variables, or applying it to a non-relevant population. This was the experience with some early research in “management science“, where one in essence disregarded the human factor on inputs for a production process.
If I cite 5 studies that all support that corporations that have corporate social responsibility initiatives all perform better as measured by ROI, one could make the argument that CSR initiatives improve performance, however one could also argue that high performing companies are more likely to have the required discretionary funds to spend on CSR initiatives.
The goal of a study is to measure a phenomena, within a defined set of constraints and arrive at a measurement of that phenomena that can be replicated and validated by other researchers. Over time, this can weaken or strengthen a finding, create new avenues for research, and incrementally help us obtain a closer understanding of reality. Once one has a measurement of a large enough sample, one can generalize from those findings to whole populations. However, this doesn’t mean that it’s accurate in every case, just a certain percentage of the population.
Summary and Conclusions
I didn’t write this essay to drag down the value of having academic studies to inform you when forming a view on a given topic. I frequently use studies or writings by other people when I seek to inform myself about a new topic and this is one of our major competitive advantages as a species; the ability to learn from others. I wrote it because I’ve become concerned with the trend towards utilizing studies as hammers in a debate to prove the opposition wrong rather than to build a framework of information in order to increase the accuracy of one’s own view or to create questions about the accuracy of the other party’s view.
This was brought on by reading some comments in a Youtube video where a commenter said “X cited 5 studies, Y didn’t cite any studies“, which to me indicates that it’s the presence of the citations themselves that lend credibility in the eyes of the viewer (appeal to authority) rather than the content of said studies or the logical reasoning presented in the argument.
The trouble we are experiencing is that at this point in time, you can find a study or multiple studies that support just about every position on every single topic. When combined with confirmation bias, subjective perception and a few other cognitive traps we are susceptible to, it becomes easy to confirm one’s own view rather than challenge it.
In most cases, if you put in a search term to Google Scholar, it will spit out some results that support whatever position you want to take. Within economics, you can find studies that support every organization of the economic system from hard-core communism to laissez-faire capitalism.
Thus, what happens is that both sides of a debate engage in a Gish Gallop of sorts where they provide countless sources that confirm their position and dismiss the sources the other person refers to as being wrong. This is not a debate, or attempting to refine one’s own view of the world through challenging it, it’s a case of death by a thousand google searches.
The best learning outcome of that debate is that people leave having learned the difference between APA, MLA and the Harvard method of citing references. Their opinions haven’t changed, they have not received any food for thought, and the debaters congratulate themselves on having “debunked” each other.
To bring this back to the original question asked by Illimitable Man, the process of arriving at a true fact in science is usually observation – theory – experiment – analysis. A person makes an observation, they formulate a theory about it, they test the theory through experimentation, then analyze their findings. The other factors such as peer review, and the scientific method exists in order to ensure that the process is structured and has been adhered to so that it is possible to replicate and thus validate the study. Furthermore, that the outcome is valid, in essence, that the process is sound, and that the conclusions follow.
To give an example, I know that in my n=1 study of my individual nutrition, my digestion, energy levels, and general feeling of well-being is better if I do not eat wheat. If a study or government analysis were to come out and state “everyone should have at least 5 servings of wheat every day“, that study might be correct in the general population, but not in my particular case.
The final part I’m going to talk about is rhetoric, which I suspect is a major driver of this trend. I observed some years ago, that in debates between what could be called rationalist-empiricist atheists and religious apologists, the latter would tend to adopt a position very close to deism, rather than theism. I have also not found one who elected to openly defend a creationist position in a debate centered in the rationalist-empiricist frame. This is simply due to the fact that making arguments in favor of ideas such as immaculate conception or the re-animation of the dead is impossible within the framework of scientific evidence and rationalist reasoning.
Thus, one builds a rhetorical framework based on a much stronger position. For instance, if my position was that monogamous marriage is the only ethically correct structure for a sexual relationship. I could argue it based on moral philosophy or divine revelation. However, I would be able to build a much stronger argument if I based it in economics, using data such as better life outcomes of children raised in a stable, two-parent home.
This is because within those domains you can utilize what most people would interpret as more objective evidence, rather than arguments based in morality. Furthermore, that nobody was ever convinced to change something that is pleasurable to them because someone referred to them as a degenerate, sinner or overall horrible person. If someone feels personally attacked, and get defensive, you can be 100% correct and it still won’t convince them.
To conclude, “citation needed” began as a rhetorical tool for a situation in which an author or content creator made a statement as if it was axiomatic, when it is not. You do not need to cite a reputable scholarly source for something that is axiomatic, meaning self-evident. You can make a perfectly valid logical argument without a single citation, however you may need to provide citations to prove the soundness of your premises, provided that they are not axiomatic.
However, in order for reason to give rise to new knowledge or theories, one has to, by definition go beyond that which one can conclusively prove, after all if one’s conclusions have already been proven and validated by formal inquiry, then they are per definition not new.