(or, “On the Proper Use of Anecdotes”)
According to most people, the pinnacle of scientific investigation is the randomized controlled trial (RCT). In this setup, the procedure and investigation are determined in advance, and a representative sample of subjects are randomly assigned to different treatment groups, whose scores on some measure are compared. The idea is to minimize bias in the sampling / group assignment steps, and to average out any specific characteristics of any particular person(s) in the study. The quirks or personal experiences of any one person are precisely what I don’t care about in this design: if one person is very old / comes from a particular culture / has an unusual family history, I hope that by mixing them into a group with people who are young / have other cultural backgrounds / have different family histories, I can remove the effect of these quirks in the final analysis.
But in some fields one also finds Case Studies, which have a very different design in that it focuses on one particular subject. Often the subject of a Case Study is someone who is as unusual as possible. It has been summarized thus:
An average, or typical case, is often not the richest in information. In clarifying lines of history and causation it is more useful to select subjects that offer an interesting, unusual or particularly revealing set of circumstances. A case selection that is based on representativeness will seldom be able to produce these kinds of insights.
The RCT is an excellent design precisely because it’s maximally generalizable, from the study in the Real World up to the Meta Level; the result of a good RCT gives you confidence that you’ve learned something you can apply to the population at large. A Case Study is, in this sense, a very counterintuitive design. Even if you were trying to select a “typical” person for a Case Study, you’d run into all the biases and particulars that are intrinsic to that person, and anyway if you knew the characteristics of a “typical” person you wouldn’t need to investigate one. On top of this, one often selects not for typical, but rather for extreme cases. A Case Study, to put it bluntly, is an anecdote, and there’s little reason to expect that an individual person’s anecdote will apply to anyone else.
Said another way: If you interview someone in the very tail of some distribution, you can say very little about others lying closer to the mean. The most extreme schizophrenic can tell you about their experience, and it will be as interesting as anything can be, but what does it tell you about schizophrenia in general? What can we take from a Case Study that is useful at the Meta Level?
The puzzle disappears when you start to think of science not merely as the turn-crank set of rules we teach our children about, but rather as a creative, generative enterprise. A good RCT is designed to test for some property that you’re already interested in, but how do you know what is interesting in the first place? One answer is: A good Case Study. Before there were RCTs on schizophrenia, before there was a notion of “schizophrenia”, there were a lot of people with what looked like extreme symptoms of delusion or hallucination, and some (proto) scientists who investigated what was going on with them. As more and more cases were investigated and found to be similar, a consensus emerged that there was ‘a thing’ out there afflicting people and it was given a name; once it had a name, it became the fodder for normal science, including diagnosis, DSMs, and RCTs. A Case Study represents the early days of this sort of process.
Pirsig talks about questions that cut broadly and those that cut deeply, which is precisely the point here. RCTs cut widely and cover a lot of ground you can already see, whereas a good Case Study cuts deeply into unknown territory. In a very real sense, the author of a Case Study is feeling around in the dark, not seeing clearly what they might be learning or how it might apply to something else. If you’re too rigid, then you’ll think you already know what you’re looking for and are likely wasting your time; if you’re too flexible, you’ll generalize too fast, and will likely confuse particulars of the individual for general properties of the population. There’s a sweet spot to be found in a Case Study, and this is where new insights can hide.
\\
As it is with scientists, so it is with humans. The Case Study is a model for new experiences in life, ones that you possibly didn’t know would be interesting. Some examples come to mind, each of which deserves a post of its own:
Experiential Case Studies: Trying new things, not knowing whether you’ll like them or not. What often arises is a cutting through to something new that you didn’t know could be experienced. On the benign end is trying new foods or meeting new people at new venues, knowing full-well these might not be useful or enjoyable. At the extreme end lie pharmaceutical experiences, e.g. from hallucinogenic drugs, or thrill-seeking like scuba or sky diving.
Epistemological Case Studies: Listening to crackpot theories or extreme views on a topic, even when they’re unlikely to be true. Say what you will, but crackpots are creative. Most of what they say will be literally false and it won’t be clear what can be used for real insight, but that’s partly the point.
Political Case Studies: The recent political emphasis on “lived experience” of particular people at particular times in particular environments. No one person’s lived experience can or should trump a statistical analysis, but lived experiences can and should guide what analyses might be interesting to conduct in the future.
Life is an experiment, but it’s often not an RCT. Trying new things, listening to crackpots, and learning from “lived experiences” can be a kind of Case Study in creativity, in experience, or in your own mental capability. Approach them with interest and curiosity, not knowing what you will take away.