The Use of Consensus Reasoning

At my debate at Liberty University with Max Andrews, one of the things that many theists apparently thought was lousy about my argument was what they perceived to be appeals to authority. But, really, I was doing something quite stronger; I was appealing to scientific consensus.

My friend Cory has an excellent explanation on his blog Cornelioid, where he analyzes the use of consensus reasoning in arguments over water flouridation:

An especially common authority cited with regard to an issue is the expert—someone trained, knowledgeable, and experienced in the issue itself or in some discipline(s) relevant thereto, who can convey and contextualize current best knowledge on the subject and its implications for people’s lives. We rely on the experts who prescribe our meds and tune up our bikes as much as we rely on those who inform our healthcare professionals and policymakers. When an expert speaks about their area of expertise, we tend to listen. (While it’s another story in itself, we even tend to listen carefully to experts when we’re predisposed to—even intent upon—doubting them. When we fall into denial, we crave their validation even as we reject their credibility.)

There is, however, a categorically stronger authority than any one expert or even team of experts, whose testimony may bear directly upon an issue, and that is an expert consensus. While there are a variety of important reasons for this, especially in the sciences—familiarity with the nuances of the issue, the statistical training and critical thinking skills required to understand current knowledge and make reasonable (and financially attractive) predictions, the (increasingly) collaborative nature of research, the formal peer-review process, the lure of novelty and especially of debunkery, etc.—a simple probabilistic analogy illustrates how, in the absence of other intelligible evidence, an expert consensus is far and away a more reliable source than any individual expert.

He continues by giving a statistical analysis of scientific consensus:

Conservatively taking a consensus to consist of 90% of experts and (ultraconservatively) 100 experts to exist, on the assumption of independent assessments of the existing evidence we would expect, when any given expert might be wrong with probability p, a consensus of experts to simultaneously be wrong with probability

q=P(90\leq X\leq 100|X\sim B(100,p))=\sum_{k=90}^{100}{100\choose k}p^k(1-p)^{100-k},

which could only ever be 5% larger than p (q/p is maximized at around p=0.95) and is profoundly less than p whenever p is not close to 1 (p<0.8 already implies that q<p/140)—and in almost any setting it is absurd to expect that any expert would incorrectly assess a body of evidence with probability greater than 0.5 (since it would be absurd to consider anyone an expert in the first place who could not provide a more reliable assessment than pure chance). While the assumption of independent assessments is itself incredibly naïve, in cases (such as this) involving a community of experts tens of thousands strong, it is entirely reasonable to expect that at least 100 independent assessments of the evidence have been conducted, and therefore that a few orders of magnitude constitute a charitable bound on the relative reasonableness of an expectation that some experts have it wrong and one that a consensus has it wrong.* (The historico-mathematically literate will have recognized the Marquis de Condorcet’s jury principle.)

Read the rest of his post here.

5 Comments

  • bruce
    March 30, 2013 - 1:52 pm | Permalink

    This model abstracts away important elements of the scientific process, making consensus appear more reliable than it is in reality. The reason is that a strong consensus influences what research is done and what evidence is discovered. When all the department chairs, journal editors, and government grant givers are supporters of an idea, a researcher is wise not to dissent.

    This situation doesn’t pose a significant problem when the evidence is clear because the initial consensus had powerful evidence to support their view and it is likely correct. However, this can pose a significant problem when we only receive a slow trickle of weak evidence. In this case, we can imagine the unbiased conclusion of the data as a random process over time that randomly bounces around hypotheses with a slow drift towards the correct one. When the effect of consensus is included with this model, the unbiased conclusion of the data becomes stuck to the first hypothesis to gain a consensus because this turns off the trickle of evidence. Updating based on the prior derived from this model, consensus could lower one’s credence in an idea because it can be a sign that unknown evidence is biased against the idea.

    • Dan Linford
      March 30, 2013 - 2:28 pm | Permalink

      Thank you for the qualification.

      I don’t think I’d object to that, nor do I think that it invalidates my appeal to scientific consensus in the debate that I participated in. In the debate, I had stated that I did not know where the universe came from and would await the scientific consensus of physical cosmologists. That was a point which a large portion of the audience apparently didn’t like, but I’d stick by it; and I think that your qualification does not destroy the justification that non-cosmologists have in appealing to the scientific consensus amongst cosmologists for an explanation as to the origins of the universe.

    • April 13, 2013 - 7:41 pm | Permalink

      bruce, you’re absolutely right that the model abstracts the process almost comically; it serves only as a useful point of reference for thinking about the relative reliability of lone experts versus expert communities. However, i’ll suggest a couple of considerations regarding your objection: First, contrary results are sexy. The potential prestige of overturning even an emerging consensus far outweighs the risk of disappointing one’s sponsors, at least among scientists i’ve encountered. Second, “established” results do not stop being tested once they become widely accepted. I’ll have a bit to say about that in my follow-up.

  • fonzo
    March 30, 2013 - 2:32 pm | Permalink

    interesting read, but he left out one notable exception that almost unravels the whole thing, and that is the problem of zealotry and bias, even amongst experts. In the case of zealotry a supposed “expert” can do far worse than 50% accuracy if he unabashedly denies sources of contradictory information. But that’s not even the worse part.

    Zealotry can unravel even the most informed scientific consensus by the most tempting of all arguments: an appeal to ignorance. It is enough for a moderately informed zealot to simply don the cap of an expert and pose the question “you don’t know what they are talking about so how can you believe them?” while presenting an alternative the audience (consisting of lay persons) is incapable of refuting. This is how I believe religion, loop quantum gravity, and planet x theories originate and enter the mainstream consciousness and why they are so hard to eradicate.

    • April 13, 2013 - 8:24 pm | Permalink

      fonzo, those are both good caveats. To save myself a bit of face, i’ll point out that i did acknowledge the first problem you mention in my penultimate paragraph, where i say that one may indeed refute a scientific consensus by effectively becoming a skeptical expert: expert enough to understand the literature, and skeptic enough to prevent being cajoled into the same biases and prejudices that have compromised it. My favorite contemporary example is Cordelia Fine, whose exposé of gender difference studies has, i understand, been a rude awakening for many neuroscientists and evolutionary psychologists.

      The second problem you mention is not one of consensus itself but of how it is conveyed (or not) to a lay public, i.e. it is a problem not of science but of science communication (though i believe, and Dan can check me on this, that from a lay epistemic perspective the problems are the same). The damage done by scientist-zealots has indeed been severe. With respect to community water fluoridation, there in fact exists an entire journal, Fluoride, founded and edited by anti-CWF activists, which has spent 45 years demonizing element #9. I’ll get into more detail in a future post.

  • Leave a Reply