Tuesday, August 31, 2010

Reference Manual on Scientific Evidence

The Federal Judicial Center's Reference Manual on Scientific Evidence is a very interesting look at how the United States federal judiciary has been briefed on scientific evidence.

Of particular interest to paranormal investigators and so-called "proto-scientists" (the term most often bandied about in support of pseudo-scientific paranormal hucksters who claim a lot of evidence, but never deliver the goods) is the section titled "How Science Works", from p. 68 to p. 82, by Dr. David Goodstein (pictured at left), who served as the Vice Provost of the California Institute of Technology from 1988 until 2007.

Dr. Goodstein identified a very useful series of what he termed "myths" and "facts" about science:

“In matters of science,” Galileo wrote, “the authority of thousands is not worth the humble reasoning of one single person.” Doing battle with the Aristotelian professors of his day, Galileo believed that appeal to authority was the enemy of reason. But, contrary to Galileo’s famous remark, the fact is that authority is of fundamental importance to science. If a paper’s author is a famous scientist, I think the paper is probably worth reading. However, an appeal from a scientific wanna-be, asking that his great new discovery be brought to the attention of the scientific world, is almost surely not worth reading (such papers arrive in my office, on the average, about once a week).
The triumph of reason over authority is just one of the many myths about science. Here’s a brief list of others:

Myth: Scientists must have open minds, being ready to discard old ideas in favor of new ones.

Fact: Because science is an adversary process in which each idea deserves the most vigorous possible defense, it is useful for the successful progress of science that scientists tenaciously hang on to their own ideas, even in the face of contrary evidence (and they do, they do).

Myth: Science must be an open book. For example, every new experiment must be described so completely that any other scientist can reproduce it.

Fact: There is a very large component of skill in making cutting-edge experiments work. Often, the only way to import a new technique into a laboratory is to hire someone (usually a postdoctoral fellow) who has already made it work elsewhere. Nevertheless, scientists have a solemn responsibility to describe the methods they use as fully and accurately as possible. And, eventually, the skill will be acquired by enough people to make the new technique commonplace.

Myth: When a new theory comes along, the scientist’s duty is to falsify it.

Fact: When a new theory comes along, the scientist’s instinct is to verify it. When a theory is new, the effect of a decisive experiment that shows it to be wrong is that both the theory and the experiment are quickly forgotten. This result leads to no progress for anyone in the reward system. Only when a theory is well established and widely accepted does it pay off to prove that it’s wrong.

Myth: Real science is easily distinguished from pseudoscience.

Fact: This is what philosophers call the problem of demarcation: One of Popper’s principal motives in proposing his standard of falsifiability was precisely to provide a means of demarcation between real science and impostors. For example, Einstein’s theory of relativity (with which Popper was deeply impressed) made clear predictions that could certainly be falsified if they were not correct. In contrast, Freud’s theories of psychoanalysis (with which Popper was far less impressed) could never be proven wrong. Thus, to Popper, relativity was science but psychoanalysis was not. As I’ve already shown, real scientists don’t do as Popper says they should. But quite aside from that, there is another problem with Popper’s criterion (or indeed any other criterion) for demarcation: Would-be scientists read books too. If it becomes widely accepted (and to some extent it has) that falsifiable predictions are the signature of real science, then pretenders to the throne of science will make falsifiable predictions, too. There is no simple, mechanical criterion for distinguishing real science from something that is not real science. That certainly doesn’t mean, however, that the job can’t be done.

Myth: Scientific theories are just that: theories. All scientific theories are eventually proved wrong and are replaced by other theories.

Fact: The things that science has taught us about how the world works are the most secure elements in all of human knowledge. I must distinguish here between science at the frontiers of knowledge (where by definition we don’t yet understand everything and where theories are indeed vulnerable) and textbook science that is known with great confidence. Matter is made of atoms, DNA transmits the blueprints of organisms from generation to generation, light is an electromagnetic wave; these things are not likely to be proved wrong. The theory of relativity and the theory of evolution are in the same class. They are still called theories for historic reasons only. The satellite navigation system in my car routinely uses the theory of relativity to make calculations accurate enough to tell me exactly where I am and to take me to my destination with unerring precision. It should be said here that the incorrect notion that all theories must eventually be wrong is fundamental to the work of both Popper and Kuhn, and these theorists have been crucial in helping us understand how science works. Thus, their theories, like good scientific theories at the frontiers of knowledge, can be both useful and wrong.

Myth: Scientists are people of uncompromising honesty and integrity.

Fact: They would have to be if Bacon were right about how science works, but he wasn’t. Scientists are rigorously honest where honesty matters most to them: in the reporting of scientific procedures and data in peer-reviewed publications. In all else, they are ordinary mortals like all other ordinary mortals.
Good stuff, understood by all true skeptics, if not by the believers and disbelievers in the paranormal.

Paul Kimball

No comments: