Using Scientific Studies

This is from Seth Roberts’ blog, http://blog.sethroberts.net it particularly resonated with me because of a conversation I had just the other day with Dean Benton, form Leicester Tigers on this same subject. I know I am guilty of this. Thought provoking!

How to Be Wrong

February 6, 2008

There are two mistakes you can make when you read a scientific paper: You can believe it (a) too much or (b) too little. The possibility of believing something too little does not occur to most professional scientists, at least if you judge them by their public statements, which are full of cautions against too much belief and literally never against too little belief. Never. If I’m wrong — if you have ever seen a scientist warn against too little belief — please let me know. Yet too little belief is just as costly as too much.

It’s a stunning imbalance which I have never seen pointed out. And it’s not just quantity, it’s quality. One of the most foolish statements that intelligent people constantly make is “correlation does not imply causation.” There’s such a huge bias toward saying “don’t do that” and “that’s a bad thing to do” — I think because the people who say such things enjoy saying them — that the people who say this never realize the not-very-difficult concepts that (a) nothing unerringly implies causation, so don’t pick on correlations and (b) correlations increase the plausibility of causation. If your theory predicts Y and you observe Y, your theory gains credence. Causation predicts correlation.

This tendency is so common it seems unfair to give examples.

If you owned a car that could turn right but not left, you would drive off the road almost always. When I watch professional scientists react to this or that new bit of info, they constantly drive off the road: They are absurdly dismissive. The result is that, like the broken car, they fail to get anywhere: They fail to learn something they could have learned.


At 2/7/08, 10:02 AM, Anonymous Gregg said...

I get his point and each person has to find the balance between the extremes. Induction has its weakness. Just because one observes something in 30 athletes doesn't mean it is generally true. There are weaknesses with using simple linear regressions. Rarely are things a linear correlation in complex systems.

One other thing you see in athletic studies is lack of follow on studies. A small sample size is used, the study published but no subsequent studies are conducted. And yet you hear coaches quoting the study like it's the last word. The discipline of all disciplines is balance.

At 2/7/08, 11:33 AM, Anonymous Anonymous said...

Can you explain what is the difference between a pioneer and a researcher?

At 2/7/08, 11:21 PM, Anonymous sul said...

Pioneers are early settlers and communicate mostly societal wisdom i.e. sayings and cliches. A good researcher, observes, measures, studies, tests hypotheses in an "objective way" and publish their findings for others to replicate for the purpose of discovering, exploring and explaining a series of subjects, theories or events.


Post a Comment

<< Home