There is an interesting article in the New Yorker, “The Truth Wears Off ” which demonstrates somewhat dramatically a known problem in “HARD” science.
1) Repeating a measurement is hard.
2) Your improvements in measurement skill, can look like data.
How does that work?
Ok, so you are looking for a result. Occurrences of the word AND in a paper. You aren’t using a computer, so you thumb through a page and count the ANDS – there are 10 of them. Repeat – 14, Repeat – 9, Repeat 14. Ok, so you believe 14 is the correct answer, you drop the 9 and 10 as screw ups, and take the 14. Maybe you hedge your bet and say (10 + 14 + 14)/ 3 “My average answer was about 13.”
Now, we are assuming this is a HARD science, so there is a real number of ANDs. The computer may count it, and the computer comes up with 14. You feel justified. If you repeat the test, you get 14 every time. There, you believe you have done HARD science. False. You faked your error bars, so you have no idea what your real answer is or was.
Repeat the test on a new page, and get 9, 10, 11, 10. Yeah, you are better at this, so you don’t screw up as often. What you are measuring is your failure rate. Your testable error bar is roughly 1, but when you started, your error bar was much higher, like 4.
SO, when you measure something, like someone’s reaction to medicine, you have to include the uncertainty about your measurement. In many fields, like medicine, your uncertainty changes. The effect you may be measuring is emphasized by your interaction with the data. You are now measuring yourself. Your error rate.
This is a statistics class picture I stole from a physics class…and here they are taking it seriously for examining statistics, but I have seen the same picture used for invalidating scientific approach. Why? Because when you send people to look for something, they get better at it. Commonly in high school and college science classes, the early samples of an object are poor. If you send students to look for bugs, leaves, or whatever at the parking lot, and later send them to the woods, the diversity and size increases dramatically…however, if you send them to the woods first, then the parking lot, you get the same result.
Why? Because the kids get better at finding good leaves, bugs, samples. The initial error bars should be doubled, with reduction of roughly 10 – 30 % occurring with each test, until you reach your limit. Many tests are really just examining the doctors skill at measurement. Unfortunately, part of the skill is lack of excitement and attention to duty. The more excited you are about an experiment, the more likely you are to fudge the data upwards. It only takes a couple fudged points to really skew a data set. Trust me.
Obviously Biology, or Climatology, are sciences. But honestly they aren’t Hard science. They aren’t squishy like psychology or sociology, which are only science by comparison to literature. Archeology and Paleontology gather information in a very scientific manner, but interpretation is terribly squishy, with very few chances to counter-test results. Biology and Climatology require human input at almost every stage, which leads to numerous experimenter fudges. And greater error than they like to admit.
And some people use computer models to generate their error bar, which … when the results are falling far from their error bar, means that they didn’t properly generate the error bar. If it doesn’t encompass the results, you clearly screwed up. Since the solar cycle folk at NASA have screwed up 3 or 4 in a row, the models apparently don’t mean squat.
Anyway, some interesting reading. (to me) Maybe it will be fun for you. (Better than my bad poetry at least.)
Oh, and some fellow in Italy thinks he has cold fusion . Honestly, it looks like an interesting Nickel Hydride battery. This is an example of measurment problems, again. These folks are putting into the experiment more energy than they get out…but if you only measure input electricity, they seem to be making more heat than the electricity would generate. As their measurement skills improve, the results will slowly decrease, until they drop below unity. Energy in > Energy out. So saith the Lord Thermodynamics, who may not be an angel of God, but his Entropy Sword (which can be expressed as fire) hasn’t let a human into the garden of Perpetual Motion Machines since time began. Not likely to start now.