In reporting the scandal about the Norwegian cancer research, the NYT had the following unnerving comments:

A special feature of epidemiological studies like Dr. Sudbo’s is that they involve large numbers of patients and are unlikely to be repeated by other laboratories. Replication is considered the most reliable test of scientific quality.

The full text is here

It is a severe accusation to a scientific field (and in fact to all social science fields) in which observational studies are popular.

It seems to me that the author, Nicholas Wade, doesn’t understand the meaning of “replication,” at least in social science. He narrowly defined “replication” as “repeated by other laboratories.” We certainly won’t be able to replicate the study per se, and we may not be able to get the same numbers (e.g., odds ratios) from other studies. However, what is important in social science is that we can replicate the study findings, and that the findings are consistent in different populations and in different types of studies.

Randomly picking up any newspaper, there will be a column called health, and within it, there are unsolicited opinions on how we should live and what we should eat. If you follow them closely (as most health-conscious people will do), you will be quickly overwhelmed by the conflicting advice. Common people start to wonder whether these study findings make any sense at all. Indeed, it is not their fault.

The common practice in all health related studies, and even worse in politics and economic research, encourages bold statements, a little bit overstretching the truth, and downplaying the limit of generalization. In fact, journals often refuse to publish negative results. Therefore, we see thousands of positive results initially being hailed by everybody, and later being ignored by all researchers except historians. The reasons? Nobody can replicate the initial findings, and no journal is willing to take the negative report.

A JAMA article a few months ago examined the consistency between observational studies and randomized trials. They found that one fourth of the examined observational studies were refuted by later randomized trials. For example, the hormone replacement therapy used in post menopause women has been proven to be harmful by the Women’s Health Initiative Study because the combined estrogen-progestin may increase the risk of breast cancer and heart disease. The effects of vitamin E and C supplements have also been shown to be exaggerated (if not false) in observational studies. Coincidently, the first seemingly “decisively” positive findings for hormone and vitamins are all from the two population studies initiated by Harvard.

However, we should be cautious that most randomized trials were conducted among high risk patients who might already be in great danger of death or other heath consequences. If the effects of vitamins and other antioxidants are small but helpful in long term, these trials are unlikely to confirm the positive findings from observational studies.

Furthermore, we should also acknowledge that observational studies have contributed immensely to our understanding of our lifestyle, health risks, and disease progress. Without these studies, we may not know that high fat in foods is unhealthy, and high lipids in blood are detrimental to our body. We may still enjoy the sedentary life without knowing that exercise can extend our lives and improve the quality of lives. A few unverified or scandalous reports should not forgo our trust on rigorous science. But is social science rigorous?

Recent scandals and the demeaning comment in the NYT are the wakeup call to all researchers. Something should be done to rectify this. But, what and how?