TY - UNPB

T1 - There is more to seeing than meets the eye

T2 - Observational Studies, Research Synthesis, and the Social Sciences

AU - Kongsted, Hans Christian

AU - Konnerup, Merete

PY - 2009

Y1 - 2009

N2 - We argue in this paper that mechanical adoption by the social sciences of the medical standard for synthesizing research on effectiveness as embodied in the Cochrane Collaboration is likely to be associated with a loss of valid evidence on the effectiveness of public policies. The de jure standard of the Cochrane Handbook discards all research designs other than randomized controlled trials (RCT), except where the latter is deemed infeasible. Still, Cochrane also prescribes quality grading to reflect varying degrees of susceptibility to selection bias and hence varying degrees of internal validity within this RCT universe. We conduct a search of the Cochrane Library to provide quantitative evidence on the de facto standard as it unfolds in actual Cochrane reviews. We find that there is widespread inclusion of quasi-randomized trials outside the strict RCT standard. Observation-based studies, on the other hand, are rarely included. On this background, we discuss recent statistical methods for valid causal inference in observational studies and note that the assumptions required for a gold standard RCT correspond one-for-one to the assumptions required for observation-based research designs to be internally valid. A search of the Campbell Collaboration's Library shows that there is little evidence of significant differences in effect size between experimental and observational studies in current Campbell reviews. We advocate a broad inclusion criterion for research designs and a case-by-case critical assessment of internal validity of all the included primary studies - experimental as well as observational - as a preferable strategy for synthesizing research on effectiveness in social science.

AB - We argue in this paper that mechanical adoption by the social sciences of the medical standard for synthesizing research on effectiveness as embodied in the Cochrane Collaboration is likely to be associated with a loss of valid evidence on the effectiveness of public policies. The de jure standard of the Cochrane Handbook discards all research designs other than randomized controlled trials (RCT), except where the latter is deemed infeasible. Still, Cochrane also prescribes quality grading to reflect varying degrees of susceptibility to selection bias and hence varying degrees of internal validity within this RCT universe. We conduct a search of the Cochrane Library to provide quantitative evidence on the de facto standard as it unfolds in actual Cochrane reviews. We find that there is widespread inclusion of quasi-randomized trials outside the strict RCT standard. Observation-based studies, on the other hand, are rarely included. On this background, we discuss recent statistical methods for valid causal inference in observational studies and note that the assumptions required for a gold standard RCT correspond one-for-one to the assumptions required for observation-based research designs to be internally valid. A search of the Campbell Collaboration's Library shows that there is little evidence of significant differences in effect size between experimental and observational studies in current Campbell reviews. We advocate a broad inclusion criterion for research designs and a case-by-case critical assessment of internal validity of all the included primary studies - experimental as well as observational - as a preferable strategy for synthesizing research on effectiveness in social science.

M3 - Working paper

BT - There is more to seeing than meets the eye

PB - Centre for the Philosophy of Natural and Social Science, London School of Economics

ER -