GenAI: Substitute or Complement in Evidence Synthesis: A discrete choice experiment with Italian policymakers. First draft available soon.
Abstract
Credible evidence on the causal effects of public policies is increasingly produced by the academic community, and evidence-based policymaking is increasingly demanded by governments, yet the production of high-quality evidence syntheses remains costly, slow, and often poorly aligned with policymakers’ time constraints. Recent advances in generative artificial intelligence (GenAI) promise efficiency gains in evidence synthesis, but their adoption in policy contexts depends on whether policymakers trust and value AI-generated evidence. This paper studies policymakers’ preferences for the use of GenAI in producing scientific evidence syntheses for policy design. We implement a discrete choice experiment with 813 Italian policymakers, eliciting trade-offs over synthesis production method (expert-produced, AI-generated, or hybrid AI-with-expert-review), cost, and delivery time. In a between-subject design, we additionally vary whether the AI model is described as general-purpose or specifically validated for evidence synthesis. We find strong aversion to fully automated AI-produced syntheses, even when conditional on cost and time savings. By contrast, the hybrid AI–expert approach is systematically preferred over both expert and AI approaches, conditional on cost and delivery time. Explicit validation of the AI model significantly increases acceptance of AI-assisted synthesis, reducing resistance to automation and increasing preferences for hybrid production. Heterogeneity analyses show that these effects vary with attitudes toward AI but not with gender or institutional role. Overall, the results indicate that policymakers value AI primarily as a complement to human expertise rather than as a substitute, highlighting the importance of task-specific validation and expert oversight for the legitimate adoption of AI in evidence-based policymaking.
