As I have written before, we may start to see a growing number of studies using randomized controlled trials (RCTs) to assess the validity of claims for transparency and participation. And in fact some donors in the open government space have already started to ask for RCT evaluations as a component of projects to be funded. But they might be skipping some important steps. A brief comment from the Simply Statistics blog suggests that a sequential approach (with RCTs at the end rather than at the start) might be more appropriate and cost-effective:

A really nice example where epidemiological studies are later confirmed by a randomized trial. From a statistician’s point of view, this is the idealized way that science would work. First, data that are relatively cheap (observational/retrospective studies) are used to identify potential associations of interest. After a number of these studies show a similar effect, a randomized study is performed to confirm what we suspected from the cheaper studies.

I think this consideration is particularly important for those funding open government evaluation work. Before jumping on the RCT bandwagon, one should first look at pre-existing knowledge to consider which questions are to be asked. But having followed the #opengov conversation for a while, I’d say this doesn’t happen very often.