The Office of the Director of National Intelligence has announced a multi-year research effort to develop and test “large-scale, structured collaboration methods to improve reasoning” with the goal of “improving analysts’ and decision makers’ understanding of the evidence and assumptions that support–or conflict with–their conclusions.”

The Crowdsourcing Evidence, Argumentation, Thinking and Evaluation, or CREATE, program “is a unique opportunity to extend crowdsourcing beyond its traditional applications,” the office said. “CREATE will combine crowdsourcing with structured techniques to improve reasoning on complex analytic issues. The resulting technology will be valuable not just to intelligence analysis but also to science, law, and policy–in fact, to any domain where people must think their way through complex questions.”

“The increasingly complex questions faced by today’s analysts require not only better answers, but clearer understanding and communication of conflicting evidence, knowledge gaps, and degrees of uncertainty. CREATE systems will help analysts explain to decision makers why judgments were made, why seemingly plausible alternatives were rejected, and the major gaps in what is known,” it said.

The agency has awarded research contracts to several universities, as well as separate contracts to independently test the new systems.