The Intelligence Game: Assessing Delphi Groups and Structured Question Formats

Wintle, Bonnie; Mascaro, Steven; Fidler, Fiona; McBride, Marissa; Burgman, Mark; Flander, Louisa; Saw, Geoff; Twardy, Charles; Lyon, Aidan und Manning, Brian (2012): The Intelligence Game: Assessing Delphi Groups and Structured Question Formats. In: Proceedings of the 5th Australian Security and Intelligence Conference. S. 1-26

Abstract

In 2010, the US Intelligence Advanced Research Projects Activity (IARPA) announced a 4-year forecasting “tournament”. Five collaborative research teams are attempting to outperform a baseline opinion pool in predicting hundreds of geopolitical, economic and military events. We are contributing to one of these teams by eliciting forecasts from Delphi-style groups in the US and Australia. We elicit probabilities of outcomes for 3-5 monthly questions, such as: Will Australia formally transfer uranium to India by 1 June 2012? Participants submit probabilities in a 3-step interval format, view those of others in their group, share, rate and discuss information, and then make a second private judgement. Performance is assessed using Brier scores. After Year 1, we ranked second of five teams in the competition. The Brier scores from the US Delphi groups improved on the baseline scores by 10%, the prediction market operated by our team in the US beat the baseline by 47%, and the Australian Delphi groups outperformed the baseline by 51% (answering different, matched questions to the US groups). The Australian groups were more socially and demographically diverse than the US groups. Group diversity may be an important factor determining the forecasting performance of the aggregated predictions.

• Dokumententyp: Buchbeitrag
• Fakultät: Philosophie, Wissenschaftstheorie und Religionswissenschaft > Munich Center for Mathematical Philosophy (MCMP) ; Philosophie, Wissenschaftstheorie und Religionswissenschaft > Munich Center for Mathematical Philosophy (MCMP) > Philosophy of Science
• Themengebiete: 100 Philosophie und Psychologie > 100 Philosophie
• Sprache: Englisch