The Delphi technique as a forecasting tool: issues and analysis

https://doi.org/10.1016/S0169-2070(99)00018-7Get rights and content

Abstract

This paper systematically reviews empirical studies looking at the effectiveness of the Delphi technique, and provides a critique of this research. Findings suggest that Delphi groups outperform statistical groups (by 12 studies to two with two ‘ties’) and standard interacting groups (by five studies to one with two ‘ties’), although there is no consistent evidence that the technique outperforms other structured group procedures. However, important differences exist between the typical laboratory version of the technique and the original concept of Delphi, which make generalisations about ‘Delphi’ per se difficult. These differences derive from a lack of control of important group, task, and technique characteristics (such as the relative level of panellist expertise and the nature of feedback used). Indeed, there are theoretical and empirical reasons to believe that a Delphi conducted according to ‘ideal’ specifications might perform better than the standard laboratory interpretations. It is concluded that a different focus of research is required to answer questions on Delphi effectiveness, focusing on an analysis of the process of judgment change within nominal groups.

Introduction

Since its design at the RAND Corporation over 40 years ago, the Delphi technique has become a widely used tool for measuring and aiding forecasting and decision making in a variety of disciplines. But what do we really understand about the technique and its workings, and indeed, how is it being employed? In this paper we adopt the perspective of Delphi as a judgment or forecasting or decision-aiding tool, and we review the studies that have attempted to evaluate it. These studies are actually sparse, and, we will argue, their attempts at evaluating the technique have been largely inappropriate, such that our knowledge about the potential of Delphi is still poor. In essence, this paper relates a critique of the methodology of evaluation research and suggests that we will acquire no great knowledge of the potential benefits of Delphi until we adopt a different methodological approach, which is subsequently detailed.

Section snippets

The nature of Delphi

The Delphi technique has been comprehensively reviewed elsewhere (e.g., Linstone and Turoff, 1975, Hill and Fowles, 1975, Lock, 1987, Parenté and Anderson-Parenté, 1987, Stewart, 1987, Rowe et al., 1991), and so we will present a brief review only. The Delphi technique was developed during the 1950s by workers at the RAND Corporation while involved on a U.S. Air Force sponsored project. The aim of the project was the application of expert opinion to the selection – from the point of view of a

The study of Delphi

Since the 1950s, use of Delphi has spread from its origins in the defence community in the U.S.A. to a wide variety of areas in numerous countries. Its applications have extended from the prediction of long-range trends in science and technology to applications in policy formation and decision making. An examination of recent literature, for example, reveals how widespread is the use of Delphi, with applications in areas as diverse as the health care industry (Hudak, Brooke, Finstuen & Riley,

Evaluative studies of Delphi

We attempted to gather together details of all published (English-language) studies involving evaluation of the Delphi technique. There were a number of types of studies that we decided not to include in our analysis. Unpublished PhD theses, technical reports (e.g., of the RAND Corporation), and conference papers were excluded because their quality is less assured than peer-reviewed journal articles and (arguably) book chapters. It may also be argued that if the studies reported in these

Findings

In this section, we consider the results obtained by the evaluative studies as summarised in Table 2, to which the reader is referred. We will return to the details in Table 1 in our subsequent critique.

A critique of technique-comparison studies

Much of the criticism of early Delphi studies centred on their ‘sloppy execution’ (e.g., Stewart, 1987). Among specific criticisms were claims that Delphi questionnaires were poorly worded and ambiguous (e.g., Hill & Fowles, 1975) and that the analysis of responses was often superficial (Linstone, 1975). Reasons given for the poor conduct of early studies ranged from the technique’s ‘apparent simplicity’ encouraging people without the requisite skills to use it (Linstone & Turoff, 1975), to

Findings of process studies

Technique-comparison studies ask the question: ‘does Delphi work?’, and yet they use technique forms that differ from one study to the next and that often differ from the ideal of Delphi. Process studies ask: ‘what is it about Delphi that makes it work?’, and this involves both asking and answering the question: ‘what is Delphi?’

We believe that the emphasis of research should be shifted from technique-comparison studies to process studies. The latter should focus on the way in which an initial

Conclusion

This paper reviews research conducted on the Delphi technique. In general, accuracy tends to increase over Delphi rounds, and hence tends to be greater than in comparative staticized groups, while Delphi panels also tend to be more accurate than unstructured interacting groups. The technique has shown no clear advantages over other structured procedures.

Various difficulties exist in research of this technique-comparison type, however. Our main concern is with the sheer variety of technique

Biographies: Gene ROWE is an experimental psychologist who gained his PhD from the Bristol Business School at the University of the West of England (UWE). After some years at UWE, then at the University of Surrey, he is now at The Institute of Food Research (IFR), Norwich. His research interests have ranged from expert systems and group decision support, to judgment and decision making more generally. Lately, he has been involved in research on risk perception and public participation

References (64)

  • M.R Kastein et al.

    Delphi, the issue of reliability: a qualitative Delphi study in primary health care in the Netherlands

    Technological Forecasting and Social Change

    (1993)
  • D.G Myers

    Polarizing effects of social comparisons

    Journal of Experimental Social Psychology

    (1978)
  • R Ono et al.

    Assessing the validity of the Delphi technique

    Futures

    (1994)
  • W.E Riggs

    The Delphi method: an experimental evaluation

    Technological Forecasting and Social Change

    (1983)
  • J Rohrbaugh

    Improving the quality of group judgment: social judgment analysis and the Delphi technique

    Organizational Behavior and Human Performance

    (1979)
  • G Rowe et al.

    The impact of task characteristics on the performance of structured group forecasting techniques

    International Journal of Forecasting

    (1996)
  • G Rowe et al.

    The Delphi technique: a re-evaluation of research and theory

    Technological Forecasting and Social Change

    (1991)
  • J.A Sniezek

    An examination of group process in judgmental forecasting

    International Journal of Forecasting

    (1989)
  • J.A Sniezek

    Groups under uncertainty: an examination of confidence in group decision making

    Organizational Behavior and Human Decision Processes

    (1992)
  • J.A.G.M Van Dijk

    Delphi questionnaires versus individual and group interviews: a comparison case

    Technological Forecasting and Social Change

    (1990)
  • G Wright et al.

    The role and validity of judgment in forecasting

    International Journal of Forecasting

    (1996)
  • J.S Armstrong

    Long range forecasting: from crystal ball to computer

    (1985)
  • J.S Armstrong et al.

    Return postage in mail surveys: a meta-analysis

    Public Opinion Quarterly

    (1987)
  • R.J Best

    An experiment in Delphi estimation in marketing decision making

    Journal of Marketing Research

    (1974)
  • D.M Boje et al.

    Group confidence pressures in iterative decisions

    Management Science

    (1982)
  • K Brockhoff

    The performance of forecasting groups in computer dialogue and face to face discussions

  • K Brockhoff

    Forecasting quality and information

    Journal of Forecasting

    (1984)
  • J Cohen

    The Earth is round (p<.05)

    American Psychologist

    (1994)
  • F Dagenais

    The reliability and convergence of the Delphi technique

    The Journal of General Psychology

    (1978)
  • N.C Dalkey et al.

    An experimental application of the Delphi method to the use of experts

    Management Science

    (1963)
  • M Deutsch et al.

    A study of normative and informational social influences upon individual judgment

    Journal of Abnormal and Social Psychology

    (1955)
  • H.J Einhorn et al.

    Quality of group judgment

    Psychological Bulletin

    (1977)
  • Cited by (1603)

    View all citing articles on Scopus

    Biographies: Gene ROWE is an experimental psychologist who gained his PhD from the Bristol Business School at the University of the West of England (UWE). After some years at UWE, then at the University of Surrey, he is now at The Institute of Food Research (IFR), Norwich. His research interests have ranged from expert systems and group decision support, to judgment and decision making more generally. Lately, he has been involved in research on risk perception and public participation mechanisms in risk assessment and management.

    George WRIGHT is a psychologist with an interest in the judgmental aspects of forecasting and decision making. He is Editor of the Journal of Behavioral Decision Making and an Associate Editor of the International Journal of Forecasting and the Journal of Forecasting. He has published in such journals as Management Science, Current Anthropology, Journal of Direct Marketing, Memory and Cognition, and the International Journal of Information Management.

    View full text