The Delphi technique as a forecasting tool: issues and analysis
Introduction
Since its design at the RAND Corporation over 40 years ago, the Delphi technique has become a widely used tool for measuring and aiding forecasting and decision making in a variety of disciplines. But what do we really understand about the technique and its workings, and indeed, how is it being employed? In this paper we adopt the perspective of Delphi as a judgment or forecasting or decision-aiding tool, and we review the studies that have attempted to evaluate it. These studies are actually sparse, and, we will argue, their attempts at evaluating the technique have been largely inappropriate, such that our knowledge about the potential of Delphi is still poor. In essence, this paper relates a critique of the methodology of evaluation research and suggests that we will acquire no great knowledge of the potential benefits of Delphi until we adopt a different methodological approach, which is subsequently detailed.
Section snippets
The nature of Delphi
The Delphi technique has been comprehensively reviewed elsewhere (e.g., Linstone and Turoff, 1975, Hill and Fowles, 1975, Lock, 1987, Parenté and Anderson-Parenté, 1987, Stewart, 1987, Rowe et al., 1991), and so we will present a brief review only. The Delphi technique was developed during the 1950s by workers at the RAND Corporation while involved on a U.S. Air Force sponsored project. The aim of the project was the application of expert opinion to the selection – from the point of view of a
The study of Delphi
Since the 1950s, use of Delphi has spread from its origins in the defence community in the U.S.A. to a wide variety of areas in numerous countries. Its applications have extended from the prediction of long-range trends in science and technology to applications in policy formation and decision making. An examination of recent literature, for example, reveals how widespread is the use of Delphi, with applications in areas as diverse as the health care industry (Hudak, Brooke, Finstuen & Riley,
Evaluative studies of Delphi
We attempted to gather together details of all published (English-language) studies involving evaluation of the Delphi technique. There were a number of types of studies that we decided not to include in our analysis. Unpublished PhD theses, technical reports (e.g., of the RAND Corporation), and conference papers were excluded because their quality is less assured than peer-reviewed journal articles and (arguably) book chapters. It may also be argued that if the studies reported in these
Findings
In this section, we consider the results obtained by the evaluative studies as summarised in Table 2, to which the reader is referred. We will return to the details in Table 1 in our subsequent critique.
A critique of technique-comparison studies
Much of the criticism of early Delphi studies centred on their ‘sloppy execution’ (e.g., Stewart, 1987). Among specific criticisms were claims that Delphi questionnaires were poorly worded and ambiguous (e.g., Hill & Fowles, 1975) and that the analysis of responses was often superficial (Linstone, 1975). Reasons given for the poor conduct of early studies ranged from the technique’s ‘apparent simplicity’ encouraging people without the requisite skills to use it (Linstone & Turoff, 1975), to
Findings of process studies
Technique-comparison studies ask the question: ‘does Delphi work?’, and yet they use technique forms that differ from one study to the next and that often differ from the ideal of Delphi. Process studies ask: ‘what is it about Delphi that makes it work?’, and this involves both asking and answering the question: ‘what is Delphi?’
We believe that the emphasis of research should be shifted from technique-comparison studies to process studies. The latter should focus on the way in which an initial
Conclusion
This paper reviews research conducted on the Delphi technique. In general, accuracy tends to increase over Delphi rounds, and hence tends to be greater than in comparative staticized groups, while Delphi panels also tend to be more accurate than unstructured interacting groups. The technique has shown no clear advantages over other structured procedures.
Various difficulties exist in research of this technique-comparison type, however. Our main concern is with the sheer variety of technique
Biographies: Gene ROWE is an experimental psychologist who gained his PhD from the Bristol Business School at the University of the West of England (UWE). After some years at UWE, then at the University of Surrey, he is now at The Institute of Food Research (IFR), Norwich. His research interests have ranged from expert systems and group decision support, to judgment and decision making more generally. Lately, he has been involved in research on risk perception and public participation
References (64)
Combining the judgments of experts: how many and which ones?
Organizational Behaviour and Human Decision Processes
(1986)Participants’ response to the Delphi method: an attitudinal perspective
Technological Forecasting and Social Change
(1984)- et al.
Assessing the quality of expert judgment: issues and analysis
Decision Support Systems
(1994) - et al.
The Delphi method III: use of self-ratings to improve group estimates
Technological Forecasting
(1970) Methods for analyzing data from Delphi panels: some evidence from a forecasting study
Technological Forecasting and Social Change
(1987)When oracles fail – a comparison of four procedures for aggregating subjective probability forecasts
Organizational Behavior and Human Performance
(1981)- et al.
The efforts of alternative forms of knowledge representation on decision making consensus
International Journal of Man–Machine Studies
(1993) - et al.
A comparison study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups and nominal groups
Organizational Behavior and Human Performance
(1973) - et al.
The methodological worth of the Delphi forecasting technique
Technological Forecasting and Social Change
(1975) A note on aggregating opinions
Organizational Behaviour and Human Performance
(1978)
Delphi, the issue of reliability: a qualitative Delphi study in primary health care in the Netherlands
Technological Forecasting and Social Change
Polarizing effects of social comparisons
Journal of Experimental Social Psychology
Assessing the validity of the Delphi technique
Futures
The Delphi method: an experimental evaluation
Technological Forecasting and Social Change
Improving the quality of group judgment: social judgment analysis and the Delphi technique
Organizational Behavior and Human Performance
The impact of task characteristics on the performance of structured group forecasting techniques
International Journal of Forecasting
The Delphi technique: a re-evaluation of research and theory
Technological Forecasting and Social Change
An examination of group process in judgmental forecasting
International Journal of Forecasting
Groups under uncertainty: an examination of confidence in group decision making
Organizational Behavior and Human Decision Processes
Delphi questionnaires versus individual and group interviews: a comparison case
Technological Forecasting and Social Change
The role and validity of judgment in forecasting
International Journal of Forecasting
Long range forecasting: from crystal ball to computer
Return postage in mail surveys: a meta-analysis
Public Opinion Quarterly
An experiment in Delphi estimation in marketing decision making
Journal of Marketing Research
Group confidence pressures in iterative decisions
Management Science
The performance of forecasting groups in computer dialogue and face to face discussions
Forecasting quality and information
Journal of Forecasting
The Earth is round (p<.05)
American Psychologist
The reliability and convergence of the Delphi technique
The Journal of General Psychology
An experimental application of the Delphi method to the use of experts
Management Science
A study of normative and informational social influences upon individual judgment
Journal of Abnormal and Social Psychology
Quality of group judgment
Psychological Bulletin
Cited by (1603)
The impact of the metaverse on e-commerce business models – A delphi-based scenario study
2024, Technology in SocietyConsensus methodology to investigate the crucial referral criteria to pharmacist-led counseling clinics in Makkah City
2024, Saudi Pharmaceutical JournalBlockchain's value proposition for online gambling: The operators' perspective
2024, Technological Forecasting and Social Change
Biographies: Gene ROWE is an experimental psychologist who gained his PhD from the Bristol Business School at the University of the West of England (UWE). After some years at UWE, then at the University of Surrey, he is now at The Institute of Food Research (IFR), Norwich. His research interests have ranged from expert systems and group decision support, to judgment and decision making more generally. Lately, he has been involved in research on risk perception and public participation mechanisms in risk assessment and management.
George WRIGHT is a psychologist with an interest in the judgmental aspects of forecasting and decision making. He is Editor of the Journal of Behavioral Decision Making and an Associate Editor of the International Journal of Forecasting and the Journal of Forecasting. He has published in such journals as Management Science, Current Anthropology, Journal of Direct Marketing, Memory and Cognition, and the International Journal of Information Management.