Subject: Summary of Responses: Delphi Research From: "Leslie William Young" Date: Wed, 28 Aug 2002 21:30:42 -0400 X-Message-Number: 3 Dear Colleagues, In March 2002, I had posted a request on Web-Based Delphi Studies. I would like to thank everyone who responded to the request. I am posting below my original request followed by the summary of the responses. I also apologise for the lateness of this summary Original Request ================ Dear colleagues, I am currently looking for source material or references concerning DELPHI studies using expert panels, specifically DELPHI studies that have been administered over the web. Any references or insight to this will be highly appreciated. I'd also enjoy hearing from anyone with special interest in or experience with DELPHI methods. As is customary, I will post a summary of responses at a later date. Leslie Young PostGrad Room L3.45 Department of Information Systems (DIS) 207 Bouverie St The University of Melbourne Email: l.young@pgrad.unimelb.edu.au x40883 Summary of Responses =================== Summary of Literature Review ------------------------------------------ General Delphi Literature review: Dalkey, N. C. (1969). "The Delphi Method: An Experimental Study of Group Opinion." RM-5888-PR, Rand Corporation, Santa Monica, CA. Dalkey, N. C. (1963). "An Experimental Application of the Delphi Method to the Use of Experts." Management Science, 9(3), P458-467. Armstrong, J. S. (1985). Long-Range Forecasting. From Crystal Ball to Computer, Wiley-Internscience Publication, United States of America. Computer related approaches: Brockhoff, K. (1975). The performance of forecasting groups in computer dialogue and face-to-face discussion., Addison-Wesley. Bowman, J. P. (1992). "The Influence of Electronic Transmission on Written Communication." Mid-American Journal of Business, 7(2), p3-10. McCoy, R. W. (2001). "Computer Competencies for the 21st Century Information Systems Educator." Information Technology, Learning, and Performance Journal, 19(2), 21-35. Schmidt, R., Lyytinen, K., Keil, M., and Paul, C. (2001). "Identifying Software Project Risks: An International Delphi Study." Journal of Management Information Systems, 17(4), P5-36 Delphi used as a survey approach: Dickson, G. W., and Nechis, M. (1984). "Key Information Systems Issues for the 1980." MIS Quarterly, 8(3), P135-159. Brancheau, J. C., and Wetherbe, J. C. (1987). "Key Issues in Information Systems." MIS Quarterly, 11(1), 23-45. Couger, J. D. (1988). "Key Human Resource Issues in IS in the 1990s" Views of IS Executives versus Human Resource Executives." Information & Management, 14, p161-174. Brancheau, J. C., Janz, B. D., and Wetherbe, J. C. (1996). "Key Issues in Information Systems Management: 1994-95 SIM Delphi Result." MIS Quarterly, 20(2), P225-242. Chang, S.-I., Gable, G., Smythe, E., and Timbrell, G. (2000). "A Delphi Examination of Public Sector ERP Implementation Issues." 21st International Conference of Information Systems, Brisbane, Australia., 494-500. Schmidt, R. (1997). "Managing Delphi Surveys Using Nonparametric Statistical Techniques." Decision Sciences, 28(3), P763-774. Other References provided -------------------------- Chang, S.-I. and Gable, G. "A Critique of the Delphi Method in the Context of IS Key Issues Studies," Proceedings of the Pacific Asia Conference on Information System, Hong Kong, 2000, Chang, S.-I., Gable, G., Smythe, E. and Timbrell, G. "Methods for Distilling IS Key Issues Using a Delphi Approach," Proceedings of the ACIS, 2000, pp. 1-11. Chang, S.-I. and Gable, G.G. "Major Issues with SAP Financials in Queensland Government," Proceedings of the Americas Conference on Information Systems, Long Beach, California, 2000, pp. 972-976. Bedford, Michael T,. "The Value of Competing Panels of Experts and the Impact of 'Drop-outs' on Delphi Results", Delphi: The Bell Canada Experience, Bell Canada, October 1972 Cline, Alan. "Prioritization Process Using Delphi Technique", Http://www.carolla.com/wp-delphi.htm. Dekleva, S. and Zupancic, J. (1996) "Key Issue in Information Systems Management: A Delphi Study in Slovenia," Information and Management, 31(1), 1-11 Delbeq, Andre L., Andrew H. Van De Ven and David H. Gustafson (1975), Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes, Scott, Foresman and Company, Glenview, Illinois. Doke, E.R. and Swanson, N.E (1995), Decision Variables for Selecting Prototyping in Information Systems Development: A Delphi Study of MIS Managers, Information and Management, 29, 173-182 Fischer, R. (1978), The Delphi Method: A Description, Review and Criticism, Journal of Academic Librarianship, 4, 64-70 Hill, K.Q. and Fowles, J (1975), The methodological worth of the Delphi forecasting technique, Technological forecasting and social change, 7, 193-194 Linstone, H. and Turoff, M (1975) The Delphi Method: Techniques and Applications, Addison-Wesley Millar, V (1984), The Delphi Technique, Information Strategy: The Executive's Journal, 1, 32-36 Sackman, H (1974), Delphi Assessment: Expert Opinion, Forecasting, and Group Process. The Rand Corporation Report #R-1283-PR, April Turoff, Marray, and Hiltz, Starr Roxanne, Computer Based Delphi Processes" will appear as a chapter in Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, http://eies.njit.edu/~turoff/Papers/delphi3.html Good source for other Turoff papers: http://eies.njit.edu/~turoff Researchers who responded ----------------------------------------- Michael Lang [Michael.Lang@nuigalway.ie] Schmidt, Roy [roy@bumail.bradley.edu] Maha Shakir [m.shakir@massey.ac.nz] Researchers who are presently conducting or have recently conducted Delphi research ------------------------------------------------------------------------------------------------------------------------------- Mark Dixon [mdixon@ecel.uwa.edu.au] Katherine Denise Patterson [kdp@sunset.backbone.olemiss.edu] who kindly sent me her working paper. Ben Brunk [brunkb@ils.unc.edu] proposal online at http://ils.unc.edu/~brunkb/dissertation.html. Casey Cegielski [casey@business.auburn.edu] Respondents with comments --------------------------------------------- Viehland, Dennis [D.Viehland@massey.ac.nz] You raise a very interesting question because there is a danger here. It is quite acceptable to use Web pages as a substitute for paper and to poll your panel members. For example, instead of posting or e-mailing 2nd round results to your panellists, put the results on a Web page and direct them there. And instead of having them rank, rate or comment on paper forms, ask them to do so in an on-line form. Obviously you have to consider the experiences and attitudes of your panel--I would not do this with people who have little experience with the Web--and you should always give them the option of paper or Web. The danger is if you use a Web-based discussion group to "facilitate interaction" within the Delphi panel. This is an absolute no-no because you undermine the independent judgement of the panellists and you are likely to impede the discussion. An important factor in the validity of Delphi survey results is the independence of judgment exercised by the panel members. So if you create a Web-based discussion group where Panellist B can see ratings and rankings of Panellist A before submitting her results, then the principle of independent judgement has been violated. It is also possible that this undermines the anonymity of panel members, which also makes the judgments less independent. A Web-based discussion between the panellists can impede, not increase, the discussion. A panellist may enter the discussion with a certain view, but after reading what others have said may decide not to voice that view or consider "I have nothing new to add", which is, of course, exactly what you don't want.