Skip to main content
SpringerLink
Log in

Validation of design methods: lessons from medicine

  • Original Paper
  • Published:
Research in Engineering Design Aims and scope Submit manuscript

Abstract

This paper discusses the validation of design methods. The challenges and opportunities in validation are illustrated by drawing an analogy to medical research and development. Specific validation practices such as clinical studies and use of models of human disease are discussed, including specific ways to adapt them to engineering design. The implications are explored for three active areas of design research: robust design, axiomatic design, and design decision making. It is argued that medical research and development has highly-developed, well-documented validation methods and that many specific practices such as natural experiments and model-based evaluations can profitably be adapted for use in engineering design research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • American Institute of Aeronautics and Astronautics (1998) Guide for the verification and validation of computational fluid dynamics simulations, AIAA G-077–1998

  • Argyris C (1991) Teaching smart people how to learn. Harvard Business Review, Reprint No. 91301

  • Audi R (ed) (1995) The Cambridge dictionary of philosophy. Cambridge University Press, Cambridge

  • Bland M (1987) An introduction to medical statistics. Oxford University Press, New York

    Google Scholar 

  • Borror CM, Montgomery DC (2000) Mixed resolution designs as alternatives to Taguchi inner/outer array designs for robust design problems. Qual Reliability Int 16:117–127

    Article  Google Scholar 

  • Box GEP, Hunter WG, Hunter JS (1978) Statistics for experimenters: an introduction to design, data analysis, and model building. Wiley, New York

    MATH  Google Scholar 

  • Box GEP, Liu PTY (1999) Statistics as a catalyst to learning by scientific method. J Qual Technol 31(1):1–29

    Google Scholar 

  • Cacciabue PC, Hollnagel E (1995) Simulation of cognitive applications. Expertise and technology: cognition and human computer cooperation. Lawrence Erlbaum Associates, Mahwah, pp 55–73

  • Chakrabarti A, Morgenstern S, Knaab H (2004) Identification and application of requirements and their impact on the design process: a protocol study. Res Eng Des 15(1):22–39

    Article  Google Scholar 

  • Doll R, Hill AB (1950) Smoking and carcinoma of the lung: preliminary report. Br Med J 2:739–748

    Article  Google Scholar 

  • Dorst K, Vermass PE (2005) John Gero’s function–behavior–structure model of designing: a critical analysis. Res Eng Des 16(1):17–26

    Article  Google Scholar 

  • Dym CL (1994a) Representing designed objects: the languages of engineering design. Arch Comput Methods Eng 1(1):75–108

    Article  MathSciNet  Google Scholar 

  • Dym CL (1994b) Engineering Design: A Synthesis of Views. Cambridge University, New York

  • Dym CL, Wood WH, Scott MJ (2002) Rank ordering engineering designs: pairwise comparison charts and boorda counts. Res Eng Des 13:236–242

    Google Scholar 

  • Dym CL (2004) Principles of mathematical modeling, 2nd edn. Elsevier, Boston

    Google Scholar 

  • Dym CL, Little P (2004) Engineering design: a project-based introduction, 2nd edn. Wiley, New York

    Google Scholar 

  • Easterbrook PJ, Berlin JA, Gopalan R, Mathews DR (1991) Publication bias in clinical research. Lancet 337:867–872

    Article  Google Scholar 

  • Fisher RA (1958) Cigarettes, cancer, and statistics. Centennial Rev II(2):151–166

    Google Scholar 

  • Frey DD, Li X (2004) Validating Robust parameter design methods, DETC2004-57518. In: ASME design engineering technical conference, September 28 to October 2, Salt Lake City, Utah

  • Gainetdinov RR, Wetsel WC, Jones SR, Levin ED, Jaber M, Caron MG (1999) Role of setotonin in the paradoxical calming effect of psychostimulants on hyperactivity. Science 283:397–401

    Article  Google Scholar 

  • Griffin A (1989) Evaluating development processes: QFD as an example. Marketing Science Institute Report, pp 91–121

  • Hasselman TK (2001) Quantification of uncertainty in structural dynamics models. J Aerospace Eng 14(4):158–165

    Article  Google Scholar 

  • Hasselman TK, Anderson MC, Gan W (1998) Principal components analysis for non-linear model correlation, updating, and uncertainty evaluation. In: Proceedings of the 16th IMAC, Bethel, pp 644–651

  • Hauser J, Clausing DP (1988) The house of quality, Harvard Business Review

  • Hazelrigg GA( 1998) A framework for decision-based engineering design. ASME J Mech Des 120:653–658

    Article  Google Scholar 

  • Hazelrigg GA (1999) An axiomatic framework for engineering design. ASME J Mech Des 121:342–347

    Article  Google Scholar 

  • Hazelrigg GA (2003) Thoughts on model validation for engineering design, DETC 2003/DTM48632. In: ASME design engineering technical conference, September 2–6, Chicago

  • Hill BA (1966) The environment and disease: association or causation?. Proc Roy Soc Med 58:295

    Google Scholar 

  • Hirschi NW, Frey DD (2002) Cognition and complexity: an experiment on the effect of coupling in parameter design. Res Eng Des 13(3):123–131

    Google Scholar 

  • Institute of Electrical and Electronics Engineers (1998) IEEE Standard for Software Verification and Validation, IEEE Std 1012–1998

  • Klein G, Ross KG, Moon BM, Klein DE, Hoffman RR, Hollnagel E (2003) Macrocognition. IEEE Intell Syst 18(3):81–84

    Article  Google Scholar 

  • Kunert J (2004) A comparison of Taguchi’s product array and the combined array in robust-parameter-design. In: Eleventh annual spring research conference (SRC) on statistics in industry and technology, Gaithersburg, May 19–21

  • Kuppuraju N, Ittimakin P, Mistree F (1985) Design through selection: a method that works. Des Stud 6(2):91–106

    Google Scholar 

  • Li X, Frey DD (2005) A study of factor effects in design of experiments. DETC2005-85486. In: Proceedings of ASME design engineering technical conferences, Long Beach, September 24, pp 1–10

  • McAdams DA, Dym CL (2004) Modeling and information in the design process. In: Proceedings of the 2004 ASME design engineering technical conferences, Salt Lake City, September 2004

  • Moss J, Cagan J (2004) Learning from design experience in an agent-based design system. Res Eng Des 15(2):77–92

    Google Scholar 

  • Olewnik AT, Lewis KE (2005) On validating engineering design decision support tools. Concurr Eng Res Appl 13(2):111–122

    Article  Google Scholar 

  • Pedersen K, Emblemsvag J, Bailey R, Allen JK, Mistree F (2000) Validating design methods & research: the validation square. DETC2000/DTM14579. In: Proceedings of the ASME design engineering technical conference, Baltimore

  • Phadke MS (1989) Quality engineering using robust design. PTR Prentice-Hall, Inc., A Simon & Schuster Company, Englewood Cliffs

    Google Scholar 

  • Reich Y (1994) Layered models of research methodologies. Artif Intell Eng Des Anal Manuf 8(4):263–274

    Google Scholar 

  • Reich Y, Kohlberg E, Levin I (2006) Designing contexts for learning design. Int J Eng Educ (in press)

  • Saari DG (2001) Decisions and elections: explaining the unexpected. Cambridge University Press, New York

    MATH  Google Scholar 

  • Savage LJ (1954) The foundations of statistics. Dover Publications Inc., New York

    MATH  Google Scholar 

  • Schön DA (1983) The reflective practitioner: how professionals think in action. Basic Books, New York

    Google Scholar 

  • Schön DA, Argyris C (1975) Theory in practice: increasing professional effectiveness. Jossey-Bass, San Fransisco

    Google Scholar 

  • Simon HA (1990) Invariants of human behavior. Annu Rev Psychol 41:1–19

    Article  Google Scholar 

  • Simon HA (1996) The sciences of the artificial, 3rd edn. MIT Press, Cambridge

    Google Scholar 

  • Suh NP (1990) The principles of design. Oxford University Press, Oxford

    Google Scholar 

  • Suh NP (1998) Axiomatic design theory for systems. Res Eng Des 10:189–209

    Article  Google Scholar 

  • Taguchi G (1987) System of experimental design: engineering methods to optimize quality and minimize costs. Translated by Tung LW, QualityResources: a Division of the Kraus Organization Limited, White Plains, and American Supplier Institute, Inc. Dearborn, vol 1, pp 1–531

  • Todd PM, Gigerenzer G (2003) Bounding rationality to the World. J Econ Psychol 24:143–165

    Article  Google Scholar 

  • U. S. Department of Health and Human Services—Food and Drug Administration (1996) Good clinical practice: consolidated guidance. http://www.fda.gov/cder/guidance/idex.htm

  • U. S. Department of Health and Human Services—Food and Drug Administration (1998a) Providing clinical evidence of effectiveness for human drug and biological products, http://www.fda.gov/cder/guidance/idex.htm

  • U. S. Department of Health and Human Services—Food and Drug Administration (1998b) Statistical principles for clinical trials, http://www.fda.gov/cder/guidance/idex.htm

  • Wilkening S, Sobek DK (2004) Relating design activity to quality of outcome: a regression analysis of student projects. In: ASME international design engineering technical conferences. Salt Lake City, Sept 28–Oct 2

  • Wu CFJ, Hamada M (2000) Experiments: planning, analysis, and parameter design optimization. Wiley, New York

    MATH  Google Scholar 

Download references

Acknowledgments

D. D. Frey gratefully acknowledges the financial support of the National Science Foundation (award #0448972) and the Ford/MIT Alliance. The extensive comments offered by Yoram Reich have been very beneficial to the authors in completing this manuscript, as have the suggestions of the anonymous reviewers.

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering and Engineering Systems Division, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA, 02139, USA

    Daniel D. Frey

  2. Department of Engineering, Harvey Mudd College, 301 Platt Boulevard, Claremont, CA, 91711–5990, USA

    Clive L. Dym

Authors
  1. Daniel D. Frey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Clive L. Dym
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel D. Frey.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Frey, D.D., Dym, C.L. Validation of design methods: lessons from medicine. Res Eng Design 17, 45–57 (2006). https://doi.org/10.1007/s00163-006-0016-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00163-006-0016-4

Keywords

Search

Navigation