Abstract
The importance of software product evaluations will grow with the awareness of the need for better software quality. The process to conduct such evaluations is crucial to get evaluation results that can be applied and meet customers' expectations. This paper reviews a well-known evaluation process: the ISO 14598 standard. The review focuses on the difficulties in selecting and evaluating the appropriate evaluation techniques. The review shows that the standard has problems in applying evaluation processes in practice due to insufficient attention to goal definition and to relationships between activities being implicit. Also, the standard ignores the trade-off between goals and resources and pays insufficient attention to feedback. To address these deficiencies, the W-process is proposed. It extends the standard through an improved process structure and additional guidelines.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Afotec. 1996. Software Maintainability Evaluation Guide. Kirtland, USA, Department of the Air Force.
Azuma, M. 2001. SQuaRE: The next generation of the ISO/IEC 9126 and 14598 international standards series on software product quality, Proceedings of the ESCOM 2001, April 2001, London, pp. 337-346.
Bache, R. and Bazzana, G. 1994. Software Metrics for Product Assessment. London, McGraw-Hill Book Company.
Basili, V. and Weiss, D. 1984. A methodology for collecting valid software engineering data, IEEE Transactions on Software Engineering 10: 728-738.
Boegh, J., Depanfilis, S., Kitchenham, B. and Pasquini, A. 1999. A method for software quality, planning, control and evaluation, IEEE Software 16: 69-77.
Briand, L., Wüst, J. and Lounis, H. 1999. Using coupling measurement for impact analysis in OO systems, Proceedings of International Conference on Software Maintenance (ISCM), pp. 475-482.
Cartwright, M. and Shepperd, M. 2000. An empirical investigation of an object-oriented software system, IEEE Transactions on Software Engineering 26: 786-796.
Colombo, R. and Guerra, A. 2002. The evaluation method for software products, Proceedings of ICSSEA'2002—International Conference Software &; Systems Engineering and Their Applications, Paris.
Dromey, G. 1996. Cornering the Chimera, IEEE Software 13: 33-43.
Fenton, N. and Pfleeger, S. 1996. Software Metrics, a Rigorous &; Practical Approach, London, International Thomson Computer Press.
Hausen, H. and Welzel, D. 1993. Guides to software evaluation, Arbeitspapiere der GMD 746.
Heemstra, F., Kusters, R. and Trienekens, J. 1994. From Quality Needs to Quality Requirements (in Dutch). Leidschendam, Lansa Publishing.
IEEE 1471. 2000. IEEE Recommended Practice for Architecture Description, IEEE Std. 1471.
ISO 8402. 1994. Quality Management and Quality Assurance Vocabulary. Geneve, ISO/IEC.
ISO 9126. 1991. Information Technology—Software Product Evaluation—Quality Characteristics and Guidelines for Their Use. Genève, ISO/IEC.
ISO 14598. 1999. Information Technology—Software Product Evaluation, Parts 1-5. Genève, ISO/IEC.
ISO FDIS 14598. 1999. Information Technology-Software Product Evaluation, Part 6: Documentation of Evaluation Modules. Genève, ISO/IEC.
Kazman, R., Klein, M., Barbacci, M., Longstaff, T., Lipson, H. and Carrière, J. 1998. The architecture tradeoff analysis method, Proceedings of the 4th International Conference on Engineering of Complex Computer Systems, pp. 68-78.
Kitchenham, B., Lawrence Pfleeger, S. and Fenton, N. 1995. Towards a framework for software measurement validation, IEEE Transactions on Software Engineering 21: 929-943.
Koshgoftaar, T.M. and Allen, E.B. 1998. Predicting the order of fault-prone modules in legacy software, IEEE Transactions on Software Engineering 24: 344-353.
Mayrand, J. and Coallier, F. 1996. System acquisition based on software product assessment, Proceedings of International Conference on Software Engineering (ICSE), pp. 210-219.
Miller, K. and Dunn, D. 1997. Post implementation evaluation of IS/IT: a survey of UK practice, Proceedings of 4th European conference on the Evaluation of Information Technology (EVIT), pp. 47-56.
Park, R., Goethert, W. and Florac, W. 1996. Goal-Driven Software Measurement—A Guidebook, SEI report CMU/SEI-96-HB-002.
Pivka, M. and Potocan, V. 1996. How can software packages certification improve software process, Proceedings of International Conference on Software Quality, Additional paper No. 3, 9 pages.
Punter, T. 1998. Developing an evaluation module to assess software maintainability, Proceedings of Conference on Empirical Assessment in Software Engineering (EASE), Keele, 1998.
Punter, T. 2001. Goal Oriented Software Evaluation (in Dutch), Ph.D. Thesis, Eindhoven University of Technology.
Punter, T. and Lami, G. 1998. Factors of software quality evaluation, Proceedings European Software Control and Metrics Conferences (ESCOM), pp. 257-266.
Punter, T., van Solingen, R. and Trienekens, J. 1997. Software product evaluation—current status and future needs for customers and industry, Proceedings of 4th European Conference on Evaluation of Information Technology, pp. 57-66.
Rae, A., Hausen, H. and Robert, P. 1995. Software Evaluation for Certification. Principles, Practice and Legal Liability. London, McGraw-Hill.
Space-Ufo Consortium. 1998. The Space Ufo Methodology—User Guide. Esprit project P22290.
Starlander, M. and Popescu-Belis, A. 2002. Corpus-based evaluation of a french spelling and grammar checker, Proceedings of 3rd International Conference on Language Resources and Evaluation, pp. 268-274.
van Solingen, R. and Berghout, E. 1999. The Goal/Question/Metric Method—A Practical Guide for Quality Improvement of Software Development. London, McGraw-Hill.
van der Zwan, M. 1995. A Quality Profile—A Foundation to Certify Software Product Quality (in Dutch), M.Sc. thesis, Eindhoven University of Technology.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Punter, T., Kusters, R., Trienekens, J. et al. The W-Process for Software Product Evaluation: A Method for Goal-Oriented Implementation of the ISO 14598 Standard. Software Quality Journal 12, 137–158 (2004). https://doi.org/10.1023/B:SQJO.0000024060.32026.a2
Issue Date:
DOI: https://doi.org/10.1023/B:SQJO.0000024060.32026.a2